[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468839016 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.032 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002331] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006878] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4800.06 BogoMIPS (lpj=2400032) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009129] LSM: Security Framework initializing [ 0.010048] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012087] *** VALIDATE selinux *** [ 0.021078] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026229] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.031285] *** VALIDATE proc *** [ 0.032241] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035141] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042572] debug: unmapping init [mem 0xffffffffb4c59000-0xffffffffb4c60fff] [ 0.044162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045716] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053302] rcu: Hierarchical SRCU implementation. [ 0.055486] smp: Bringing up secondary CPUs ... [ 0.056475] x86: Booting SMP configuration: [ 0.057020] .... node #0, CPUs: #1 #2 #3 [ 0.060801] smp: Brought up 1 node, 4 CPUs [ 0.061815] smpboot: Max logical packages: 1 [ 0.062011] smpboot: Total of 4 processors activated (19200.25 BogoMIPS) [ 0.142013] node 0 deferred pages initialised in 79ms [ 0.145150] devtmpfs: initialized [ 0.146201] x86/mm: Memory block size: 128MB [ 0.148698] gcov: version magic: 0x41383552 [ 0.149572] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151238] pinctrl core: initialized pinctrl subsystem [ 0.152185] [ 0.153009] ************************************************************* [ 0.154015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155013] ** ** [ 0.156011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157012] ** ** [ 0.158015] ** This means that this kernel is built to expose internal ** [ 0.159013] ** IOMMU data structures, which may compromise security on ** [ 0.160013] ** your system. ** [ 0.161018] ** ** [ 0.162013] ** If you see this message and you are not debugging the ** [ 0.163016] ** kernel, report this immediately to your vendor! ** [ 0.164012] ** ** [ 0.165014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166014] ************************************************************* [ 0.167528] NET: Registered protocol family 16 [ 0.168426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171609] cpuidle: using governor menu [ 0.173795] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176609] PCI: Using configuration type 1 for base access [ 0.179125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189100] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192081] cryptd: max_cpu_qlen set to 1000 [ 0.195755] ACPI: Added _OSI(Module Device) [ 0.198016] ACPI: Added _OSI(Processor Device) [ 0.200028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206595] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214397] ACPI: Interpreter enabled [ 0.216058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.217010] ACPI: Using IOAPIC for interrupt routing [ 0.219109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232745] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.247334] acpiphp: Slot [2] registered [ 0.249081] acpiphp: Slot [5] registered [ 0.251113] acpiphp: Slot [6] registered [ 0.253116] acpiphp: Slot [7] registered [ 0.255141] acpiphp: Slot [8] registered [ 0.256106] acpiphp: Slot [9] registered [ 0.258131] acpiphp: Slot [10] registered [ 0.260113] acpiphp: Slot [3] registered [ 0.261091] acpiphp: Slot [4] registered [ 0.263106] acpiphp: Slot [11] registered [ 0.264100] acpiphp: Slot [12] registered [ 0.266106] acpiphp: Slot [13] registered [ 0.267095] acpiphp: Slot [14] registered [ 0.269100] acpiphp: Slot [15] registered [ 0.271125] acpiphp: Slot [16] registered [ 0.272123] acpiphp: Slot [17] registered [ 0.274077] acpiphp: Slot [18] registered [ 0.276145] acpiphp: Slot [19] registered [ 0.278120] acpiphp: Slot [20] registered [ 0.280123] acpiphp: Slot [21] registered [ 0.282095] acpiphp: Slot [22] registered [ 0.284102] acpiphp: Slot [23] registered [ 0.285099] acpiphp: Slot [24] registered [ 0.287107] acpiphp: Slot [25] registered [ 0.288103] acpiphp: Slot [26] registered [ 0.290100] acpiphp: Slot [27] registered [ 0.291060] acpiphp: Slot [28] registered [ 0.292093] acpiphp: Slot [29] registered [ 0.294107] acpiphp: Slot [30] registered [ 0.296113] acpiphp: Slot [31] registered [ 0.297051] PCI host bridge to bus 0000:00 [ 0.299017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.301020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.303018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.310026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.312169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.315904] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.319644] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.331016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.335559] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.338020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.340016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.342012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.344521] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347871] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.353696] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.358013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.371013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.376015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.382401] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.388015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.392014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408053] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415461] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.421022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.429016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.459635] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.465015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.472013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.490014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.500377] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.507014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.517015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.533018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.543400] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.550014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.559024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.576015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.588108] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.607015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.624014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.635418] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.638383] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.642457] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.644499] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.647261] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.653025] iommu: Default domain type: Passthrough [ 0.654436] SCSI subsystem initialized [ 0.656122] ACPI: bus type USB registered [ 0.657113] usbcore: registered new interface driver usbfs [ 0.660079] usbcore: registered new interface driver hub [ 0.662073] usbcore: registered new device driver usb [ 0.664286] pps_core: LinuxPPS API ver. 1 registered [ 0.666011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.669060] PTP clock support registered [ 0.671133] EDAC MC: Ver: 3.0.0 [ 0.673097] PCI: Using ACPI for IRQ routing [ 0.675932] NetLabel: Initializing [ 0.677011] NetLabel: domain hash size = 128 [ 0.678009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.680077] NetLabel: unlabeled traffic allowed by default [ 0.682102] vgaarb: loaded [ 0.683251] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.685012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.692602] clocksource: Switched to clocksource kvm-clock [ 0.799503] VFS: Disk quotas dquot_6.6.0 [ 0.801446] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.804736] *** VALIDATE ramfs *** [ 0.806262] *** VALIDATE hugetlbfs *** [ 0.808187] pnp: PnP ACPI init [ 0.811089] pnp: PnP ACPI: found 6 devices [ 0.829188] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.832866] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.835510] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.837797] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.839976] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.842043] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.844719] NET: Registered protocol family 2 [ 0.847079] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.851571] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.855567] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.862098] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.865476] TCP: Hash tables configured (established 65536 bind 65536) [ 0.868614] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.871348] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873611] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.876792] NET: Registered protocol family 1 [ 0.879375] RPC: Registered named UNIX socket transport module. [ 0.881365] RPC: Registered udp transport module. [ 0.883088] RPC: Registered tcp transport module. [ 0.884492] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.886527] NET: Registered protocol family 44 [ 0.887910] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.889764] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.891695] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.893856] PCI: CLS 0 bytes, default 64 [ 0.895402] Unpacking initramfs... [ 2.277990] debug: unmapping init [mem 0xffff89d9fcc54000-0xffff89d9fffbffff] [ 2.282228] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.283892] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.286479] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 2.761877] Initialise system trusted keyrings [ 2.763019] Key type blacklist registered [ 2.764407] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.774587] zbud: loaded [ 2.778303] *** VALIDATE nfs *** [ 2.779686] *** VALIDATE nfs4 *** [ 2.781120] pstore: using deflate compression [ 2.784394] Platform Keyring initialized [ 2.882587] NET: Registered protocol family 38 [ 2.883942] Key type asymmetric registered [ 2.885386] Asymmetric key parser 'x509' registered [ 2.886980] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.889734] io scheduler mq-deadline registered [ 2.891554] io scheduler kyber registered [ 2.893805] io scheduler bfq registered [ 2.895558] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.898317] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.900232] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.902694] ACPI: Power Button [PWRF] [ 2.908364] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.916280] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.931189] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.939383] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.951966] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.980508] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.010373] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.015676] Non-volatile memory driver v1.3 [ 3.016974] Linux agpgart interface v0.103 [ 3.047176] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.050328] vda: detected capacity change from 0 to 34836480 [ 3.065761] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.068852] vdb: detected capacity change from 0 to 1073741824 [ 3.084407] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.087328] vdc: detected capacity change from 0 to 2621440000 [ 3.101511] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.104649] vdd: detected capacity change from 0 to 2621440000 [ 3.120753] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.124087] vde: detected capacity change from 0 to 4294967296 [ 3.139144] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.143887] vdf: detected capacity change from 0 to 4294967296 [ 3.152287] libphy: Fixed MDIO Bus: probed [ 3.156826] usbcore: registered new interface driver usbserial_generic [ 3.158915] usbserial: USB Serial support registered for generic [ 3.161220] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.166818] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.169258] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.171655] mousedev: PS/2 mouse device common for all mice [ 3.174869] rtc_cmos 00:05: RTC can wake from S4 [ 3.177366] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.182265] rtc_cmos 00:05: registered as rtc0 [ 3.182678] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.184408] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.189437] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.190176] intel_pstate: CPU model not supported [ 3.195310] hid: raw HID events driver (C) Jiri Kosina [ 3.197605] usbcore: registered new interface driver usbhid [ 3.199907] usbhid: USB HID core driver [ 3.201598] drop_monitor: Initializing network drop monitor service [ 3.204431] Initializing XFRM netlink socket [ 3.206409] NET: Registered protocol family 10 [ 3.209343] Segment Routing with IPv6 [ 3.210687] NET: Registered protocol family 17 [ 3.212642] mpls_gso: MPLS GSO support [ 3.217830] RAS: Correctable Errors collector initialized. [ 3.219765] AVX version of gcm_enc/dec engaged. [ 3.221274] AES CTR mode by8 optimization enabled [ 3.281864] sched_clock: Marking stable (3281844219, 0)->(4213314511, -931470292) [ 3.285193] registered taskstats version 1 [ 3.287361] Loading compiled-in X.509 certificates [ 3.288799] zswap: loaded using pool lzo/zbud [ 3.312593] Key type big_key registered [ 3.322585] Key type encrypted registered [ 3.324351] ima: No TPM chip found, activating TPM-bypass! [ 3.327615] ima: Allocated hash algorithm: sha1 [ 3.329115] ima: No architecture policies found [ 3.330913] evm: Initialising EVM extended attributes: [ 3.332671] evm: security.selinux [ 3.333478] evm: security.ima [ 3.334311] evm: security.capability [ 3.335615] evm: HMAC attrs: 0x1 [ 3.337624] rtc_cmos 00:05: setting system clock to 2026-04-14 14:59:51 UTC (1776178791) [ 3.343737] debug: unmapping init [mem 0xffffffffb5c03000-0xffffffffb5dfffff] [ 3.346993] debug: unmapping init [mem 0xffffffffb4982000-0xffffffffb4c58fff] [ 3.354081] Write protecting the kernel read-only data: 28672k [ 3.358129] debug: unmapping init [mem 0xffffffffb3003000-0xffffffffb31fffff] [ 3.359878] debug: unmapping init [mem 0xffffffffb3914000-0xffffffffb39fffff] [ 3.388186] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.395819] systemd[1]: Detected virtualization kvm. [ 3.397620] systemd[1]: Detected architecture x86-64. [ 3.399545] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.426829] systemd[1]: No hostname configured. [ 3.428905] systemd[1]: Set hostname to . [ 3.431073] random: systemd: uninitialized urandom read (16 bytes read) [ 3.433793] systemd[1]: Initializing machine ID from random generator. [ 3.472814] random: ln: uninitialized urandom read (6 bytes read) [ 3.547758] random: systemd: uninitialized urandom read (16 bytes read) [ 3.550262] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.555335] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.557898] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.082156] device-mapper: uevent: version 1.0.3 [ 4.083743] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.885843] random: fast init done [ 4.922706] virtio_net virtio0 ens2: renamed from eth0 [ 4.951528] scsi host0: ata_piix [ 4.971254] scsi host1: ata_piix [ 4.981279] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.984188] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.768670] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.850276] random: crng init done [ 9.852048] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.299192] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.495954] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.749775] SELinux: Disabled at runtime. [ 11.815668] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.825456] systemd[1]: Detected virtualization kvm. [ 11.827469] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.345381] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.348891] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.353490] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.358388] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.361640] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.369637] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.374852] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ 12.494566] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.838082] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.225784] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.245163] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.398240] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.422511] EDAC sbridge: Ver: 1.1.2 [ 15.002639] Key type dns_resolver registered [ 15.328513] NFS: Registering the id_resolver key type [ 15.330728] Key type id_resolver registered [ 15.332771] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-server login: [ 37.496943] libcfs: loading out-of-tree module taints kernel. [ 37.517781] alg: No test for adler32 (adler32-zlib) [ 38.270468] Key type ._llcrypt registered [ 38.272017] Key type .llcrypt registered [ 38.320963] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 50.611569] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 51.949201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.510270] Lustre: Lustre: Build Version: 2.15.8_13_g05ba8a9 [ 53.302189] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 53.305864] LNet: Accept secure, port 988 [ 55.025259] Key type lgssc registered [ 56.530988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.891063] hrtimer: interrupt took 24602600 ns [ 77.603456] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 80.332649] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.331372] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.269131] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.963634] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.638737] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 139.204844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 139.281400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.754413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 140.802155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 140.872649] Lustre: lustre-MDT0000: new disk, initializing [ 140.945584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.961443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.480273] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 159.160344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 159.258991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.307680] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 159.335174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 159.346762] Lustre: Skipped 1 previous similar message [ 159.474168] Lustre: lustre-MDT0001: new disk, initializing [ 159.585894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.629494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.646965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 164.423050] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 178.843392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.980680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 178.994964] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 179.230928] Lustre: lustre-OST0000: new disk, initializing [ 179.236990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 179.357358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.433027] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 187.454873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 187.474571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.328358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.374523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.379194] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 201.451819] Lustre: lustre-OST0001: new disk, initializing [ 201.454739] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 201.500419] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.567459] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 208.895887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 208.904054] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.787796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.977625] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 232.508145] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.384427] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing yml_node [ 241.163948] Lustre: DEBUG MARKER: Client: 2.15.8.13 [ 243.812408] Lustre: DEBUG MARKER: MDS: 2.15.8.13 [ 246.399625] Lustre: DEBUG MARKER: OSS: 2.15.8.13 [ 247.747125] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Apr 14 11:03:54 EDT 2026 [ 255.101803] Lustre: DEBUG MARKER: excepting tests: 110f 131b 59 [ 260.101740] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing check_config_client /mnt/lustre [ 274.895800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 277.355499] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 280.673566] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 285.295300] Lustre: DEBUG MARKER: == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 11:04:31 (1776179071) [ 286.550516] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 286.552823] LustreError: 6268:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000000f2aca32 x1862458504427904/t4294967300(0) o1000->lustre-MDT0000-mdtlov_UUID@0@lo:55/0 lens 384/4320 e 0 to 0 dl 1776179080 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up1-0.0' [ 287.942277] Lustre: Failing over lustre-MDT0001 [ 288.050184] Lustre: server umount lustre-MDT0001 complete [ 290.784405] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 290.786302] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.786976] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.811662] Lustre: Skipped 2 previous similar messages [ 292.390876] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.403618] LustreError: Skipped 2 previous similar messages [ 293.855809] Lustre: 7709:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179074/real 1776179074] req@000000000d69cb51 x1862458504427904/t0(0) o1000->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 384/4320 e 0 to 1 dl 1776179081 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'osp_up1-0.0' [ 295.909914] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.925534] LustreError: Skipped 2 previous similar messages [ 301.026632] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.039299] LustreError: Skipped 3 previous similar messages [ 305.475671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 305.680380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 306.985176] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 309.184919] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 310.757713] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 310.776091] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 317.514861] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 318.799232] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 326.036948] Lustre: DEBUG MARKER: == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 11:05:12 (1776179112) [ 326.907341] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 326.910798] LustreError: 6264:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000009223e379 x1862458490773056/t4294967362(0) o36->5ac71b10-8319-4077-b5b0-a5dd565c5c3a@192.168.203.45@tcp:133/0 lens 560/448 e 0 to 0 dl 1776179158 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 328.271984] Lustre: Failing over lustre-MDT0000 [ 328.453961] Lustre: server umount lustre-MDT0000 complete [ 331.231685] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 331.232948] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.236913] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.236928] LustreError: Skipped 1 previous similar message [ 331.266812] Lustre: Skipped 3 previous similar messages [ 338.401053] Lustre: 3348:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179119/real 1776179119] req@00000000a86ce968 x1862458504444992/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179126 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 338.420686] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.490544] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.503223] LustreError: Skipped 13 previous similar messages [ 345.869197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 355.818661] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.823985] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873b908f4 to 0xedbf33d873b91a19 [ 355.845080] LustreError: Skipped 23 previous similar messages [ 355.862662] Lustre: MGC192.168.203.145@tcp: Connection restored to (at 0@lo) [ 355.866473] Lustre: Skipped 2 previous similar messages [ 356.030894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 356.870741] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 360.301440] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 361.442551] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 361.462916] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 361.473087] Lustre: 6263:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000fe2a2d1e x1862458490773056/t4294967362(0) o36->5ac71b10-8319-4077-b5b0-a5dd565c5c3a@192.168.203.45@tcp:167/0 lens 560/448 e 0 to 0 dl 1776179192 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 361.521325] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 361.528675] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 368.977213] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 370.343502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 380.652481] Lustre: DEBUG MARKER: == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 11:06:06 (1776179166) [ 389.394776] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 392.242816] Lustre: Failing over lustre-MDT0001 [ 392.257281] LustreError: 6249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776179180 with bad export cookie 17131468512288119405 [ 392.269550] LustreError: 6249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 392.756992] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 397.285972] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.290961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 397.292577] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 397.297236] Lustre: Skipped 2 previous similar messages [ 397.314097] Lustre: Skipped 2 previous similar messages [ 398.001302] Lustre: server umount lustre-MDT0001 complete [ 402.404290] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 402.423263] LustreError: Skipped 2 previous similar messages [ 412.115909] LDISKFS-fs (dm-1): recovery complete [ 412.119612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.386556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.410723] LustreError: 16937:0:(mdt_handler.c:7430:mdt_iocontrol()) lustre-MDT0001: Aborting MDT recovery [ 412.429802] LustreError: 16960:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 0, retries 0, failed: rc = -108 [ 413.194310] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 416.764877] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 417.765262] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 417.770079] Lustre: Skipped 3 previous similar messages [ 417.895296] Lustre: lustre-MDT0001: disconnecting 1 stale clients [ 417.904784] LustreError: 16961:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 0 != fo_tot_granted 2097152 [ 417.934454] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 417.955967] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 417.981712] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 1 recovered and 1 was evicted. [ 418.026986] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:46 to 0x2c0000400:65 [ 418.027247] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:46 to 0x280000400:65 [ 475.570457] Lustre: Failing over lustre-MDT0001 [ 475.754703] Lustre: server umount lustre-MDT0001 complete [ 479.200445] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.201609] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.216244] Lustre: Skipped 1 previous similar message [ 479.254821] LustreError: Skipped 7 previous similar messages [ 494.508347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.843606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 495.133650] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 500.021722] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 500.216876] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 500.224398] Lustre: Skipped 2 previous similar messages [ 500.257051] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 500.338837] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:46 to 0x2c0000400:97 [ 500.339228] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:46 to 0x280000400:97 [ 508.608643] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 510.202581] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 517.911909] Lustre: DEBUG MARKER: == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 11:08:24 (1776179304) [ 534.364510] Lustre: Failing over lustre-MDT0001 [ 534.495737] Lustre: server umount lustre-MDT0001 complete [ 536.036758] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.037064] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.054291] Lustre: Skipped 2 previous similar messages [ 540.858426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.127154] LustreError: 19691:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0001: Aborting client recovery [ 541.132949] LustreError: 19691:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery [ 541.135862] LustreError: 19713:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -108 [ 541.139941] Lustre: 19715:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 541.154396] Lustre: lustre-MDT0001: disconnecting 2 stale clients [ 541.159517] Lustre: lustre-MDT0001-osd: cancel update llog [0x2400013a0:0x1:0x0] [ 541.170886] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000bd1:0x1:0x0] [ 541.211152] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:46 to 0x280000400:129 [ 541.218199] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:46 to 0x2c0000400:129 [ 544.975222] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 546.277126] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 546.283617] Lustre: Skipped 2 previous similar messages [ 546.291994] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 565.796586] Lustre: DEBUG MARKER: == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 11:09:12 (1776179352) [ 573.793975] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 597.601964] Lustre: Failing over lustre-MDT0000 [ 597.762223] Lustre: server umount lustre-MDT0000 complete [ 602.593050] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.602798] Lustre: Skipped 3 previous similar messages [ 602.605864] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 606.510688] LDISKFS-fs (dm-0): recovery complete [ 606.513442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.604453] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.613435] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873b91a19 to 0xedbf33d873ba1ed9 [ 606.620843] Lustre: MGC192.168.203.145@tcp: Connection restored to (at 0@lo) [ 606.626694] Lustre: Skipped 2 previous similar messages [ 606.780544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 606.784881] Lustre: Skipped 1 previous similar message [ 606.811318] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 606.811489] LustreError: 21674:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 606.824729] LustreError: 21674:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 606.832895] Lustre: 21707:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 606.841380] Lustre: 21707:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 606.858473] LustreError: 21706:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -108 [ 606.865761] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 606.870923] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 606.880824] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 606.918335] Lustre: lustre-OST0001: deleting orphan objects from 0x0:39 to 0x0:65 [ 606.926168] Lustre: lustre-OST0000: deleting orphan objects from 0x0:38 to 0x0:65 [ 610.021300] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 611.818236] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 696.170117] Lustre: DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 11:11:22 (1776179482) [ 697.362854] Lustre: *** cfs_fail_loc=159, val=0*** [ 743.014684] Lustre: lustre-MDT0000: Client 5ac71b10-8319-4077-b5b0-a5dd565c5c3a (at 192.168.203.45@tcp) reconnecting [ 751.916313] Lustre: DEBUG MARKER: == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 11:12:18 (1776179538) [ 753.776604] Lustre: *** cfs_fail_loc=15a, val=0*** [ 796.225818] Lustre: lustre-MDT0001: Client 5ac71b10-8319-4077-b5b0-a5dd565c5c3a (at 192.168.203.45@tcp) reconnecting [ 796.266746] Lustre: 7588:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000add14575 x1862458492278784/t21474843527(0) o36->5ac71b10-8319-4077-b5b0-a5dd565c5c3a@192.168.203.45@tcp:601/0 lens 488/3152 e 0 to 0 dl 1776179626 ref 1 fl Interpret:/2/0 rc 0/0 job:'chmod.0' [ 796.311267] Lustre: 7588:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 6 previous similar messages [ 804.388564] Lustre: DEBUG MARKER: == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 11:13:10 (1776179590) [ 812.511421] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 813.865799] Lustre: *** cfs_fail_loc=15a, val=0*** [ 813.867768] Lustre: Skipped 6 previous similar messages [ 817.686538] Lustre: Failing over lustre-MDT0000 [ 817.831353] Lustre: server umount lustre-MDT0000 complete [ 821.727866] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 821.734334] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 821.735624] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 821.735631] LustreError: Skipped 25 previous similar messages [ 821.813792] Lustre: Skipped 3 previous similar messages [ 828.831937] Lustre: 3348:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179609/real 1776179609] req@000000002815a242 x1862458504922240/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179616 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 828.861315] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 840.158990] LDISKFS-fs (dm-0): recovery complete [ 840.167596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.311674] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873ba1ed9 to 0xedbf33d873bc384a [ 846.327855] Lustre: MGC192.168.203.145@tcp: Connection restored to (at 0@lo) [ 846.332863] Lustre: Skipped 4 previous similar messages [ 846.577534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.630668] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 846.641128] Lustre: Skipped 2 previous similar messages [ 848.310764] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 850.283955] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 852.019448] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 852.072899] Lustre: lustre-OST0001: deleting orphan objects from 0x0:73 to 0x0:97 [ 852.073995] Lustre: lustre-OST0000: deleting orphan objects from 0x0:73 to 0x0:97 [ 859.188242] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 861.573517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 870.362974] Lustre: DEBUG MARKER: == replay-single test 102d: check replay [ 871.729093] Lustre: *** cfs_fail_loc=15a, val=0*** [ 871.735903] Lustre: Skipped 6 previous similar messages [ 876.467781] Lustre: Failing over lustre-MDT0001 [ 876.648961] Lustre: server umount lustre-MDT0001 complete [ 877.537345] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.540029] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 877.544487] Lustre: Skipped 3 previous similar messages [ 895.613442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.943307] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 897.138070] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 899.827725] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 901.111022] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 901.165556] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1137 to 0x2c0000400:1153 [ 901.176141] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1137 to 0x280000400:1153 [ 901.187205] Lustre: 6264:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000065d524fa x1862458492308608/t21474843568(0) o36->5ac71b10-8319-4077-b5b0-a5dd565c5c3a@192.168.203.45@tcp:706/0 lens 488/3152 e 0 to 0 dl 1776179731 ref 1 fl Interpret:/2/0 rc 0/0 job:'chmod.0' [ 901.210878] Lustre: 6264:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 6 previous similar messages [ 906.445864] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 907.694571] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 916.018147] Lustre: DEBUG MARKER: == replay-single test 103: Check otr_next_id overflow ==== 11:15:02 (1776179702) [ 921.567613] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 928.735500] Lustre: 3345:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179709/real 1776179709] req@00000000daa2db07 x1862458504957632/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179716 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 928.773941] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.226462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.123749] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873bc384a to 0xedbf33d873bc5037 [ 945.149814] Lustre: MGC192.168.203.145@tcp: Connection restored to (at 0@lo) [ 945.161329] Lustre: Skipped 7 previous similar messages [ 945.430023] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 945.674557] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 949.648526] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 950.776359] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 950.827078] Lustre: lustre-OST0001: deleting orphan objects from 0x0:113 to 0x0:129 [ 950.827295] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:129 [ 957.340166] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 958.552558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 965.175599] Lustre: DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 11:15:51 (1776179751) [ 971.383327] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 972.815788] Lustre: Failing over lustre-MDT0000 [ 972.818589] Lustre: Skipped 1 previous similar message [ 972.991969] Lustre: server umount lustre-MDT0000 complete [ 972.995116] Lustre: Skipped 1 previous similar message [ 976.354336] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.372712] Lustre: Skipped 6 previous similar messages [ 983.519139] Lustre: 3346:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179764/real 1776179764] req@000000008502c455 x1862458504977600/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179771 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 983.538705] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 992.822138] LDISKFS-fs (dm-0): recovery complete [ 992.824756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.917566] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873bc5037 to 0xedbf33d873bc54f9 [ 1000.157427] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1000.974343] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1003.638975] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1005.597507] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1005.629953] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:161 [ 1005.630826] Lustre: lustre-OST0001: deleting orphan objects from 0x0:113 to 0x0:161 [ 1011.374497] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1012.557802] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1019.551170] Lustre: DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 11:16:46 (1776179806) [ 1025.124389] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1026.934714] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776179815 with bad export cookie 17131468512288331092 [ 1026.934800] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 1026.941043] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1026.958269] LustreError: Skipped 1 previous similar message [ 1026.961847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.966296] Lustre: Skipped 1 previous similar message [ 1038.304236] Lustre: 3345:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179819/real 1776179819] req@0000000087be7da5 x1862458504996160/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179826 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 1038.335867] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1051.090481] LDISKFS-fs (dm-0): recovery complete [ 1051.094307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.715117] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873bc54f9 to 0xedbf33d873bc5b43 [ 1056.134767] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1060.815445] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1071.706722] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1074.276960] Lustre: lustre-MDT0000: Denying connection for new client d30ce603-5e0a-4710-b828-215f7c630a4f (at 192.168.203.45@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:27 [ 1079.302048] Lustre: lustre-MDT0000: Denying connection for new client d30ce603-5e0a-4710-b828-215f7c630a4f (at 192.168.203.45@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:22 [ 1084.459052] Lustre: lustre-MDT0000: Denying connection for new client d30ce603-5e0a-4710-b828-215f7c630a4f (at 192.168.203.45@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:17 [ 1089.555430] Lustre: lustre-MDT0000: Denying connection for new client d30ce603-5e0a-4710-b828-215f7c630a4f (at 192.168.203.45@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:12 [ 1094.670121] Lustre: lustre-MDT0000: Denying connection for new client d30ce603-5e0a-4710-b828-215f7c630a4f (at 192.168.203.45@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:07 [ 1104.905508] Lustre: lustre-MDT0000: Denying connection for new client d30ce603-5e0a-4710-b828-215f7c630a4f (at 192.168.203.45@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:57 [ 1104.925297] Lustre: Skipped 1 previous similar message [ 1125.384595] Lustre: lustre-MDT0000: Denying connection for new client d30ce603-5e0a-4710-b828-215f7c630a4f (at 192.168.203.45@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:36 [ 1125.411963] Lustre: Skipped 3 previous similar messages [ 1161.225169] Lustre: lustre-MDT0000: Denying connection for new client d30ce603-5e0a-4710-b828-215f7c630a4f (at 192.168.203.45@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:00 [ 1161.245191] Lustre: Skipped 6 previous similar messages [ 1162.001072] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1162.007277] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1162.024520] Lustre: lustre-MDT0000: Recovery over after 1:41, of 2 clients 1 recovered and 1 was evicted. [ 1162.029111] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1162.029119] Lustre: Skipped 13 previous similar messages [ 1162.070121] Lustre: lustre-OST0001: deleting orphan objects from 0x0:113 to 0x0:193 [ 1162.071866] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:193 [ 1174.102340] Lustre: DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 11:19:20 (1776179960) [ 1179.758562] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1181.451176] Lustre: Failing over lustre-MDT0001 [ 1181.456266] Lustre: Skipped 1 previous similar message [ 1181.463895] LustreError: 22306:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776179969 with bad export cookie 17131468512288332709 [ 1181.606787] Lustre: server umount lustre-MDT0001 complete [ 1181.609474] Lustre: Skipped 1 previous similar message [ 1181.745068] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1181.762838] LustreError: Skipped 157 previous similar messages [ 1184.227442] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1184.241366] Lustre: Skipped 7 previous similar messages [ 1204.061628] LDISKFS-fs (dm-1): recovery complete [ 1204.064498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.340385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1204.348139] Lustre: Skipped 4 previous similar messages [ 1204.365606] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 1206.335815] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1206.340612] Lustre: Skipped 1 previous similar message [ 1208.797293] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1209.883827] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1137 to 0x2c0000400:1185 [ 1209.885096] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1137 to 0x280000400:1185 [ 1217.456954] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1219.422679] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1228.527824] Lustre: DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 11:20:15 (1776180015) [ 1236.148258] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1240.558504] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1261.184523] LDISKFS-fs (dm-1): recovery complete [ 1261.187617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.557684] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 1265.954148] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1274.605320] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1276.426785] Lustre: lustre-MDT0001: Denying connection for new client aee9f7d0-81d4-4c34-97b0-6bbdde6ee553 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:30 [ 1367.002133] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [ 1367.007614] Lustre: lustre-MDT0001: disconnecting 1 stale clients [ 1367.032575] Lustre: lustre-MDT0001: Recovery over after 1:41, of 2 clients 1 recovered and 1 was evicted. [ 1367.045682] Lustre: Skipped 1 previous similar message [ 1367.089093] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1137 to 0x2c0000400:1217 [ 1367.092480] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1137 to 0x280000400:1217 [ 1375.453928] Lustre: DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 11:22:42 (1776180162) [ 1382.727250] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1389.733621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1395.221931] LustreError: 22305:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776180183 with bad export cookie 17131468512288332611 [ 1395.223893] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.230955] LustreError: 22305:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1419.797137] LDISKFS-fs (dm-0): recovery complete [ 1419.803261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.322076] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873bc5b43 to 0xedbf33d873bc6bf1 [ 1428.339648] Lustre: MGC192.168.203.145@tcp: Connection restored to (at 0@lo) [ 1428.360451] Lustre: Skipped 6 previous similar messages [ 1428.945655] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1436.351611] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1441.248946] Lustre: 3346:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180187/real 1776180187] req@000000004c16ccf0 x1862458505087168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776180229 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1459.533283] LDISKFS-fs (dm-1): recovery complete [ 1459.540270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.318968] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1465.315439] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1465.322242] Lustre: Skipped 1 previous similar message [ 1466.030038] Lustre: lustre-OST0001: deleting orphan objects from 0x0:113 to 0x0:225 [ 1466.032539] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:225 [ 1472.929578] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1475.032888] Lustre: lustre-MDT0001: Denying connection for new client 8867edd6-1e1d-4c75-a3f2-5865a80ac164 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 1475.049847] Lustre: Skipped 17 previous similar messages [ 1535.002790] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [ 1535.010781] Lustre: lustre-MDT0001: disconnecting 1 stale clients [ 1535.098836] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1137 to 0x280000400:1249 [ 1535.099120] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1137 to 0x2c0000400:1249 [ 1545.014503] Lustre: DEBUG MARKER: SKIP: replay-single test_110f skipping excluded test 110f [ 1547.073750] Lustre: DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 11:25:33 (1776180333) [ 1556.076919] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1564.450089] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1566.295879] Lustre: Failing over lustre-MDT0000 [ 1566.299757] Lustre: Skipped 3 previous similar messages [ 1566.449830] Lustre: server umount lustre-MDT0000 complete [ 1566.459983] Lustre: Skipped 3 previous similar messages [ 1567.715842] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1567.736243] Lustre: Skipped 12 previous similar messages [ 1569.838158] LustreError: 22305:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776180357 with bad export cookie 17131468512288336881 [ 1569.839520] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.848627] LustreError: 22305:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1578.911137] Lustre: 3347:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180360/real 1776180360] req@00000000f9aeaaeb x1862458505127296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776180367 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1578.931644] Lustre: 3347:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 24 previous similar messages [ 1592.569180] LDISKFS-fs (dm-0): recovery complete [ 1592.575652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.552729] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873bc6bf1 to 0xedbf33d873bc74bf [ 1603.970144] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1603.974494] Lustre: Skipped 1 previous similar message [ 1607.639482] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1630.473037] LDISKFS-fs (dm-1): recovery complete [ 1630.476414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.688585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1630.730642] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1630.741143] LustreError: Skipped 2 previous similar messages [ 1634.625324] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1637.007630] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1637.021282] Lustre: Skipped 2 previous similar messages [ 1637.044805] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1137 to 0x2c0000400:1281 [ 1637.045667] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1137 to 0x280000400:1281 [ 1643.911490] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1706.000199] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1706.014656] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1706.084557] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:257 [ 1706.090808] Lustre: lustre-OST0001: deleting orphan objects from 0x0:113 to 0x0:257 [ 1718.301500] Lustre: DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 11:28:24 (1776180504) [ 1727.153227] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1731.552156] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.565894] LustreError: Skipped 73 previous similar messages [ 1740.767825] Lustre: 3346:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180521/real 1776180521] req@0000000038583361 x1862458505163264/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180528 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 1740.793836] Lustre: 3346:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 1740.803383] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.717116] LDISKFS-fs (dm-0): recovery complete [ 1753.722870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.166841] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873bc74bf to 0xedbf33d873bc7dfd [ 1757.513754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.516921] Lustre: Skipped 5 previous similar messages [ 1757.718779] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1757.724967] Lustre: Skipped 3 previous similar messages [ 1762.074376] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1762.919249] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:289 [ 1762.919340] Lustre: lustre-OST0001: deleting orphan objects from 0x0:113 to 0x0:289 [ 1771.242536] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1772.860655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1779.569944] Lustre: DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 11:29:26 (1776180566) [ 1785.750617] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1787.568709] LustreError: 22306:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776180575 with bad export cookie 17131468512288341585 [ 1810.490799] LDISKFS-fs (dm-1): recovery complete [ 1810.494153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.622956] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1822.792387] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1825.031201] Lustre: lustre-MDT0001: Denying connection for new client d03f26c8-3607-4d86-ab79-c8c1ba138b83 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 1825.063506] Lustre: Skipped 24 previous similar messages [ 1886.000729] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [ 1886.012171] Lustre: lustre-MDT0001: disconnecting 1 stale clients [ 1886.116316] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1137 to 0x2c0000400:1313 [ 1886.125082] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1137 to 0x280000400:1313 [ 1894.834122] Lustre: DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 11:31:21 (1776180681) [ 1904.776511] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1914.726371] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1922.469972] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776180710 with bad export cookie 17131468512288341501 [ 1922.484843] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.490073] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1930.655259] Lustre: 3347:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180711/real 1776180711] req@000000004db86f7a x1862458505214016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776180718 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1945.291745] LDISKFS-fs (dm-0): recovery complete [ 1945.295342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.297165] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873bc7dfd to 0xedbf33d873bc892c [ 1955.318059] Lustre: MGC192.168.203.145@tcp: Connection restored to (at 0@lo) [ 1955.322295] Lustre: Skipped 21 previous similar messages [ 1955.718099] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1955.733593] Lustre: Skipped 3 previous similar messages [ 1959.455882] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1981.490429] LDISKFS-fs (dm-1): recovery complete [ 1981.496924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.072652] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1988.114393] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1137 to 0x2c0000400:1345 [ 1988.117115] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1137 to 0x280000400:1345 [ 1995.299780] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2057.000258] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 2057.014561] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2057.087852] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:321 [ 2057.090684] Lustre: lustre-OST0001: deleting orphan objects from 0x0:113 to 0x0:321 [ 2068.844325] Lustre: DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 11:34:15 (1776180855) [ 2078.465856] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 2086.676336] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2088.410429] Lustre: Failing over lustre-MDT0000 [ 2088.414900] Lustre: Skipped 5 previous similar messages [ 2088.675861] Lustre: server umount lustre-MDT0000 complete [ 2088.678851] Lustre: Skipped 5 previous similar messages [ 2090.464101] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.485341] Lustre: Skipped 17 previous similar messages [ 2092.771662] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776180880 with bad export cookie 17131468512288344364 [ 2092.784035] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2115.895568] LDISKFS-fs (dm-0): recovery complete [ 2115.911126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.422549] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2151.377409] LDISKFS-fs (dm-1): recovery complete [ 2151.380372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.524178] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2151.534310] LustreError: Skipped 3 previous similar messages [ 2155.763417] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2157.369470] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2157.377139] Lustre: Skipped 5 previous similar messages [ 2157.405515] Lustre: lustre-OST0001: deleting orphan objects from 0x0:113 to 0x0:353 [ 2157.406709] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:353 [ 2165.111776] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2227.001037] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [ 2227.011199] Lustre: lustre-MDT0001: disconnecting 1 stale clients [ 2227.070298] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1137 to 0x2c0000400:1377 [ 2227.075182] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1137 to 0x280000400:1377 [ 2238.219628] Lustre: DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 11:37:04 (1776181024) [ 2247.533156] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 2255.586150] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2261.669106] LustreError: 9230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776181049 with bad export cookie 17131468512288346527 [ 2261.684783] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2261.685405] LustreError: 9230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2261.710935] LustreError: Skipped 1 previous similar message [ 2271.647143] Lustre: 3345:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181052/real 1776181052] req@000000009c77bf8e x1862458505288768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776181059 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 2271.681544] Lustre: 3345:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 2281.966346] LDISKFS-fs (dm-0): recovery complete [ 2281.971215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.241212] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xedbf33d873bc919f to 0xedbf33d873bc9a82 [ 2294.269534] Lustre: Skipped 1 previous similar message [ 2299.415920] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2313.228583] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2313.237473] Lustre: Skipped 5 previous similar messages [ 2321.110197] LDISKFS-fs (dm-1): recovery complete [ 2321.113224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.170892] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2326.673996] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1137 to 0x2c0000400:1409 [ 2326.674965] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1137 to 0x280000400:1409 [ 2326.743943] Lustre: lustre-OST0001: deleting orphan objects from 0x0:113 to 0x0:385 [ 2326.747190] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:385 [ 2332.397918] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2333.610657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2334.575268] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2340.872759] Lustre: DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 11:38:47 (1776181127) [ 2347.107315] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2352.612343] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 2357.222080] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.247346] LustreError: Skipped 118 previous similar messages [ 2357.575414] LustreError: 9230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776181145 with bad export cookie 17131468512288348802 [ 2357.596927] LustreError: 9230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2384.055045] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 2384.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 2384.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 2384.059017] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2384.059120] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2384.059125] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2384.059132] Workqueue: events free_work [libcfs] [ 2384.059177] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2384.059222] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2384.059226] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2384.059231] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2384.059234] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2384.059237] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2384.059240] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2384.059242] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2384.059246] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2384.059249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2384.059251] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2384.059259] Call Trace: [ 2384.060406] [ 2384.060416] ? show_regs.cold.9+0x22/0x2f [ 2384.060426] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2384.060429] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2384.060432] ? watchdog+0x40/0x40 [ 2384.060435] ? hrtimer_interrupt+0x128/0x2c0 [ 2384.060437] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2384.060443] ? apic_timer_interrupt+0xf/0x20 [ 2384.060447] [ 2384.060448] ? on_each_cpu+0x51/0x120 [ 2384.060451] ? smp_call_function_many_cond+0x3db/0x540 [ 2384.060454] ? smp_call_function_many_cond+0x3a8/0x540 [ 2384.060456] ? load_new_mm_cr3+0x120/0x120 [ 2384.060460] ? load_new_mm_cr3+0x120/0x120 [ 2384.060463] on_each_cpu+0x51/0x120 [ 2384.060465] flush_tlb_kernel_range+0x93/0xb0 [ 2384.060469] free_unmap_vmap_area+0x63/0x70 [ 2384.060473] remove_vm_area+0xe7/0x110 [ 2384.060475] __vunmap+0x9c/0x400 [ 2384.060478] vfree+0x4d/0x100 [ 2384.060483] free_work+0x34/0x50 [libcfs] [ 2384.060506] process_one_work+0x2c8/0x700 [ 2384.060511] worker_thread+0x296/0x6e0 [ 2384.060513] ? rescuer_thread+0x570/0x570 [ 2384.060516] kthread+0x1d1/0x200 [ 2384.060520] ? set_kthread_struct+0x70/0x70 [ 2384.060522] ret_from_fork+0x1f/0x30 [ 2384.063386] Modules linked in: zfs(O) [ 2384.066092] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2384.066183] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2384.066190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2384.066193] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2384.066205] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2384.066209] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2384.066214] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2384.066216] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2384.066219] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2384.066221] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2384.066224] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2384.066227] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2384.066230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2384.066233] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2384.066241] Call Trace: [ 2384.066245] [ 2384.066250] ? show_regs.cold.9+0x22/0x2f [ 2384.066258] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2384.066264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2384.066268] ? watchdog+0x40/0x40 [ 2384.066272] ? hrtimer_interrupt+0x128/0x2c0 [ 2384.066276] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2384.066284] ? apic_timer_interrupt+0xf/0x20 [ 2384.066288] [ 2384.066290] ? on_each_cpu_cond_mask+0x46/0x150 [ 2384.066294] ? __x64_sys_fspick+0x210/0x210 [ 2384.066301] ? smp_call_function_many_cond+0x3d9/0x540 [ 2384.066305] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2384.066309] ? __x64_sys_fspick+0x210/0x210 [ 2384.066313] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2384.066317] on_each_cpu_cond_mask+0x46/0x150 [ 2384.066321] on_each_cpu_cond+0x1c/0x30 [ 2384.066324] invalidate_bh_lrus+0x27/0x30 [ 2384.066328] invalidate_bdev+0x5a/0x90 [ 2384.066333] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2384.066457] generic_shutdown_super+0xb1/0x1b0 [ 2384.066465] kill_block_super+0x29/0x80 [ 2384.066470] deactivate_locked_super+0x52/0xd0 [ 2384.066475] deactivate_super+0x83/0x90 [ 2384.066479] cleanup_mnt+0x5f/0xe0 [ 2384.066485] __cleanup_mnt+0x16/0x20 [ 2384.066490] task_work_run+0xc6/0x110 [ 2384.066497] exit_to_usermode_loop+0x1dd/0x1f0 [ 2384.066504] do_syscall_64+0x3d6/0x440 [ 2384.066508] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2384.066513] RIP: 0033:0x7fbb019648fb [ 2384.066519] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2384.066522] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2384.066526] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2384.066528] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2384.066530] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2384.066532] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2384.066535] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2384.375122] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2384.399481] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2384.407957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2384.411781] Workqueue: events free_work [libcfs] [ 2384.413740] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2384.416239] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2384.423969] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2384.427381] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2384.430728] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2384.433313] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2384.436542] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2384.440017] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2384.443310] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2384.446985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2384.449314] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2384.452367] Call Trace: [ 2384.453412] [ 2384.454396] ? show_regs.cold.9+0x22/0x2f [ 2384.456070] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2384.458158] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2384.460499] ? watchdog+0x40/0x40 [ 2384.461937] ? hrtimer_interrupt+0x128/0x2c0 [ 2384.463310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2384.465309] ? apic_timer_interrupt+0xf/0x20 [ 2384.467084] [ 2384.468025] ? on_each_cpu+0x51/0x120 [ 2384.469452] ? smp_call_function_many_cond+0x3d9/0x540 [ 2384.471575] ? smp_call_function_many_cond+0x3a8/0x540 [ 2384.474066] ? load_new_mm_cr3+0x120/0x120 [ 2384.478166] ? load_new_mm_cr3+0x120/0x120 [ 2384.481270] on_each_cpu+0x51/0x120 [ 2384.484080] flush_tlb_kernel_range+0x93/0xb0 [ 2384.486164] free_unmap_vmap_area+0x63/0x70 [ 2384.487719] remove_vm_area+0xe7/0x110 [ 2384.490537] __vunmap+0x9c/0x400 [ 2384.492075] vfree+0x4d/0x100 [ 2384.493012] free_work+0x34/0x50 [libcfs] [ 2384.494533] process_one_work+0x2c8/0x700 [ 2384.496236] worker_thread+0x296/0x6e0 [ 2384.498503] ? rescuer_thread+0x570/0x570 [ 2384.499798] kthread+0x1d1/0x200 [ 2384.500853] ? set_kthread_struct+0x70/0x70 [ 2384.502628] ret_from_fork+0x1f/0x30 [ 2390.495221] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 32s! [ 2390.505779] Showing busy workqueues and worker pools: [ 2390.516174] workqueue events: flags=0x0 [ 2390.521104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2390.524052] in-flight: 56293:free_work [libcfs] [ 2390.527253] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2390.529957] in-flight: 22277:free_work [libcfs] [ 2390.532565] pending: kfree_rcu_monitor, key_garbage_collector [ 2390.535135] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2390.538912] pending: nfs4_renew_state [nfsv4] [ 2390.541667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2390.551822] in-flight: 47596:free_work [libcfs] [ 2390.553554] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2390.569296] workqueue events_unbound: flags=0x2 [ 2390.574356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2390.581983] pending: flush_memcg_stats_dwork [ 2390.584888] workqueue events_power_efficient: flags=0x80 [ 2390.592098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2390.598577] pending: neigh_periodic_work, do_cache_clean [ 2390.604315] workqueue mm_percpu_wq: flags=0x8 [ 2390.609278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2390.612455] pending: vmstat_update [ 2390.614597] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2390.620405] pending: vmstat_update [ 2390.626509] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2390.637943] pending: vmstat_update [ 2390.640151] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2390.645289] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2390.650598] pending: cgroup_pidlist_destroy_work_fn [ 2390.655048] workqueue writeback: flags=0x4a [ 2390.659263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2390.666152] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2390.672461] workqueue knbd0-recv: flags=0x1a [ 2390.676463] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2390.681638] in-flight: 655:recv_work [ 2390.683039] workqueue ptlrpc_pinger: flags=0x2 [ 2390.690790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2390.694614] pending: ptlrpc_pinger_main [ptlrpc] [ 2390.700340] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=32s workers=5 idle: 53684 27131 54850 46584 [ 2390.710470] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=32s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2390.718369] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=32s workers=6 idle: 53660 37674 46417 53664 46418 [ 2390.727743] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2380s workers=2 idle: 657 [ 2412.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2412.059007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2412.061184] Modules linked in: zfs(O) [ 2412.068535] Modules linked in: [ 2412.068539] spl(O) [ 2412.069913] zfs(O) [ 2412.071185] lustre(O) [ 2412.072026] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 [ 2412.072854] osp(O) [ 2412.074659] auth_rpcgss [ 2412.083018] ofd(O) [ 2412.084956] nfsv4 [ 2412.086633] lod(O) [ 2412.088090] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2412.088124] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2412.088131] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2412.088137] Workqueue: events free_work [libcfs] [ 2412.088178] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2412.088187] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2412.088191] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2412.088195] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2412.088197] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2412.088199] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2412.088201] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2412.088203] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2412.088206] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2412.088208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2412.088210] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2412.088218] Call Trace: [ 2412.088224] [ 2412.088229] ? show_regs.cold.9+0x22/0x2f [ 2412.089753] ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2412.089829] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2412.089836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2412.089839] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2412.089849] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2412.089853] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2412.089858] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2412.089860] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2412.089862] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2412.089864] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2412.089867] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2412.089869] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2412.091544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2412.104756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2412.104765] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2412.104774] Call Trace: [ 2412.104780] [ 2412.104786] ? show_regs.cold.9+0x22/0x2f [ 2412.104799] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2412.104805] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2412.104809] ? watchdog+0x40/0x40 [ 2412.104814] ? hrtimer_interrupt+0x128/0x2c0 [ 2412.110343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2412.110361] ? watchdog+0x40/0x40 [ 2412.110368] ? hrtimer_interrupt+0x128/0x2c0 [ 2412.110373] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2412.110381] ? apic_timer_interrupt+0xf/0x20 [ 2412.110387] [ 2412.110391] ? on_each_cpu+0x51/0x120 [ 2412.110396] ? smp_call_function_many_cond+0x3db/0x540 [ 2412.110400] ? smp_call_function_many_cond+0x3a8/0x540 [ 2412.110403] ? load_new_mm_cr3+0x120/0x120 [ 2412.110409] ? load_new_mm_cr3+0x120/0x120 [ 2412.110412] on_each_cpu+0x51/0x120 [ 2412.110417] flush_tlb_kernel_range+0x93/0xb0 [ 2412.110422] free_unmap_vmap_area+0x63/0x70 [ 2412.110427] remove_vm_area+0xe7/0x110 [ 2412.110431] __vunmap+0x9c/0x400 [ 2412.110434] vfree+0x4d/0x100 [ 2412.110441] free_work+0x34/0x50 [libcfs] [ 2412.110478] process_one_work+0x2c8/0x700 [ 2412.110486] worker_thread+0x296/0x6e0 [ 2412.110490] ? rescuer_thread+0x570/0x570 [ 2412.110494] kthread+0x1d1/0x200 [ 2412.110498] ? set_kthread_struct+0x70/0x70 [ 2412.116827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2412.116847] ? apic_timer_interrupt+0xf/0x20 [ 2412.116852] [ 2412.116856] ? on_each_cpu_cond_mask+0x46/0x150 [ 2412.116861] ? __x64_sys_fspick+0x210/0x210 [ 2412.116869] ? smp_call_function_many_cond+0x3db/0x540 [ 2412.116874] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2412.116877] ? __x64_sys_fspick+0x210/0x210 [ 2412.116882] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2412.116885] on_each_cpu_cond_mask+0x46/0x150 [ 2412.116891] on_each_cpu_cond+0x1c/0x30 [ 2412.116894] invalidate_bh_lrus+0x27/0x30 [ 2412.116898] invalidate_bdev+0x5a/0x90 [ 2412.116906] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2412.119721] ret_from_fork+0x1f/0x30 [ 2412.122678] generic_shutdown_super+0xb1/0x1b0 [ 2412.122700] kill_block_super+0x29/0x80 [ 2412.122706] deactivate_locked_super+0x52/0xd0 [ 2412.122711] deactivate_super+0x83/0x90 [ 2412.122716] cleanup_mnt+0x5f/0xe0 [ 2412.122721] __cleanup_mnt+0x16/0x20 [ 2412.122724] task_work_run+0xc6/0x110 [ 2412.122733] exit_to_usermode_loop+0x1dd/0x1f0 [ 2412.420394] do_syscall_64+0x3d6/0x440 [ 2412.425817] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2412.428858] RIP: 0033:0x7fbb019648fb [ 2412.435351] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2412.447925] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2412.453014] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2412.456937] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2412.463638] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2412.466940] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2412.474413] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2416.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 2416.059228] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2416.094361] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2416.110920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2416.130051] Workqueue: events free_work [libcfs] [ 2416.134217] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2416.142954] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2416.157254] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2416.162640] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2416.168089] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2416.174570] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2416.179175] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2416.188951] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2416.194605] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2416.208046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2416.216236] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2416.223241] Call Trace: [ 2416.223913] [ 2416.224386] ? show_regs.cold.9+0x22/0x2f [ 2416.226765] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2416.228748] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2416.240390] ? watchdog+0x40/0x40 [ 2416.248210] ? hrtimer_interrupt+0x128/0x2c0 [ 2416.250807] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2416.252679] ? apic_timer_interrupt+0xf/0x20 [ 2416.264811] [ 2416.269240] ? on_each_cpu+0x51/0x120 [ 2416.272029] ? smp_call_function_many_cond+0x3d9/0x540 [ 2416.276201] ? smp_call_function_many_cond+0x3a8/0x540 [ 2416.280704] ? load_new_mm_cr3+0x120/0x120 [ 2416.286629] ? load_new_mm_cr3+0x120/0x120 [ 2416.290217] on_each_cpu+0x51/0x120 [ 2416.291620] flush_tlb_kernel_range+0x93/0xb0 [ 2416.293543] free_unmap_vmap_area+0x63/0x70 [ 2416.294684] remove_vm_area+0xe7/0x110 [ 2416.297529] __vunmap+0x9c/0x400 [ 2416.299251] vfree+0x4d/0x100 [ 2416.300920] free_work+0x34/0x50 [libcfs] [ 2416.303053] process_one_work+0x2c8/0x700 [ 2416.306462] worker_thread+0x296/0x6e0 [ 2416.309152] ? rescuer_thread+0x570/0x570 [ 2416.311677] kthread+0x1d1/0x200 [ 2416.312562] ? set_kthread_struct+0x70/0x70 [ 2416.313629] ret_from_fork+0x1f/0x30 [ 2417.732014] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 2417.736653] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 2417.745629] (detected by 2, t=60002 jiffies, g=626849, q=1398) [ 2417.751492] Sending NMI from CPU 2 to CPUs 3: [ 2417.754334] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 2417.754343] rcu: rcu_sched kthread starved for 59644 jiffies! g626849 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2417.772374] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2417.783555] rcu: RCU grace-period kthread stack dump: [ 2417.788392] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2417.798713] Call Trace: [ 2417.800395] __schedule+0x351/0xcb0 [ 2417.803989] schedule+0xc0/0x180 [ 2417.807692] schedule_timeout+0xb4/0x190 [ 2417.814962] ? do_raw_spin_unlock+0x75/0x190 [ 2417.818446] ? __next_timer_interrupt+0x160/0x160 [ 2417.824398] ? prepare_to_swait_event+0xf3/0x140 [ 2417.830487] rcu_gp_kthread+0x8dd/0x15b0 [ 2417.836304] ? force_qs_rnp+0x2c0/0x2c0 [ 2417.840096] kthread+0x1d1/0x200 [ 2417.848886] ? set_kthread_struct+0x70/0x70 [ 2417.857013] ret_from_fork+0x1f/0x30 [ 2417.859541] rcu: Stack dump where RCU GP kthread last ran: [ 2417.864400] Sending NMI from CPU 2 to CPUs 0: [ 2417.866914] NMI backtrace for cpu 0 [ 2417.866919] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2417.866922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2417.866923] Workqueue: events free_work [libcfs] [ 2417.866926] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2417.866928] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2417.866930] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 [ 2417.866931] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2417.866933] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2417.866934] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2417.866935] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2417.866937] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2417.866938] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2417.866939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2417.866940] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2417.866941] Call Trace: [ 2417.866942] [ 2417.866943] ? show_regs.cold.9+0x22/0x2f [ 2417.866944] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2417.866945] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2417.866946] ? nmi_handle+0xbd/0x210 [ 2417.866946] ? default_do_nmi+0x4b/0x1b0 [ 2417.866947] ? do_nmi+0x280/0x380 [ 2417.866948] ? end_repeat_nmi+0x16/0x1f [ 2417.866949] ? on_each_cpu+0x51/0x120 [ 2417.866950] ? smp_call_function_many_cond+0x3db/0x540 [ 2417.866951] ? smp_call_function_many_cond+0x3db/0x540 [ 2417.866952] ? smp_call_function_many_cond+0x3db/0x540 [ 2417.866953] [ 2417.866953] ? load_new_mm_cr3+0x120/0x120 [ 2417.866954] ? load_new_mm_cr3+0x120/0x120 [ 2417.866955] on_each_cpu+0x51/0x120 [ 2417.866956] flush_tlb_kernel_range+0x93/0xb0 [ 2417.866957] free_unmap_vmap_area+0x63/0x70 [ 2417.866957] remove_vm_area+0xe7/0x110 [ 2417.866958] __vunmap+0x9c/0x400 [ 2417.866959] vfree+0x4d/0x100 [ 2417.866960] free_work+0x34/0x50 [libcfs] [ 2417.866961] process_one_work+0x2c8/0x700 [ 2417.866961] worker_thread+0x296/0x6e0 [ 2417.866962] ? rescuer_thread+0x570/0x570 [ 2417.866963] kthread+0x1d1/0x200 [ 2417.866964] ? set_kthread_struct+0x70/0x70 [ 2417.866965] ret_from_fork+0x1f/0x30 [ 2421.215030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 62s! [ 2421.223551] Showing busy workqueues and worker pools: [ 2421.229575] workqueue events: flags=0x0 [ 2421.232224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2421.238445] in-flight: 56293:free_work [libcfs] [ 2421.249694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2421.256400] in-flight: 22277:free_work [libcfs] [ 2421.264221] pending: kfree_rcu_monitor, key_garbage_collector [ 2421.270406] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2421.277702] pending: nfs4_renew_state [nfsv4] [ 2421.288623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2421.292727] in-flight: 47596:free_work [libcfs] [ 2421.299356] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2421.307445] workqueue events_unbound: flags=0x2 [ 2421.317964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2421.326283] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2421.337668] workqueue events_power_efficient: flags=0x80 [ 2421.343770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2421.348175] pending: neigh_periodic_work, do_cache_clean [ 2421.355337] workqueue mm_percpu_wq: flags=0x8 [ 2421.356919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2421.378596] pending: vmstat_update [ 2421.383255] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2421.397542] pending: vmstat_update [ 2421.402191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2421.408820] pending: vmstat_update [ 2421.412501] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2421.420401] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2421.422976] pending: cgroup_pidlist_destroy_work_fn [ 2421.426258] workqueue writeback: flags=0x4a [ 2421.429391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2421.438438] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2421.447336] workqueue knbd0-recv: flags=0x1a [ 2421.451311] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2421.453010] in-flight: 655:recv_work [ 2421.453960] workqueue ptlrpc_pinger: flags=0x2 [ 2421.456767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2421.460193] pending: ptlrpc_pinger_main [ptlrpc] [ 2421.463437] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=63s workers=5 idle: 53684 27131 54850 46584 [ 2421.472355] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=63s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2421.479546] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=63s workers=6 idle: 53660 37674 46417 53664 46418 [ 2421.488800] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2411s workers=2 idle: 657 [ 2440.059373] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2440.068930] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2440.112342] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2440.118298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2440.123057] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2440.124840] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2440.137995] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2440.142604] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2440.145792] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2440.148682] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2440.151637] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2440.154882] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2440.158955] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2440.166800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2440.170238] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2440.174681] Call Trace: [ 2440.176330] [ 2440.177284] ? show_regs.cold.9+0x22/0x2f [ 2440.179096] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2440.181536] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2440.183325] ? watchdog+0x40/0x40 [ 2440.184952] ? hrtimer_interrupt+0x128/0x2c0 [ 2440.186647] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2440.188252] ? apic_timer_interrupt+0xf/0x20 [ 2440.190360] [ 2440.190958] ? on_each_cpu_cond_mask+0x46/0x150 [ 2440.192945] ? __x64_sys_fspick+0x210/0x210 [ 2440.194669] ? smp_call_function_many_cond+0x3db/0x540 [ 2440.196635] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2440.198090] ? __x64_sys_fspick+0x210/0x210 [ 2440.199913] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2440.201740] on_each_cpu_cond_mask+0x46/0x150 [ 2440.203129] on_each_cpu_cond+0x1c/0x30 [ 2440.205322] invalidate_bh_lrus+0x27/0x30 [ 2440.211933] invalidate_bdev+0x5a/0x90 [ 2440.212868] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2440.214931] generic_shutdown_super+0xb1/0x1b0 [ 2440.217086] kill_block_super+0x29/0x80 [ 2440.219050] deactivate_locked_super+0x52/0xd0 [ 2440.220781] deactivate_super+0x83/0x90 [ 2440.222315] cleanup_mnt+0x5f/0xe0 [ 2440.223823] __cleanup_mnt+0x16/0x20 [ 2440.228538] task_work_run+0xc6/0x110 [ 2440.229563] exit_to_usermode_loop+0x1dd/0x1f0 [ 2440.231104] do_syscall_64+0x3d6/0x440 [ 2440.233768] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2440.237772] RIP: 0033:0x7fbb019648fb [ 2440.239361] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2440.245328] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2440.247759] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2440.249920] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2440.252326] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2440.254298] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2440.256966] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2444.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 2444.062071] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2444.081729] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2444.086076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2444.093598] Workqueue: events free_work [libcfs] [ 2444.095315] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2444.101802] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2444.114058] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2444.120687] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2444.127257] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2444.131929] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2444.137354] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2444.140102] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2444.143300] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2444.146496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2444.148713] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2444.151561] Call Trace: [ 2444.152627] [ 2444.153525] ? show_regs.cold.9+0x22/0x2f [ 2444.155229] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2444.157227] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2444.159057] ? watchdog+0x40/0x40 [ 2444.159998] ? hrtimer_interrupt+0x128/0x2c0 [ 2444.161109] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2444.162799] ? apic_timer_interrupt+0xf/0x20 [ 2444.163887] [ 2444.164563] ? on_each_cpu+0x51/0x120 [ 2444.165576] ? smp_call_function_many_cond+0x3db/0x540 [ 2444.167608] ? smp_call_function_many_cond+0x3a8/0x540 [ 2444.169811] ? load_new_mm_cr3+0x120/0x120 [ 2444.171612] ? load_new_mm_cr3+0x120/0x120 [ 2444.173419] on_each_cpu+0x51/0x120 [ 2444.174523] flush_tlb_kernel_range+0x93/0xb0 [ 2444.175912] free_unmap_vmap_area+0x63/0x70 [ 2444.177066] remove_vm_area+0xe7/0x110 [ 2444.178114] __vunmap+0x9c/0x400 [ 2444.179444] vfree+0x4d/0x100 [ 2444.180155] free_work+0x34/0x50 [libcfs] [ 2444.181806] process_one_work+0x2c8/0x700 [ 2444.183504] worker_thread+0x296/0x6e0 [ 2444.185038] ? rescuer_thread+0x570/0x570 [ 2444.186782] kthread+0x1d1/0x200 [ 2444.188109] ? set_kthread_struct+0x70/0x70 [ 2444.189819] ret_from_fork+0x1f/0x30 [ 2448.055172] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2448.062767] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2448.092667] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2448.097003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2448.105769] Workqueue: events free_work [libcfs] [ 2448.111948] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2448.117475] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2448.125662] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2448.130590] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2448.133501] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2448.136370] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2448.139218] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2448.142203] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2448.145063] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2448.148380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2448.150456] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2448.153043] Call Trace: [ 2448.154065] [ 2448.154958] ? show_regs.cold.9+0x22/0x2f [ 2448.156642] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2448.158018] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2448.160076] ? watchdog+0x40/0x40 [ 2448.161669] ? hrtimer_interrupt+0x128/0x2c0 [ 2448.163252] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2448.165125] ? apic_timer_interrupt+0xf/0x20 [ 2448.167089] [ 2448.168059] ? on_each_cpu+0x51/0x120 [ 2448.169728] ? smp_call_function_many_cond+0x3db/0x540 [ 2448.171715] ? smp_call_function_many_cond+0x3a8/0x540 [ 2448.174535] ? load_new_mm_cr3+0x120/0x120 [ 2448.176160] ? load_new_mm_cr3+0x120/0x120 [ 2448.179530] on_each_cpu+0x51/0x120 [ 2448.181142] flush_tlb_kernel_range+0x93/0xb0 [ 2448.183184] free_unmap_vmap_area+0x63/0x70 [ 2448.184678] remove_vm_area+0xe7/0x110 [ 2448.187814] __vunmap+0x9c/0x400 [ 2448.190204] vfree+0x4d/0x100 [ 2448.192626] free_work+0x34/0x50 [libcfs] [ 2448.195754] process_one_work+0x2c8/0x700 [ 2448.198130] worker_thread+0x296/0x6e0 [ 2448.204278] ? rescuer_thread+0x570/0x570 [ 2448.207533] kthread+0x1d1/0x200 [ 2448.209490] ? set_kthread_struct+0x70/0x70 [ 2448.212087] ret_from_fork+0x1f/0x30 [ 2451.935034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 93s! [ 2451.938080] Showing busy workqueues and worker pools: [ 2451.939959] workqueue events: flags=0x0 [ 2451.941475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2451.945178] in-flight: 56293:free_work [libcfs] [ 2451.950517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2451.953655] in-flight: 22277:free_work [libcfs] [ 2451.955801] pending: kfree_rcu_monitor, key_garbage_collector [ 2451.958564] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2451.961606] pending: nfs4_renew_state [nfsv4] [ 2451.963762] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2451.966882] in-flight: 47596:free_work [libcfs] [ 2451.969089] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2451.973966] workqueue events_unbound: flags=0x2 [ 2451.975683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2451.977811] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2451.980419] workqueue events_power_efficient: flags=0x80 [ 2451.983085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2451.986418] pending: neigh_periodic_work, do_cache_clean [ 2451.989595] workqueue mm_percpu_wq: flags=0x8 [ 2451.991701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2451.994723] pending: vmstat_update [ 2451.996531] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2451.999778] pending: vmstat_update [ 2452.001597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2452.004316] pending: vmstat_update [ 2452.006018] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2452.008577] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2452.011971] pending: cgroup_pidlist_destroy_work_fn [ 2452.014666] workqueue writeback: flags=0x4a [ 2452.016689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2452.019630] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2452.025378] workqueue knbd0-recv: flags=0x1a [ 2452.027170] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2452.041627] in-flight: 655:recv_work [ 2452.044029] workqueue ptlrpc_pinger: flags=0x2 [ 2452.047079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2452.053757] pending: ptlrpc_pinger_main [ptlrpc] [ 2452.059470] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=94s workers=5 idle: 53684 27131 54850 46584 [ 2452.066407] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=94s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2452.070982] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=94s workers=6 idle: 53660 37674 46417 53664 46418 [ 2452.075474] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2441s workers=2 idle: 657 [ 2468.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2468.065961] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2468.150675] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2468.160675] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2468.169957] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2468.174626] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2468.184544] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2468.188408] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2468.191556] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2468.195083] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2468.198331] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2468.201539] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2468.208807] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2468.214768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2468.218371] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2468.224310] Call Trace: [ 2468.226625] [ 2468.228634] ? show_regs.cold.9+0x22/0x2f [ 2468.233349] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2468.236763] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2468.239295] ? watchdog+0x40/0x40 [ 2468.241057] ? hrtimer_interrupt+0x128/0x2c0 [ 2468.243324] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2468.245729] ? apic_timer_interrupt+0xf/0x20 [ 2468.248354] [ 2468.250118] ? on_each_cpu_cond_mask+0x46/0x150 [ 2468.252420] ? __x64_sys_fspick+0x210/0x210 [ 2468.255313] ? smp_call_function_many_cond+0x3db/0x540 [ 2468.260461] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2468.264082] ? __x64_sys_fspick+0x210/0x210 [ 2468.266250] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2468.268674] on_each_cpu_cond_mask+0x46/0x150 [ 2468.271373] on_each_cpu_cond+0x1c/0x30 [ 2468.274314] invalidate_bh_lrus+0x27/0x30 [ 2468.276705] invalidate_bdev+0x5a/0x90 [ 2468.278436] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2468.282556] generic_shutdown_super+0xb1/0x1b0 [ 2468.285935] kill_block_super+0x29/0x80 [ 2468.287263] deactivate_locked_super+0x52/0xd0 [ 2468.290161] deactivate_super+0x83/0x90 [ 2468.292334] cleanup_mnt+0x5f/0xe0 [ 2468.294365] __cleanup_mnt+0x16/0x20 [ 2468.296660] task_work_run+0xc6/0x110 [ 2468.299380] exit_to_usermode_loop+0x1dd/0x1f0 [ 2468.304775] do_syscall_64+0x3d6/0x440 [ 2468.309267] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2468.314727] RIP: 0033:0x7fbb019648fb [ 2468.321258] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2468.337189] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2468.347071] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2468.350404] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2468.358108] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2468.364486] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2468.369820] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2472.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 2472.062228] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2472.092399] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2472.098644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2472.105469] Workqueue: events free_work [libcfs] [ 2472.107823] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2472.112411] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2472.122537] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2472.126252] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2472.129543] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2472.134048] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2472.143114] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2472.145688] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2472.153766] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2472.161342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2472.168180] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2472.173972] Call Trace: [ 2472.175690] [ 2472.176695] ? show_regs.cold.9+0x22/0x2f [ 2472.178691] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2472.180665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2472.182858] ? watchdog+0x40/0x40 [ 2472.186458] ? hrtimer_interrupt+0x128/0x2c0 [ 2472.189332] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2472.195481] ? apic_timer_interrupt+0xf/0x20 [ 2472.202500] [ 2472.203595] ? on_each_cpu+0x51/0x120 [ 2472.206012] ? smp_call_function_many_cond+0x3db/0x540 [ 2472.209814] ? smp_call_function_many_cond+0x3a8/0x540 [ 2472.212841] ? load_new_mm_cr3+0x120/0x120 [ 2472.215820] ? load_new_mm_cr3+0x120/0x120 [ 2472.218831] on_each_cpu+0x51/0x120 [ 2472.221430] flush_tlb_kernel_range+0x93/0xb0 [ 2472.227649] free_unmap_vmap_area+0x63/0x70 [ 2472.230480] remove_vm_area+0xe7/0x110 [ 2472.233690] __vunmap+0x9c/0x400 [ 2472.238663] vfree+0x4d/0x100 [ 2472.240761] free_work+0x34/0x50 [libcfs] [ 2472.243421] process_one_work+0x2c8/0x700 [ 2472.246816] worker_thread+0x296/0x6e0 [ 2472.249205] ? rescuer_thread+0x570/0x570 [ 2472.251564] kthread+0x1d1/0x200 [ 2472.253889] ? set_kthread_struct+0x70/0x70 [ 2472.257306] ret_from_fork+0x1f/0x30 [ 2480.055009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2480.059219] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2480.110470] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2480.126088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2480.135191] Workqueue: events free_work [libcfs] [ 2480.136876] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2480.145908] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2480.163126] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2480.169786] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2480.176145] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2480.180469] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2480.188851] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2480.194789] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2480.200960] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2480.207189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2480.214217] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2480.220399] Call Trace: [ 2480.221223] [ 2480.221827] ? show_regs.cold.9+0x22/0x2f [ 2480.224828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2480.228880] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2480.232794] ? watchdog+0x40/0x40 [ 2480.235856] ? hrtimer_interrupt+0x128/0x2c0 [ 2480.238944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2480.241347] ? apic_timer_interrupt+0xf/0x20 [ 2480.256117] [ 2480.258815] ? on_each_cpu+0x51/0x120 [ 2480.262086] ? smp_call_function_many_cond+0x3d9/0x540 [ 2480.265819] ? smp_call_function_many_cond+0x3a8/0x540 [ 2480.269851] ? load_new_mm_cr3+0x120/0x120 [ 2480.271899] ? load_new_mm_cr3+0x120/0x120 [ 2480.275184] on_each_cpu+0x51/0x120 [ 2480.278463] flush_tlb_kernel_range+0x93/0xb0 [ 2480.280893] free_unmap_vmap_area+0x63/0x70 [ 2480.282268] remove_vm_area+0xe7/0x110 [ 2480.283570] __vunmap+0x9c/0x400 [ 2480.286080] vfree+0x4d/0x100 [ 2480.288055] free_work+0x34/0x50 [libcfs] [ 2480.290803] process_one_work+0x2c8/0x700 [ 2480.293744] worker_thread+0x296/0x6e0 [ 2480.296793] ? rescuer_thread+0x570/0x570 [ 2480.300119] kthread+0x1d1/0x200 [ 2480.302539] ? set_kthread_struct+0x70/0x70 [ 2480.305744] ret_from_fork+0x1f/0x30 [ 2482.143375] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 123s! [ 2482.151635] Showing busy workqueues and worker pools: [ 2482.155221] workqueue events: flags=0x0 [ 2482.165010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2482.174062] in-flight: 56293:free_work [libcfs] [ 2482.178847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2482.185449] in-flight: 22277:free_work [libcfs] [ 2482.189108] pending: kfree_rcu_monitor, key_garbage_collector [ 2482.199341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2482.206030] pending: nfs4_renew_state [nfsv4] [ 2482.209146] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2482.213836] in-flight: 47596:free_work [libcfs] [ 2482.217700] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2482.231260] workqueue events_unbound: flags=0x2 [ 2482.238125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2482.249004] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2482.256454] workqueue events_power_efficient: flags=0x80 [ 2482.260125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2482.269188] pending: neigh_periodic_work, do_cache_clean [ 2482.284108] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2482.289939] pending: check_lifetime [ 2482.296094] workqueue mm_percpu_wq: flags=0x8 [ 2482.301258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2482.317008] pending: vmstat_update [ 2482.319903] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2482.329995] pending: vmstat_update [ 2482.331454] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2482.336162] pending: vmstat_update [ 2482.338928] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2482.343185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2482.347783] pending: cgroup_pidlist_destroy_work_fn [ 2482.354144] workqueue writeback: flags=0x4a [ 2482.355765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2482.362128] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2482.368031] workqueue ipv6_addrconf: flags=0x40008 [ 2482.376554] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2482.387071] pending: addrconf_verify_work [ 2482.390738] workqueue knbd0-recv: flags=0x1a [ 2482.393236] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2482.400897] in-flight: 655:recv_work [ 2482.403447] workqueue ptlrpc_pinger: flags=0x2 [ 2482.405885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2482.410104] pending: ptlrpc_pinger_main [ptlrpc] [ 2482.415162] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=124s workers=5 idle: 53684 27131 54850 46584 [ 2482.423619] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=124s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2482.433784] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=124s workers=6 idle: 53660 37674 46417 53664 46418 [ 2482.447917] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2472s workers=2 idle: 657 [ 2496.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2496.062260] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2496.087514] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2496.092279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2496.097270] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2496.101076] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2496.113869] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2496.118272] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2496.123730] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2496.128603] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2496.132835] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2496.137067] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2496.141120] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2496.146355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2496.151760] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2496.156745] Call Trace: [ 2496.158537] [ 2496.159853] ? show_regs.cold.9+0x22/0x2f [ 2496.162790] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2496.165916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2496.169076] ? watchdog+0x40/0x40 [ 2496.172133] ? hrtimer_interrupt+0x128/0x2c0 [ 2496.175269] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2496.177930] ? apic_timer_interrupt+0xf/0x20 [ 2496.181353] [ 2496.182053] ? on_each_cpu_cond_mask+0x46/0x150 [ 2496.184301] ? __x64_sys_fspick+0x210/0x210 [ 2496.186387] ? smp_call_function_many_cond+0x3d9/0x540 [ 2496.188022] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2496.189512] ? __x64_sys_fspick+0x210/0x210 [ 2496.190855] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2496.193381] on_each_cpu_cond_mask+0x46/0x150 [ 2496.194631] on_each_cpu_cond+0x1c/0x30 [ 2496.195811] invalidate_bh_lrus+0x27/0x30 [ 2496.196981] invalidate_bdev+0x5a/0x90 [ 2496.198822] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2496.202742] generic_shutdown_super+0xb1/0x1b0 [ 2496.203999] kill_block_super+0x29/0x80 [ 2496.204868] deactivate_locked_super+0x52/0xd0 [ 2496.207543] deactivate_super+0x83/0x90 [ 2496.209901] cleanup_mnt+0x5f/0xe0 [ 2496.211048] __cleanup_mnt+0x16/0x20 [ 2496.213371] task_work_run+0xc6/0x110 [ 2496.215305] exit_to_usermode_loop+0x1dd/0x1f0 [ 2496.217007] do_syscall_64+0x3d6/0x440 [ 2496.218652] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2496.220881] RIP: 0033:0x7fbb019648fb [ 2496.222630] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2496.236061] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2496.242471] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2496.249611] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2496.254622] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2496.260180] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2496.264830] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2500.059405] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2500.066471] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2500.104391] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2500.113310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2500.117624] Workqueue: events free_work [libcfs] [ 2500.120689] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2500.124689] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2500.136006] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2500.141040] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2500.145646] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2500.153726] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2500.160403] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2500.164793] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2500.170324] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2500.175482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2500.179390] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2500.183314] Call Trace: [ 2500.184675] [ 2500.185862] ? show_regs.cold.9+0x22/0x2f [ 2500.187545] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2500.189834] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2500.192239] ? watchdog+0x40/0x40 [ 2500.194069] ? hrtimer_interrupt+0x128/0x2c0 [ 2500.196078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2500.198327] ? apic_timer_interrupt+0xf/0x20 [ 2500.200322] [ 2500.201469] ? on_each_cpu+0x51/0x120 [ 2500.203230] ? smp_call_function_many_cond+0x3d9/0x540 [ 2500.205487] ? smp_call_function_many_cond+0x3a8/0x540 [ 2500.207731] ? load_new_mm_cr3+0x120/0x120 [ 2500.209453] ? load_new_mm_cr3+0x120/0x120 [ 2500.211034] on_each_cpu+0x51/0x120 [ 2500.212591] flush_tlb_kernel_range+0x93/0xb0 [ 2500.214479] free_unmap_vmap_area+0x63/0x70 [ 2500.216527] remove_vm_area+0xe7/0x110 [ 2500.217852] __vunmap+0x9c/0x400 [ 2500.219065] vfree+0x4d/0x100 [ 2500.220394] free_work+0x34/0x50 [libcfs] [ 2500.221966] process_one_work+0x2c8/0x700 [ 2500.223556] worker_thread+0x296/0x6e0 [ 2500.225987] ? rescuer_thread+0x570/0x570 [ 2500.229898] kthread+0x1d1/0x200 [ 2500.231746] ? set_kthread_struct+0x70/0x70 [ 2500.233861] ret_from_fork+0x1f/0x30 [ 2508.055023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2508.060595] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2508.103695] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2508.109569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2508.113209] Workqueue: events free_work [libcfs] [ 2508.114940] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2508.117694] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2508.128900] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2508.132189] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2508.135261] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2508.138342] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2508.142575] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2508.147742] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2508.151863] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2508.163707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2508.169943] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2508.175889] Call Trace: [ 2508.177555] [ 2508.179283] ? show_regs.cold.9+0x22/0x2f [ 2508.181327] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2508.183275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2508.185685] ? watchdog+0x40/0x40 [ 2508.187365] ? hrtimer_interrupt+0x128/0x2c0 [ 2508.190914] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2508.192845] ? apic_timer_interrupt+0xf/0x20 [ 2508.196207] [ 2508.197598] ? on_each_cpu+0x51/0x120 [ 2508.199507] ? smp_call_function_many_cond+0x3db/0x540 [ 2508.202250] ? smp_call_function_many_cond+0x3a8/0x540 [ 2508.205340] ? load_new_mm_cr3+0x120/0x120 [ 2508.209327] ? load_new_mm_cr3+0x120/0x120 [ 2508.212119] on_each_cpu+0x51/0x120 [ 2508.213742] flush_tlb_kernel_range+0x93/0xb0 [ 2508.216307] free_unmap_vmap_area+0x63/0x70 [ 2508.218247] remove_vm_area+0xe7/0x110 [ 2508.219650] __vunmap+0x9c/0x400 [ 2508.222351] vfree+0x4d/0x100 [ 2508.223809] free_work+0x34/0x50 [libcfs] [ 2508.227444] process_one_work+0x2c8/0x700 [ 2508.229039] worker_thread+0x296/0x6e0 [ 2508.231279] ? rescuer_thread+0x570/0x570 [ 2508.233255] kthread+0x1d1/0x200 [ 2508.235669] ? set_kthread_struct+0x70/0x70 [ 2508.239392] ret_from_fork+0x1f/0x30 [ 2512.863027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 154s! [ 2512.866156] Showing busy workqueues and worker pools: [ 2512.868013] workqueue events: flags=0x0 [ 2512.869594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2512.871645] in-flight: 56293:free_work [libcfs] [ 2512.873342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2512.877649] in-flight: 22277:free_work [libcfs] [ 2512.880255] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2512.884080] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2512.885862] pending: nfs4_renew_state [nfsv4] [ 2512.888863] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2512.891770] in-flight: 47596:free_work [libcfs] [ 2512.896689] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2512.900348] workqueue events_unbound: flags=0x2 [ 2512.901918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2512.904350] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2512.907039] workqueue events_power_efficient: flags=0x80 [ 2512.908952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2512.914159] pending: neigh_periodic_work, do_cache_clean [ 2512.918336] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2512.921827] pending: check_lifetime [ 2512.925336] workqueue mm_percpu_wq: flags=0x8 [ 2512.927651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2512.933622] pending: vmstat_update [ 2512.934658] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2512.940207] pending: vmstat_update [ 2512.945452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2512.950314] pending: vmstat_update [ 2512.952405] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2512.956157] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2512.962149] pending: cgroup_pidlist_destroy_work_fn [ 2512.963506] workqueue writeback: flags=0x4a [ 2512.967232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2512.972089] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2512.978021] workqueue ipv6_addrconf: flags=0x40008 [ 2512.980794] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2512.987082] pending: addrconf_verify_work [ 2512.990339] workqueue knbd0-recv: flags=0x1a [ 2512.995406] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2513.000376] in-flight: 655:recv_work [ 2513.002499] workqueue ptlrpc_pinger: flags=0x2 [ 2513.008038] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2513.010448] pending: ptlrpc_pinger_main [ptlrpc] [ 2513.017084] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=155s workers=5 idle: 53684 27131 54850 46584 [ 2513.021373] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=155s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2513.032091] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=155s workers=6 idle: 53660 37674 46417 53664 46418 [ 2513.037010] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2502s workers=2 idle: 657 [ 2524.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2524.063681] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2524.095143] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2524.101529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2524.106893] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2524.113735] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2524.125120] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2524.129235] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2524.132896] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2524.138983] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2524.146561] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2524.153416] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2524.157837] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2524.161534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2524.164144] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2524.167555] Call Trace: [ 2524.168957] [ 2524.169902] ? show_regs.cold.9+0x22/0x2f [ 2524.171245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2524.173942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2524.176106] ? watchdog+0x40/0x40 [ 2524.178515] ? hrtimer_interrupt+0x128/0x2c0 [ 2524.181872] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2524.184961] ? apic_timer_interrupt+0xf/0x20 [ 2524.188024] [ 2524.189308] ? on_each_cpu_cond_mask+0x46/0x150 [ 2524.191991] ? __x64_sys_fspick+0x210/0x210 [ 2524.194119] ? smp_call_function_many_cond+0x3d9/0x540 [ 2524.196584] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2524.198335] ? __x64_sys_fspick+0x210/0x210 [ 2524.203360] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2524.208074] on_each_cpu_cond_mask+0x46/0x150 [ 2524.210501] on_each_cpu_cond+0x1c/0x30 [ 2524.212589] invalidate_bh_lrus+0x27/0x30 [ 2524.214921] invalidate_bdev+0x5a/0x90 [ 2524.216972] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2524.219544] generic_shutdown_super+0xb1/0x1b0 [ 2524.222152] kill_block_super+0x29/0x80 [ 2524.224168] deactivate_locked_super+0x52/0xd0 [ 2524.226477] deactivate_super+0x83/0x90 [ 2524.228188] cleanup_mnt+0x5f/0xe0 [ 2524.232680] __cleanup_mnt+0x16/0x20 [ 2524.234487] task_work_run+0xc6/0x110 [ 2524.236423] exit_to_usermode_loop+0x1dd/0x1f0 [ 2524.238726] do_syscall_64+0x3d6/0x440 [ 2524.240900] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2524.243503] RIP: 0033:0x7fbb019648fb [ 2524.245452] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2524.255016] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2524.259443] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2524.262367] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2524.266254] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2524.272338] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2524.276330] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2528.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2528.062835] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2528.086350] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2528.092423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2528.097403] Workqueue: events free_work [libcfs] [ 2528.100611] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2528.104269] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2528.117226] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2528.120728] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2528.123713] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2528.125676] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2528.128459] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2528.131049] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2528.134404] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2528.140416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2528.147073] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2528.151308] Call Trace: [ 2528.153211] [ 2528.155225] ? show_regs.cold.9+0x22/0x2f [ 2528.157213] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2528.163212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2528.166168] ? watchdog+0x40/0x40 [ 2528.169827] ? hrtimer_interrupt+0x128/0x2c0 [ 2528.173214] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2528.176348] ? apic_timer_interrupt+0xf/0x20 [ 2528.177491] [ 2528.178058] ? on_each_cpu+0x51/0x120 [ 2528.178956] ? smp_call_function_many_cond+0x3db/0x540 [ 2528.183469] ? smp_call_function_many_cond+0x3a8/0x540 [ 2528.186393] ? load_new_mm_cr3+0x120/0x120 [ 2528.189051] ? load_new_mm_cr3+0x120/0x120 [ 2528.191109] on_each_cpu+0x51/0x120 [ 2528.192268] flush_tlb_kernel_range+0x93/0xb0 [ 2528.195086] free_unmap_vmap_area+0x63/0x70 [ 2528.197247] remove_vm_area+0xe7/0x110 [ 2528.199443] __vunmap+0x9c/0x400 [ 2528.201128] vfree+0x4d/0x100 [ 2528.202437] free_work+0x34/0x50 [libcfs] [ 2528.204175] process_one_work+0x2c8/0x700 [ 2528.205675] worker_thread+0x296/0x6e0 [ 2528.209146] ? rescuer_thread+0x570/0x570 [ 2528.212371] kthread+0x1d1/0x200 [ 2528.214804] ? set_kthread_struct+0x70/0x70 [ 2528.218247] ret_from_fork+0x1f/0x30 [ 2540.055733] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2540.066834] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.100917] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.105859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2540.112316] Workqueue: events free_work [libcfs] [ 2540.114368] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2540.117131] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2540.125252] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.128504] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2540.131500] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2540.134386] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2540.137754] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2540.141484] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2540.144419] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2540.147824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.150391] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2540.153604] Call Trace: [ 2540.154714] [ 2540.155585] ? show_regs.cold.9+0x22/0x2f [ 2540.157521] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.159752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.161898] ? watchdog+0x40/0x40 [ 2540.163534] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.165808] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.168018] ? apic_timer_interrupt+0xf/0x20 [ 2540.169349] [ 2540.169975] ? on_each_cpu+0x51/0x120 [ 2540.171522] ? smp_call_function_many_cond+0x3d9/0x540 [ 2540.173420] ? smp_call_function_many_cond+0x3a8/0x540 [ 2540.175580] ? load_new_mm_cr3+0x120/0x120 [ 2540.177393] ? load_new_mm_cr3+0x120/0x120 [ 2540.179142] on_each_cpu+0x51/0x120 [ 2540.180680] flush_tlb_kernel_range+0x93/0xb0 [ 2540.182512] free_unmap_vmap_area+0x63/0x70 [ 2540.184213] remove_vm_area+0xe7/0x110 [ 2540.185766] __vunmap+0x9c/0x400 [ 2540.187077] vfree+0x4d/0x100 [ 2540.188579] free_work+0x34/0x50 [libcfs] [ 2540.190622] process_one_work+0x2c8/0x700 [ 2540.192221] worker_thread+0x296/0x6e0 [ 2540.193887] ? rescuer_thread+0x570/0x570 [ 2540.195539] kthread+0x1d1/0x200 [ 2540.196966] ? set_kthread_struct+0x70/0x70 [ 2540.198795] ret_from_fork+0x1f/0x30 [ 2543.072311] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 184s! [ 2543.078104] Showing busy workqueues and worker pools: [ 2543.080301] workqueue events: flags=0x0 [ 2543.082370] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2543.086522] in-flight: 56293:free_work [libcfs] [ 2543.089991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2543.096146] in-flight: 22277:free_work [libcfs] [ 2543.103593] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2543.109580] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2543.113171] pending: nfs4_renew_state [nfsv4] [ 2543.115672] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2543.118720] in-flight: 47596:free_work [libcfs] [ 2543.123950] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2543.132066] workqueue events_unbound: flags=0x2 [ 2543.135578] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2543.138837] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2543.143464] workqueue events_power_efficient: flags=0x80 [ 2543.145868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2543.148982] pending: neigh_periodic_work, do_cache_clean [ 2543.151544] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2543.154803] pending: check_lifetime [ 2543.156770] workqueue mm_percpu_wq: flags=0x8 [ 2543.158920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2543.165094] pending: vmstat_update [ 2543.167846] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2543.171933] pending: vmstat_update [ 2543.173977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2543.177561] pending: vmstat_update [ 2543.179600] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2543.182164] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2543.185183] pending: cgroup_pidlist_destroy_work_fn [ 2543.188970] workqueue writeback: flags=0x4a [ 2543.190397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2543.194056] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2543.197768] workqueue ipv6_addrconf: flags=0x40008 [ 2543.200071] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2543.202164] pending: addrconf_verify_work [ 2543.203529] workqueue knbd0-recv: flags=0x1a [ 2543.205931] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2543.209094] in-flight: 655:recv_work [ 2543.213289] workqueue ptlrpc_pinger: flags=0x2 [ 2543.216959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2543.219816] pending: ptlrpc_pinger_main [ptlrpc] [ 2543.223276] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=185s workers=5 idle: 53684 27131 54850 46584 [ 2543.227732] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=185s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2543.233945] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=185s workers=6 idle: 53660 37674 46417 53664 46418 [ 2543.238517] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2533s workers=2 idle: 657 [ 2552.058183] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2552.060481] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2552.080852] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2552.084844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2552.090282] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2552.092614] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2552.099163] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2552.103774] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2552.105529] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2552.108471] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2552.114278] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2552.125262] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2552.132473] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2552.135695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2552.137918] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2552.142273] Call Trace: [ 2552.143800] [ 2552.145891] ? show_regs.cold.9+0x22/0x2f [ 2552.147223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2552.148541] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2552.152854] ? watchdog+0x40/0x40 [ 2552.157893] ? hrtimer_interrupt+0x128/0x2c0 [ 2552.160042] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2552.161644] ? apic_timer_interrupt+0xf/0x20 [ 2552.163475] [ 2552.164001] ? on_each_cpu_cond_mask+0x46/0x150 [ 2552.166485] ? __x64_sys_fspick+0x210/0x210 [ 2552.167610] ? smp_call_function_many_cond+0x3db/0x540 [ 2552.169756] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2552.171597] ? __x64_sys_fspick+0x210/0x210 [ 2552.172741] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2552.175481] on_each_cpu_cond_mask+0x46/0x150 [ 2552.177440] on_each_cpu_cond+0x1c/0x30 [ 2552.180394] invalidate_bh_lrus+0x27/0x30 [ 2552.183455] invalidate_bdev+0x5a/0x90 [ 2552.185491] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2552.190820] generic_shutdown_super+0xb1/0x1b0 [ 2552.191963] kill_block_super+0x29/0x80 [ 2552.195540] deactivate_locked_super+0x52/0xd0 [ 2552.198423] deactivate_super+0x83/0x90 [ 2552.199799] cleanup_mnt+0x5f/0xe0 [ 2552.201675] __cleanup_mnt+0x16/0x20 [ 2552.203536] task_work_run+0xc6/0x110 [ 2552.205539] exit_to_usermode_loop+0x1dd/0x1f0 [ 2552.210626] do_syscall_64+0x3d6/0x440 [ 2552.211632] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2552.213017] RIP: 0033:0x7fbb019648fb [ 2552.218891] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2552.239712] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2552.245469] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2552.249800] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2552.253882] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2552.259552] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2552.261950] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2556.059637] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2556.066024] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2556.097201] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2556.103113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2556.107676] Workqueue: events free_work [libcfs] [ 2556.111262] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2556.116273] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2556.129782] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2556.133146] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2556.135524] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2556.140241] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2556.144763] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2556.151782] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2556.156537] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2556.160509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2556.162385] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2556.166403] Call Trace: [ 2556.167724] [ 2556.168718] ? show_regs.cold.9+0x22/0x2f [ 2556.170819] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2556.173659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2556.175451] ? watchdog+0x40/0x40 [ 2556.176837] ? hrtimer_interrupt+0x128/0x2c0 [ 2556.179490] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2556.182217] ? apic_timer_interrupt+0xf/0x20 [ 2556.184425] [ 2556.185325] ? on_each_cpu+0x51/0x120 [ 2556.187357] ? smp_call_function_many_cond+0x3db/0x540 [ 2556.190043] ? smp_call_function_many_cond+0x3a8/0x540 [ 2556.192564] ? load_new_mm_cr3+0x120/0x120 [ 2556.194685] ? load_new_mm_cr3+0x120/0x120 [ 2556.196854] on_each_cpu+0x51/0x120 [ 2556.198643] flush_tlb_kernel_range+0x93/0xb0 [ 2556.201858] free_unmap_vmap_area+0x63/0x70 [ 2556.206680] remove_vm_area+0xe7/0x110 [ 2556.207786] __vunmap+0x9c/0x400 [ 2556.209117] vfree+0x4d/0x100 [ 2556.210351] free_work+0x34/0x50 [libcfs] [ 2556.212698] process_one_work+0x2c8/0x700 [ 2556.215178] worker_thread+0x296/0x6e0 [ 2556.216883] ? rescuer_thread+0x570/0x570 [ 2556.219303] kthread+0x1d1/0x200 [ 2556.221039] ? set_kthread_struct+0x70/0x70 [ 2556.223432] ret_from_fork+0x1f/0x30 [ 2568.055034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2568.057463] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.073895] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.077110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2568.080150] Workqueue: events free_work [libcfs] [ 2568.081978] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2568.083747] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2568.089897] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2568.092076] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2568.094794] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2568.096479] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2568.098827] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2568.100525] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2568.102304] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2568.104485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.105988] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2568.107854] Call Trace: [ 2568.108492] [ 2568.109067] ? show_regs.cold.9+0x22/0x2f [ 2568.110085] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.111351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.112471] ? watchdog+0x40/0x40 [ 2568.113365] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.114600] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.116070] ? apic_timer_interrupt+0xf/0x20 [ 2568.117248] [ 2568.117857] ? on_each_cpu+0x51/0x120 [ 2568.118959] ? smp_call_function_many_cond+0x3db/0x540 [ 2568.120465] ? smp_call_function_many_cond+0x3a8/0x540 [ 2568.121918] ? load_new_mm_cr3+0x120/0x120 [ 2568.122969] ? load_new_mm_cr3+0x120/0x120 [ 2568.123982] on_each_cpu+0x51/0x120 [ 2568.124909] flush_tlb_kernel_range+0x93/0xb0 [ 2568.126063] free_unmap_vmap_area+0x63/0x70 [ 2568.127124] remove_vm_area+0xe7/0x110 [ 2568.128283] __vunmap+0x9c/0x400 [ 2568.129194] vfree+0x4d/0x100 [ 2568.130056] free_work+0x34/0x50 [libcfs] [ 2568.131341] process_one_work+0x2c8/0x700 [ 2568.132597] worker_thread+0x296/0x6e0 [ 2568.133594] ? rescuer_thread+0x570/0x570 [ 2568.134621] kthread+0x1d1/0x200 [ 2568.135487] ? set_kthread_struct+0x70/0x70 [ 2568.137024] ret_from_fork+0x1f/0x30 [ 2573.279108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 215s! [ 2573.286729] Showing busy workqueues and worker pools: [ 2573.290846] workqueue events: flags=0x0 [ 2573.293068] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2573.297039] in-flight: 56293:free_work [libcfs] [ 2573.301365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2573.305940] in-flight: 22277:free_work [libcfs] [ 2573.309474] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2573.314184] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2573.317380] pending: nfs4_renew_state [nfsv4] [ 2573.320140] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2573.323290] in-flight: 47596:free_work [libcfs] [ 2573.324619] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2573.328436] workqueue events_unbound: flags=0x2 [ 2573.330667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2573.336342] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2573.342506] workqueue events_power_efficient: flags=0x80 [ 2573.346110] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2573.349579] pending: neigh_periodic_work, do_cache_clean [ 2573.353918] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2573.359062] pending: check_lifetime [ 2573.361762] workqueue mm_percpu_wq: flags=0x8 [ 2573.363509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2573.366947] pending: vmstat_update [ 2573.368807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2573.372978] pending: vmstat_update [ 2573.375029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2573.378514] pending: vmstat_update [ 2573.380506] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2573.384340] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2573.389391] pending: cgroup_pidlist_destroy_work_fn [ 2573.393127] workqueue writeback: flags=0x4a [ 2573.396078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2573.400053] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2573.405049] workqueue ipv6_addrconf: flags=0x40008 [ 2573.407566] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2573.410951] pending: addrconf_verify_work [ 2573.412687] workqueue knbd0-recv: flags=0x1a [ 2573.417041] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2573.422206] in-flight: 655:recv_work [ 2573.424438] workqueue ptlrpc_pinger: flags=0x2 [ 2573.426881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2573.430295] pending: ptlrpc_pinger_main [ptlrpc] [ 2573.433296] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=215s workers=5 idle: 53684 27131 54850 46584 [ 2573.437860] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=215s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2573.442898] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=215s workers=6 idle: 53660 37674 46417 53664 46418 [ 2573.451439] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2563s workers=2 idle: 657 [ 2580.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2580.061485] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2580.090595] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2580.097497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2580.104723] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2580.109974] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2580.118809] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2580.122381] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2580.125691] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2580.129016] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2580.132228] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2580.135852] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2580.140069] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2580.143476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2580.150819] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2580.157650] Call Trace: [ 2580.158570] [ 2580.164803] ? show_regs.cold.9+0x22/0x2f [ 2580.166534] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2580.175352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2580.178044] ? watchdog+0x40/0x40 [ 2580.179729] ? hrtimer_interrupt+0x128/0x2c0 [ 2580.181997] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2580.184634] ? apic_timer_interrupt+0xf/0x20 [ 2580.187023] [ 2580.188093] ? on_each_cpu_cond_mask+0x46/0x150 [ 2580.190708] ? __x64_sys_fspick+0x210/0x210 [ 2580.192781] ? smp_call_function_many_cond+0x3db/0x540 [ 2580.195126] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2580.196920] ? __x64_sys_fspick+0x210/0x210 [ 2580.199035] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2580.201347] on_each_cpu_cond_mask+0x46/0x150 [ 2580.202743] on_each_cpu_cond+0x1c/0x30 [ 2580.203859] invalidate_bh_lrus+0x27/0x30 [ 2580.205242] invalidate_bdev+0x5a/0x90 [ 2580.206596] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2580.209709] generic_shutdown_super+0xb1/0x1b0 [ 2580.213078] kill_block_super+0x29/0x80 [ 2580.214385] deactivate_locked_super+0x52/0xd0 [ 2580.217363] deactivate_super+0x83/0x90 [ 2580.219770] cleanup_mnt+0x5f/0xe0 [ 2580.221772] __cleanup_mnt+0x16/0x20 [ 2580.224102] task_work_run+0xc6/0x110 [ 2580.226095] exit_to_usermode_loop+0x1dd/0x1f0 [ 2580.228785] do_syscall_64+0x3d6/0x440 [ 2580.231121] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2580.236489] RIP: 0033:0x7fbb019648fb [ 2580.239679] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2580.252853] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2580.257814] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2580.261168] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2580.265974] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2580.272515] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2580.276956] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2584.059008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2584.062141] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2584.080172] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2584.084256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2584.087748] Workqueue: events free_work [libcfs] [ 2584.089755] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2584.092054] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2584.099243] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2584.102395] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2584.104980] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2584.109241] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2584.113019] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2584.116142] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2584.119651] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2584.123766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2584.126530] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2584.129825] Call Trace: [ 2584.130961] [ 2584.131925] ? show_regs.cold.9+0x22/0x2f [ 2584.133985] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2584.136262] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2584.138809] ? watchdog+0x40/0x40 [ 2584.140270] ? hrtimer_interrupt+0x128/0x2c0 [ 2584.142909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2584.145063] ? apic_timer_interrupt+0xf/0x20 [ 2584.146284] [ 2584.147518] ? on_each_cpu+0x51/0x120 [ 2584.148972] ? smp_call_function_many_cond+0x3d9/0x540 [ 2584.151703] ? smp_call_function_many_cond+0x3a8/0x540 [ 2584.154286] ? load_new_mm_cr3+0x120/0x120 [ 2584.156193] ? load_new_mm_cr3+0x120/0x120 [ 2584.157893] on_each_cpu+0x51/0x120 [ 2584.159580] flush_tlb_kernel_range+0x93/0xb0 [ 2584.161098] free_unmap_vmap_area+0x63/0x70 [ 2584.162549] remove_vm_area+0xe7/0x110 [ 2584.164590] __vunmap+0x9c/0x400 [ 2584.165930] vfree+0x4d/0x100 [ 2584.167284] free_work+0x34/0x50 [libcfs] [ 2584.169204] process_one_work+0x2c8/0x700 [ 2584.171198] worker_thread+0x296/0x6e0 [ 2584.173214] ? rescuer_thread+0x570/0x570 [ 2584.175185] kthread+0x1d1/0x200 [ 2584.176951] ? set_kthread_struct+0x70/0x70 [ 2584.179117] ret_from_fork+0x1f/0x30 [ 2597.738137] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 2597.745520] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 2597.757311] (detected by 2, t=240008 jiffies, g=626849, q=1398) [ 2597.764473] Sending NMI from CPU 2 to CPUs 3: [ 2597.770484] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 2597.770494] rcu: rcu_sched kthread starved for 239650 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2597.787687] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2597.799928] rcu: RCU grace-period kthread stack dump: [ 2597.802528] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2597.809185] Call Trace: [ 2597.812579] __schedule+0x351/0xcb0 [ 2597.815125] schedule+0xc0/0x180 [ 2597.818497] schedule_timeout+0xb4/0x190 [ 2597.823342] ? do_raw_spin_unlock+0x75/0x190 [ 2597.826527] ? __next_timer_interrupt+0x160/0x160 [ 2597.829773] ? prepare_to_swait_event+0xf3/0x140 [ 2597.832898] rcu_gp_kthread+0x8dd/0x15b0 [ 2597.835418] ? force_qs_rnp+0x2c0/0x2c0 [ 2597.837929] kthread+0x1d1/0x200 [ 2597.839550] ? set_kthread_struct+0x70/0x70 [ 2597.841452] ret_from_fork+0x1f/0x30 [ 2597.843330] rcu: Stack dump where RCU GP kthread last ran: [ 2597.847456] NMI backtrace for cpu 2 [ 2597.849419] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2597.856988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2597.862497] Workqueue: events free_work [libcfs] [ 2597.865371] Call Trace: [ 2597.866368] [ 2597.867219] ? dump_stack+0xbb/0x10e [ 2597.869825] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2597.873721] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2597.877860] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2597.885454] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2597.889502] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2597.895264] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2597.902307] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 2597.908057] ? update_process_times+0x6d/0xc0 [ 2597.911986] ? tick_sched_handle.isra.18+0x27/0x90 [ 2597.917720] ? tick_sched_timer+0x53/0xb0 [ 2597.922799] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2597.926531] ? tick_sched_handle.isra.18+0x90/0x90 [ 2597.929013] ? hrtimer_interrupt+0x128/0x2c0 [ 2597.931835] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2597.935108] ? apic_timer_interrupt+0xf/0x20 [ 2597.938464] [ 2597.940269] ? on_each_cpu+0x51/0x120 [ 2597.945290] ? smp_call_function_many_cond+0x3d9/0x540 [ 2597.948172] ? smp_call_function_many_cond+0x3a8/0x540 [ 2597.950177] ? load_new_mm_cr3+0x120/0x120 [ 2597.954543] ? load_new_mm_cr3+0x120/0x120 [ 2597.959303] ? on_each_cpu+0x51/0x120 [ 2597.962186] ? flush_tlb_kernel_range+0x93/0xb0 [ 2597.964673] ? free_unmap_vmap_area+0x63/0x70 [ 2597.971943] ? remove_vm_area+0xe7/0x110 [ 2597.975363] ? __vunmap+0x9c/0x400 [ 2597.978676] ? vfree+0x4d/0x100 [ 2597.980712] ? free_work+0x34/0x50 [libcfs] [ 2597.982622] ? process_one_work+0x2c8/0x700 [ 2597.985727] ? worker_thread+0x296/0x6e0 [ 2597.988627] ? rescuer_thread+0x570/0x570 [ 2597.991455] ? kthread+0x1d1/0x200 [ 2597.998593] ? set_kthread_struct+0x70/0x70 [ 2598.000332] ? ret_from_fork+0x1f/0x30 [ 2600.057249] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2600.069285] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2600.102644] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2600.114172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2600.123277] Workqueue: events free_work [libcfs] [ 2600.124732] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2600.128569] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2600.145839] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2600.150269] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2600.155886] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2600.160800] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2600.168154] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2600.172156] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2600.174810] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2600.182819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2600.186445] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2600.189668] Call Trace: [ 2600.190761] [ 2600.191769] ? show_regs.cold.9+0x22/0x2f [ 2600.194092] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2600.196223] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2600.198399] ? watchdog+0x40/0x40 [ 2600.200283] ? hrtimer_interrupt+0x128/0x2c0 [ 2600.202625] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2600.204863] ? apic_timer_interrupt+0xf/0x20 [ 2600.206925] [ 2600.208178] ? on_each_cpu+0x51/0x120 [ 2600.209784] ? smp_call_function_many_cond+0x3d9/0x540 [ 2600.212614] ? smp_call_function_many_cond+0x3a8/0x540 [ 2600.215238] ? load_new_mm_cr3+0x120/0x120 [ 2600.217290] ? load_new_mm_cr3+0x120/0x120 [ 2600.219559] on_each_cpu+0x51/0x120 [ 2600.221349] flush_tlb_kernel_range+0x93/0xb0 [ 2600.223397] free_unmap_vmap_area+0x63/0x70 [ 2600.225479] remove_vm_area+0xe7/0x110 [ 2600.227384] __vunmap+0x9c/0x400 [ 2600.228723] vfree+0x4d/0x100 [ 2600.229731] free_work+0x34/0x50 [libcfs] [ 2600.232154] process_one_work+0x2c8/0x700 [ 2600.233803] worker_thread+0x296/0x6e0 [ 2600.235413] ? rescuer_thread+0x570/0x570 [ 2600.236807] kthread+0x1d1/0x200 [ 2600.238479] ? set_kthread_struct+0x70/0x70 [ 2600.240557] ret_from_fork+0x1f/0x30 [ 2603.491265] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 245s! [ 2603.497229] Showing busy workqueues and worker pools: [ 2603.499861] workqueue events: flags=0x0 [ 2603.502160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2603.507856] in-flight: 56293:free_work [libcfs] [ 2603.509849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2603.518429] in-flight: 22277:free_work [libcfs] [ 2603.521589] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2603.532683] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2603.538367] pending: nfs4_renew_state [nfsv4] [ 2603.551478] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2603.556236] in-flight: 47596:free_work [libcfs] [ 2603.560397] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2603.567158] workqueue events_unbound: flags=0x2 [ 2603.569178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2603.571786] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2603.574555] workqueue events_power_efficient: flags=0x80 [ 2603.579665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2603.583336] pending: neigh_periodic_work, do_cache_clean [ 2603.585744] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2603.589454] pending: check_lifetime [ 2603.592470] workqueue mm_percpu_wq: flags=0x8 [ 2603.594806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2603.598194] pending: vmstat_update [ 2603.601419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2603.606266] pending: vmstat_update [ 2603.608364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2603.612393] pending: vmstat_update [ 2603.615172] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2603.618189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2603.622111] pending: cgroup_pidlist_destroy_work_fn [ 2603.625399] workqueue writeback: flags=0x4a [ 2603.628115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2603.631611] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2603.636613] workqueue ipv6_addrconf: flags=0x40008 [ 2603.639248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2603.642473] pending: addrconf_verify_work [ 2603.645233] workqueue knbd0-recv: flags=0x1a [ 2603.647320] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2603.651080] in-flight: 655:recv_work [ 2603.652697] workqueue ptlrpc_pinger: flags=0x2 [ 2603.655269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2603.658165] pending: ptlrpc_pinger_main [ptlrpc] [ 2603.660584] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=245s workers=5 idle: 53684 27131 54850 46584 [ 2603.664417] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=245s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2603.669550] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=245s workers=6 idle: 53660 37674 46417 53664 46418 [ 2603.674114] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2593s workers=2 idle: 657 [ 2608.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2608.063701] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2608.107257] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2608.114714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2608.122443] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2608.128709] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2608.137628] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2608.141135] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2608.144219] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2608.146996] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2608.150078] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2608.153033] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2608.156146] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2608.159298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2608.161500] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2608.164456] Call Trace: [ 2608.165547] [ 2608.166486] ? show_regs.cold.9+0x22/0x2f [ 2608.168266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2608.170339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2608.172310] ? watchdog+0x40/0x40 [ 2608.173622] ? hrtimer_interrupt+0x128/0x2c0 [ 2608.175444] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2608.177415] ? apic_timer_interrupt+0xf/0x20 [ 2608.179349] [ 2608.180281] ? on_each_cpu_cond_mask+0x46/0x150 [ 2608.182255] ? __x64_sys_fspick+0x210/0x210 [ 2608.184330] ? smp_call_function_many_cond+0x3db/0x540 [ 2608.188455] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2608.190890] ? __x64_sys_fspick+0x210/0x210 [ 2608.193228] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2608.195925] on_each_cpu_cond_mask+0x46/0x150 [ 2608.198363] on_each_cpu_cond+0x1c/0x30 [ 2608.202871] invalidate_bh_lrus+0x27/0x30 [ 2608.204332] invalidate_bdev+0x5a/0x90 [ 2608.205934] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2608.212099] generic_shutdown_super+0xb1/0x1b0 [ 2608.213759] kill_block_super+0x29/0x80 [ 2608.217477] deactivate_locked_super+0x52/0xd0 [ 2608.221490] deactivate_super+0x83/0x90 [ 2608.223671] cleanup_mnt+0x5f/0xe0 [ 2608.226509] __cleanup_mnt+0x16/0x20 [ 2608.229191] task_work_run+0xc6/0x110 [ 2608.230621] exit_to_usermode_loop+0x1dd/0x1f0 [ 2608.233891] do_syscall_64+0x3d6/0x440 [ 2608.235398] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2608.238655] RIP: 0033:0x7fbb019648fb [ 2608.240984] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2608.260813] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2608.267348] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2608.271308] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2608.276592] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2608.280164] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2608.283471] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2624.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2624.065930] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.101175] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.108006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2624.111522] Workqueue: events free_work [libcfs] [ 2624.114304] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2624.117288] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2624.127496] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2624.131569] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2624.135311] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2624.139131] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2624.142916] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2624.145990] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2624.149214] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2624.152801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.155864] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2624.159214] Call Trace: [ 2624.160256] [ 2624.161367] ? show_regs.cold.9+0x22/0x2f [ 2624.163975] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.166716] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.169102] ? watchdog+0x40/0x40 [ 2624.170358] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.174708] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.177002] ? apic_timer_interrupt+0xf/0x20 [ 2624.179054] [ 2624.180356] ? on_each_cpu+0x51/0x120 [ 2624.181904] ? smp_call_function_many_cond+0x3db/0x540 [ 2624.183751] ? smp_call_function_many_cond+0x3a8/0x540 [ 2624.185934] ? load_new_mm_cr3+0x120/0x120 [ 2624.188029] ? load_new_mm_cr3+0x120/0x120 [ 2624.189499] on_each_cpu+0x51/0x120 [ 2624.190870] flush_tlb_kernel_range+0x93/0xb0 [ 2624.192969] free_unmap_vmap_area+0x63/0x70 [ 2624.195380] remove_vm_area+0xe7/0x110 [ 2624.197754] __vunmap+0x9c/0x400 [ 2624.199750] vfree+0x4d/0x100 [ 2624.201382] free_work+0x34/0x50 [libcfs] [ 2624.204472] process_one_work+0x2c8/0x700 [ 2624.208781] worker_thread+0x296/0x6e0 [ 2624.210924] ? rescuer_thread+0x570/0x570 [ 2624.213124] kthread+0x1d1/0x200 [ 2624.214640] ? set_kthread_struct+0x70/0x70 [ 2624.216786] ret_from_fork+0x1f/0x30 [ 2628.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2628.058962] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2628.099859] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2628.104943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2628.108779] Workqueue: events free_work [libcfs] [ 2628.110822] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2628.113104] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2628.121897] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2628.125160] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2628.128204] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2628.130424] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2628.133873] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2628.136749] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2628.140986] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2628.144292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2628.146791] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2628.151200] Call Trace: [ 2628.152757] [ 2628.153789] ? show_regs.cold.9+0x22/0x2f [ 2628.155868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2628.158120] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2628.160917] ? watchdog+0x40/0x40 [ 2628.168833] ? hrtimer_interrupt+0x128/0x2c0 [ 2628.170911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2628.173561] ? apic_timer_interrupt+0xf/0x20 [ 2628.175159] [ 2628.175910] ? on_each_cpu+0x51/0x120 [ 2628.182483] ? smp_call_function_many_cond+0x3db/0x540 [ 2628.185771] ? smp_call_function_many_cond+0x3a8/0x540 [ 2628.188555] ? load_new_mm_cr3+0x120/0x120 [ 2628.190555] ? load_new_mm_cr3+0x120/0x120 [ 2628.193945] on_each_cpu+0x51/0x120 [ 2628.196646] flush_tlb_kernel_range+0x93/0xb0 [ 2628.201522] free_unmap_vmap_area+0x63/0x70 [ 2628.207405] remove_vm_area+0xe7/0x110 [ 2628.209866] __vunmap+0x9c/0x400 [ 2628.212003] vfree+0x4d/0x100 [ 2628.213952] free_work+0x34/0x50 [libcfs] [ 2628.215773] process_one_work+0x2c8/0x700 [ 2628.220944] worker_thread+0x296/0x6e0 [ 2628.223176] ? rescuer_thread+0x570/0x570 [ 2628.232002] kthread+0x1d1/0x200 [ 2628.233542] ? set_kthread_struct+0x70/0x70 [ 2628.240417] ret_from_fork+0x1f/0x30 [ 2633.696024] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 275s! [ 2633.700879] Showing busy workqueues and worker pools: [ 2633.703629] workqueue events: flags=0x0 [ 2633.705324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2633.709104] in-flight: 56293:free_work [libcfs] [ 2633.712027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2633.716589] in-flight: 22277:free_work [libcfs] [ 2633.719122] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2633.724844] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2633.728479] pending: nfs4_renew_state [nfsv4] [ 2633.731050] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2633.734798] in-flight: 47596:free_work [libcfs] [ 2633.738750] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2633.743686] workqueue events_unbound: flags=0x2 [ 2633.745978] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2633.749193] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2633.753033] workqueue events_power_efficient: flags=0x80 [ 2633.755200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2633.758542] pending: neigh_periodic_work, do_cache_clean [ 2633.761149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2633.764017] pending: check_lifetime [ 2633.765895] workqueue mm_percpu_wq: flags=0x8 [ 2633.768077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2633.771579] pending: vmstat_update [ 2633.773352] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2633.776170] pending: vmstat_update [ 2633.777541] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2633.780579] pending: vmstat_update [ 2633.782040] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2633.783853] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2633.787933] pending: cgroup_pidlist_destroy_work_fn [ 2633.795638] workqueue writeback: flags=0x4a [ 2633.797277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2633.802088] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2633.807874] workqueue ipv6_addrconf: flags=0x40008 [ 2633.811145] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2633.815106] pending: addrconf_verify_work [ 2633.820573] workqueue knbd0-recv: flags=0x1a [ 2633.824627] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2633.830222] in-flight: 655:recv_work [ 2633.835683] workqueue ptlrpc_pinger: flags=0x2 [ 2633.837398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2633.839844] pending: ptlrpc_pinger_main [ptlrpc] [ 2633.846214] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=276s workers=5 idle: 53684 27131 54850 46584 [ 2633.849970] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=276s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2633.853820] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=276s workers=6 idle: 53660 37674 46417 53664 46418 [ 2633.857092] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2623s workers=2 idle: 657 [ 2636.059888] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 2636.063983] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2636.110615] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2636.125905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2636.136603] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2636.143272] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2636.167968] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2636.173152] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2636.177599] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2636.181042] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2636.183522] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2636.188593] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2636.192489] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2636.195406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2636.197663] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2636.200680] Call Trace: [ 2636.201451] [ 2636.201972] ? show_regs.cold.9+0x22/0x2f [ 2636.203669] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2636.205729] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2636.207638] ? watchdog+0x40/0x40 [ 2636.210971] ? hrtimer_interrupt+0x128/0x2c0 [ 2636.213328] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2636.216993] ? apic_timer_interrupt+0xf/0x20 [ 2636.219564] [ 2636.220893] ? on_each_cpu_cond_mask+0x46/0x150 [ 2636.224133] ? __x64_sys_fspick+0x210/0x210 [ 2636.228681] ? smp_call_function_many_cond+0x3db/0x540 [ 2636.234862] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2636.238071] ? __x64_sys_fspick+0x210/0x210 [ 2636.242458] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2636.244955] on_each_cpu_cond_mask+0x46/0x150 [ 2636.247083] on_each_cpu_cond+0x1c/0x30 [ 2636.249068] invalidate_bh_lrus+0x27/0x30 [ 2636.250492] invalidate_bdev+0x5a/0x90 [ 2636.251917] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2636.253706] generic_shutdown_super+0xb1/0x1b0 [ 2636.255468] kill_block_super+0x29/0x80 [ 2636.257024] deactivate_locked_super+0x52/0xd0 [ 2636.259575] deactivate_super+0x83/0x90 [ 2636.261714] cleanup_mnt+0x5f/0xe0 [ 2636.262991] __cleanup_mnt+0x16/0x20 [ 2636.264767] task_work_run+0xc6/0x110 [ 2636.266349] exit_to_usermode_loop+0x1dd/0x1f0 [ 2636.269853] do_syscall_64+0x3d6/0x440 [ 2636.271295] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2636.273422] RIP: 0033:0x7fbb019648fb [ 2636.274993] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2636.283584] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2636.292548] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2636.296372] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2636.301040] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2636.304853] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2636.310889] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2652.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2652.062247] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.084300] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.088482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2652.091567] Workqueue: events free_work [libcfs] [ 2652.094013] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2652.096556] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2652.110094] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2652.115022] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2652.118554] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2652.120974] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2652.122944] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2652.125644] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2652.127897] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2652.130813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.132826] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2652.135719] Call Trace: [ 2652.137002] [ 2652.137788] ? show_regs.cold.9+0x22/0x2f [ 2652.139632] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.142105] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.144819] ? watchdog+0x40/0x40 [ 2652.145845] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.146992] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.148549] ? apic_timer_interrupt+0xf/0x20 [ 2652.149739] [ 2652.150456] ? on_each_cpu+0x51/0x120 [ 2652.151593] ? smp_call_function_many_cond+0x3db/0x540 [ 2652.153095] ? smp_call_function_many_cond+0x3a8/0x540 [ 2652.155219] ? load_new_mm_cr3+0x120/0x120 [ 2652.157519] ? load_new_mm_cr3+0x120/0x120 [ 2652.159136] on_each_cpu+0x51/0x120 [ 2652.160422] flush_tlb_kernel_range+0x93/0xb0 [ 2652.161627] free_unmap_vmap_area+0x63/0x70 [ 2652.162797] remove_vm_area+0xe7/0x110 [ 2652.163801] __vunmap+0x9c/0x400 [ 2652.164733] vfree+0x4d/0x100 [ 2652.165836] free_work+0x34/0x50 [libcfs] [ 2652.167202] process_one_work+0x2c8/0x700 [ 2652.168244] worker_thread+0x296/0x6e0 [ 2652.169509] ? rescuer_thread+0x570/0x570 [ 2652.170628] kthread+0x1d1/0x200 [ 2652.171537] ? set_kthread_struct+0x70/0x70 [ 2652.173026] ret_from_fork+0x1f/0x30 [ 2660.055015] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2660.062119] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2660.109452] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2660.115142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2660.119606] Workqueue: events free_work [libcfs] [ 2660.122810] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2660.126226] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2660.144347] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2660.149345] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2660.156422] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2660.165895] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2660.172386] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2660.180783] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2660.188407] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2660.197172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2660.204058] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2660.209646] Call Trace: [ 2660.210664] [ 2660.211461] ? show_regs.cold.9+0x22/0x2f [ 2660.214634] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2660.221636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2660.225337] ? watchdog+0x40/0x40 [ 2660.227207] ? hrtimer_interrupt+0x128/0x2c0 [ 2660.228824] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2660.230558] ? apic_timer_interrupt+0xf/0x20 [ 2660.233084] [ 2660.233954] ? on_each_cpu+0x51/0x120 [ 2660.238219] ? smp_call_function_many_cond+0x3d9/0x540 [ 2660.240284] ? smp_call_function_many_cond+0x3a8/0x540 [ 2660.244248] ? load_new_mm_cr3+0x120/0x120 [ 2660.246484] ? load_new_mm_cr3+0x120/0x120 [ 2660.247777] on_each_cpu+0x51/0x120 [ 2660.248859] flush_tlb_kernel_range+0x93/0xb0 [ 2660.250324] free_unmap_vmap_area+0x63/0x70 [ 2660.252726] remove_vm_area+0xe7/0x110 [ 2660.254606] __vunmap+0x9c/0x400 [ 2660.256288] vfree+0x4d/0x100 [ 2660.257819] free_work+0x34/0x50 [libcfs] [ 2660.259976] process_one_work+0x2c8/0x700 [ 2660.261987] worker_thread+0x296/0x6e0 [ 2660.263860] ? rescuer_thread+0x570/0x570 [ 2660.266008] kthread+0x1d1/0x200 [ 2660.266975] ? set_kthread_struct+0x70/0x70 [ 2660.268328] ret_from_fork+0x1f/0x30 [ 2663.903028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 305s! [ 2663.908763] Showing busy workqueues and worker pools: [ 2663.912915] workqueue events: flags=0x0 [ 2663.920330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2663.925255] in-flight: 56293:free_work [libcfs] [ 2663.931609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2663.937801] in-flight: 22277:free_work [libcfs] [ 2663.942525] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2663.954166] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2663.964196] pending: nfs4_renew_state [nfsv4] [ 2663.968334] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2663.974540] in-flight: 47596:free_work [libcfs] [ 2663.976735] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2663.981891] workqueue events_unbound: flags=0x2 [ 2663.984309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2663.987720] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2663.992040] workqueue events_power_efficient: flags=0x80 [ 2663.995345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2664.001306] pending: neigh_periodic_work, do_cache_clean [ 2664.006295] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2664.011795] pending: check_lifetime [ 2664.016154] workqueue mm_percpu_wq: flags=0x8 [ 2664.024119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2664.033466] pending: vmstat_update [ 2664.035298] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2664.042174] pending: vmstat_update [ 2664.045338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2664.054434] pending: vmstat_update [ 2664.059270] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 2664.060232] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2664.060241] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2664.060252] pending: cgroup_pidlist_destroy_work_fn [ 2664.060264] workqueue writeback: flags=0x4a [ 2664.060266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2664.060273] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2664.060295] workqueue xprtiod: flags=0xa [ 2664.060297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2664.060301] pending: xprt_autoclose [ 2664.060314] workqueue ipv6_addrconf: flags=0x40008 [ 2664.060317] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2664.060323] pending: addrconf_verify_work [ 2664.060335] workqueue knbd0-recv: flags=0x1a [ 2664.060337] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2664.060344] in-flight: 655:recv_work [ 2664.060354] workqueue ptlrpc_pinger: flags=0x2 [ 2664.060356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2664.060362] pending: ptlrpc_pinger_main [ptlrpc] [ 2664.067911] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.067997] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.068005] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2664.068008] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2664.068021] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2664.068025] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2664.068029] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2664.068032] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2664.068034] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2664.068036] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2664.068039] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2664.068041] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2664.068044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.068047] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2664.068055] Call Trace: [ 2664.068061] [ 2664.068066] ? show_regs.cold.9+0x22/0x2f [ 2664.068075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.068081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.068085] ? watchdog+0x40/0x40 [ 2664.068090] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.068095] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.068104] ? apic_timer_interrupt+0xf/0x20 [ 2664.068108] [ 2664.068110] ? on_each_cpu_cond_mask+0x46/0x150 [ 2664.068113] ? __x64_sys_fspick+0x210/0x210 [ 2664.068120] ? smp_call_function_many_cond+0x3db/0x540 [ 2664.068124] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2664.068127] ? __x64_sys_fspick+0x210/0x210 [ 2664.068132] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2664.068135] on_each_cpu_cond_mask+0x46/0x150 [ 2664.068139] on_each_cpu_cond+0x1c/0x30 [ 2664.068141] invalidate_bh_lrus+0x27/0x30 [ 2664.068145] invalidate_bdev+0x5a/0x90 [ 2664.068151] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2664.068268] generic_shutdown_super+0xb1/0x1b0 [ 2664.068277] kill_block_super+0x29/0x80 [ 2664.068283] deactivate_locked_super+0x52/0xd0 [ 2664.068288] deactivate_super+0x83/0x90 [ 2664.068293] cleanup_mnt+0x5f/0xe0 [ 2664.068298] __cleanup_mnt+0x16/0x20 [ 2664.068302] task_work_run+0xc6/0x110 [ 2664.068312] exit_to_usermode_loop+0x1dd/0x1f0 [ 2664.068320] do_syscall_64+0x3d6/0x440 [ 2664.068324] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2664.068331] RIP: 0033:0x7fbb019648fb [ 2664.068336] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2664.068340] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2664.068344] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2664.068346] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2664.068348] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2664.068350] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2664.068352] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2664.469424] [ 2664.471081] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=306s workers=5 idle: 53684 27131 54850 46584 [ 2664.475591] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=306s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2664.482275] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=306s workers=6 idle: 53660 37674 46417 53664 46418 [ 2664.489391] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2654s workers=2 idle: 657 [ 2680.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2680.061866] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.082800] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.086831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2680.089954] Workqueue: events free_work [libcfs] [ 2680.091631] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2680.093570] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2680.100543] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2680.103107] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2680.105503] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2680.107829] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2680.110495] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2680.112761] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2680.114983] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2680.117843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.119715] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2680.122440] Call Trace: [ 2680.123044] [ 2680.123835] ? show_regs.cold.9+0x22/0x2f [ 2680.125076] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.126871] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.128961] ? watchdog+0x40/0x40 [ 2680.130566] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.132652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.133965] ? apic_timer_interrupt+0xf/0x20 [ 2680.135985] [ 2680.136878] ? on_each_cpu+0x51/0x120 [ 2680.138583] ? smp_call_function_many_cond+0x3db/0x540 [ 2680.140736] ? smp_call_function_many_cond+0x3a8/0x540 [ 2680.142640] ? load_new_mm_cr3+0x120/0x120 [ 2680.144081] ? load_new_mm_cr3+0x120/0x120 [ 2680.145635] on_each_cpu+0x51/0x120 [ 2680.147348] flush_tlb_kernel_range+0x93/0xb0 [ 2680.148750] free_unmap_vmap_area+0x63/0x70 [ 2680.150049] remove_vm_area+0xe7/0x110 [ 2680.151300] __vunmap+0x9c/0x400 [ 2680.152167] vfree+0x4d/0x100 [ 2680.153481] free_work+0x34/0x50 [libcfs] [ 2680.154818] process_one_work+0x2c8/0x700 [ 2680.156405] worker_thread+0x296/0x6e0 [ 2680.157657] ? rescuer_thread+0x570/0x570 [ 2680.158938] kthread+0x1d1/0x200 [ 2680.160009] ? set_kthread_struct+0x70/0x70 [ 2680.161614] ret_from_fork+0x1f/0x30 [ 2692.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 2692.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 2692.061011] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2692.064736] Modules linked in: zfs(O) [ 2692.098153] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2692.098167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2692.100558] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2692.100892] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2692.100899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2692.100902] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2692.100914] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2692.100918] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2692.100922] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2692.100925] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2692.100927] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2692.100929] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2692.100932] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2692.100935] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2692.100938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2692.100940] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2692.100949] Call Trace: [ 2692.100954] [ 2692.100959] ? show_regs.cold.9+0x22/0x2f [ 2692.100969] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2692.100976] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2692.100980] ? watchdog+0x40/0x40 [ 2692.100984] ? hrtimer_interrupt+0x128/0x2c0 [ 2692.109158] Workqueue: events free_work [libcfs] [ 2692.109205] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2692.109227] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2692.109232] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2692.109236] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2692.109238] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2692.109240] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2692.109242] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2692.109245] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2692.109248] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2692.109251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2692.109253] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2692.109262] Call Trace: [ 2692.109267] [ 2692.109272] ? show_regs.cold.9+0x22/0x2f [ 2692.109282] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2692.109288] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2692.109293] ? watchdog+0x40/0x40 [ 2692.109297] ? hrtimer_interrupt+0x128/0x2c0 [ 2692.109301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2692.109309] ? apic_timer_interrupt+0xf/0x20 [ 2692.109314] [ 2692.109317] ? on_each_cpu+0x51/0x120 [ 2692.109321] ? smp_call_function_many_cond+0x3d9/0x540 [ 2692.109325] ? smp_call_function_many_cond+0x3a8/0x540 [ 2692.109329] ? load_new_mm_cr3+0x120/0x120 [ 2692.109334] ? load_new_mm_cr3+0x120/0x120 [ 2692.109338] on_each_cpu+0x51/0x120 [ 2692.109342] flush_tlb_kernel_range+0x93/0xb0 [ 2692.109348] free_unmap_vmap_area+0x63/0x70 [ 2692.109354] remove_vm_area+0xe7/0x110 [ 2692.109358] __vunmap+0x9c/0x400 [ 2692.109361] vfree+0x4d/0x100 [ 2692.109366] free_work+0x34/0x50 [libcfs] [ 2692.109394] process_one_work+0x2c8/0x700 [ 2692.109401] worker_thread+0x296/0x6e0 [ 2692.109406] ? rescuer_thread+0x570/0x570 [ 2692.109409] kthread+0x1d1/0x200 [ 2692.109414] ? set_kthread_struct+0x70/0x70 [ 2692.109418] ret_from_fork+0x1f/0x30 [ 2692.268326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2692.270457] ? apic_timer_interrupt+0xf/0x20 [ 2692.271825] [ 2692.272509] ? on_each_cpu_cond_mask+0x46/0x150 [ 2692.274404] ? __x64_sys_fspick+0x210/0x210 [ 2692.276159] ? smp_call_function_many_cond+0x3db/0x540 [ 2692.278497] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2692.279854] ? __x64_sys_fspick+0x210/0x210 [ 2692.281359] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2692.282759] on_each_cpu_cond_mask+0x46/0x150 [ 2692.283939] on_each_cpu_cond+0x1c/0x30 [ 2692.285146] invalidate_bh_lrus+0x27/0x30 [ 2692.286344] invalidate_bdev+0x5a/0x90 [ 2692.287843] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2692.290255] generic_shutdown_super+0xb1/0x1b0 [ 2692.291720] kill_block_super+0x29/0x80 [ 2692.293228] deactivate_locked_super+0x52/0xd0 [ 2692.295069] deactivate_super+0x83/0x90 [ 2692.296400] cleanup_mnt+0x5f/0xe0 [ 2692.297741] __cleanup_mnt+0x16/0x20 [ 2692.299437] task_work_run+0xc6/0x110 [ 2692.301164] exit_to_usermode_loop+0x1dd/0x1f0 [ 2692.303109] do_syscall_64+0x3d6/0x440 [ 2692.304748] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2692.306577] RIP: 0033:0x7fbb019648fb [ 2692.308087] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2692.315142] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2692.319680] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2692.322606] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2692.326794] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2692.332394] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2692.334886] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2694.624171] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 336s! [ 2694.628722] Showing busy workqueues and worker pools: [ 2694.631992] workqueue events: flags=0x0 [ 2694.633992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2694.636783] in-flight: 56293:free_work [libcfs] [ 2694.639226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2694.642306] in-flight: 22277:free_work [libcfs] [ 2694.644732] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2694.649237] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2694.656737] pending: nfs4_renew_state [nfsv4] [ 2694.659753] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2694.663200] in-flight: 47596:free_work [libcfs] [ 2694.665631] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2694.669402] workqueue events_unbound: flags=0x2 [ 2694.671806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2694.674770] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2694.678448] workqueue events_power_efficient: flags=0x80 [ 2694.680130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2694.683163] pending: neigh_periodic_work, do_cache_clean [ 2694.685580] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2694.688528] pending: check_lifetime [ 2694.690200] workqueue mm_percpu_wq: flags=0x8 [ 2694.693280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2694.695860] pending: vmstat_update [ 2694.701757] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2694.708401] pending: vmstat_update [ 2694.710258] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2694.715788] pending: vmstat_update [ 2694.719298] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2694.721550] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2694.724513] pending: cgroup_pidlist_destroy_work_fn [ 2694.726777] workqueue writeback: flags=0x4a [ 2694.728725] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2694.731900] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2694.735103] workqueue xprtiod: flags=0xa [ 2694.737041] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2694.739923] pending: xprt_autoclose [ 2694.741792] workqueue ipv6_addrconf: flags=0x40008 [ 2694.744143] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2694.747380] pending: addrconf_verify_work [ 2694.749436] workqueue knbd0-recv: flags=0x1a [ 2694.751279] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2694.754412] in-flight: 655:recv_work [ 2694.756048] workqueue ptlrpc_pinger: flags=0x2 [ 2694.759558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2694.762930] pending: ptlrpc_pinger_main [ptlrpc] [ 2694.769419] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=337s workers=5 idle: 53684 27131 54850 46584 [ 2694.772923] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=337s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2694.780892] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=337s workers=6 idle: 53660 37674 46417 53664 46418 [ 2694.785861] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2684s workers=2 idle: 657 [ 2708.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 2708.065780] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2708.097368] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2708.102206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2708.105890] Workqueue: events free_work [libcfs] [ 2708.109119] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2708.113080] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2708.124186] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2708.127360] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2708.131324] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2708.134942] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2708.137629] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2708.142193] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2708.149943] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2708.156956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2708.160189] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2708.163963] Call Trace: [ 2708.164637] [ 2708.165898] ? show_regs.cold.9+0x22/0x2f [ 2708.169409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2708.173094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2708.174953] ? watchdog+0x40/0x40 [ 2708.176993] ? hrtimer_interrupt+0x128/0x2c0 [ 2708.179018] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2708.181244] ? apic_timer_interrupt+0xf/0x20 [ 2708.187078] [ 2708.188276] ? on_each_cpu+0x51/0x120 [ 2708.191210] ? smp_call_function_many_cond+0x3d9/0x540 [ 2708.194883] ? smp_call_function_many_cond+0x3a8/0x540 [ 2708.197372] ? load_new_mm_cr3+0x120/0x120 [ 2708.201191] ? load_new_mm_cr3+0x120/0x120 [ 2708.204916] on_each_cpu+0x51/0x120 [ 2708.207264] flush_tlb_kernel_range+0x93/0xb0 [ 2708.210867] free_unmap_vmap_area+0x63/0x70 [ 2708.212751] remove_vm_area+0xe7/0x110 [ 2708.214828] __vunmap+0x9c/0x400 [ 2708.216244] vfree+0x4d/0x100 [ 2708.217245] free_work+0x34/0x50 [libcfs] [ 2708.219528] process_one_work+0x2c8/0x700 [ 2708.221778] worker_thread+0x296/0x6e0 [ 2708.223398] ? rescuer_thread+0x570/0x570 [ 2708.225091] kthread+0x1d1/0x200 [ 2708.226409] ? set_kthread_struct+0x70/0x70 [ 2708.229116] ret_from_fork+0x1f/0x30 [ 2720.056328] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 2720.059231] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 2720.070444] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2720.070535] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2720.070542] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2720.070559] Workqueue: events free_work [libcfs] [ 2720.070598] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2720.070616] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2720.070620] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2720.070624] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2720.070626] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2720.070629] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2720.070631] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2720.070633] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2720.070637] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2720.070639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2720.070642] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2720.070649] Call Trace: [ 2720.070654] [ 2720.070660] ? show_regs.cold.9+0x22/0x2f [ 2720.070670] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2720.070676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2720.070680] ? watchdog+0x40/0x40 [ 2720.070684] ? hrtimer_interrupt+0x128/0x2c0 [ 2720.070688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2720.070696] ? apic_timer_interrupt+0xf/0x20 [ 2720.070699] [ 2720.070701] ? on_each_cpu+0x51/0x120 [ 2720.070704] ? smp_call_function_many_cond+0x3db/0x540 [ 2720.070708] ? smp_call_function_many_cond+0x3a8/0x540 [ 2720.070711] ? load_new_mm_cr3+0x120/0x120 [ 2720.070716] ? load_new_mm_cr3+0x120/0x120 [ 2720.070719] on_each_cpu+0x51/0x120 [ 2720.070723] flush_tlb_kernel_range+0x93/0xb0 [ 2720.070728] free_unmap_vmap_area+0x63/0x70 [ 2720.070734] remove_vm_area+0xe7/0x110 [ 2720.070737] __vunmap+0x9c/0x400 [ 2720.070751] vfree+0x4d/0x100 [ 2720.070756] free_work+0x34/0x50 [libcfs] [ 2720.070782] process_one_work+0x2c8/0x700 [ 2720.070789] worker_thread+0x296/0x6e0 [ 2720.070794] ? rescuer_thread+0x570/0x570 [ 2720.070797] kthread+0x1d1/0x200 [ 2720.070801] ? set_kthread_struct+0x70/0x70 [ 2720.070805] ret_from_fork+0x1f/0x30 [ 2720.293870] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2720.331538] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2720.340621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2720.346504] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2720.350126] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2720.363757] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2720.369050] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2720.374133] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2720.378304] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2720.384011] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2720.389751] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2720.394254] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2720.400194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2720.404350] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2720.410920] Call Trace: [ 2720.412848] [ 2720.413904] ? show_regs.cold.9+0x22/0x2f [ 2720.417115] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2720.419509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2720.421700] ? watchdog+0x40/0x40 [ 2720.423159] ? hrtimer_interrupt+0x128/0x2c0 [ 2720.427271] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2720.430061] ? apic_timer_interrupt+0xf/0x20 [ 2720.432319] [ 2720.435815] ? on_each_cpu_cond_mask+0x46/0x150 [ 2720.437418] ? __x64_sys_fspick+0x210/0x210 [ 2720.444028] ? smp_call_function_many_cond+0x3db/0x540 [ 2720.447814] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2720.449856] ? __x64_sys_fspick+0x210/0x210 [ 2720.452872] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2720.457266] on_each_cpu_cond_mask+0x46/0x150 [ 2720.460093] on_each_cpu_cond+0x1c/0x30 [ 2720.461613] invalidate_bh_lrus+0x27/0x30 [ 2720.464208] invalidate_bdev+0x5a/0x90 [ 2720.465961] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2720.476785] generic_shutdown_super+0xb1/0x1b0 [ 2720.480914] kill_block_super+0x29/0x80 [ 2720.484938] deactivate_locked_super+0x52/0xd0 [ 2720.488438] deactivate_super+0x83/0x90 [ 2720.490630] cleanup_mnt+0x5f/0xe0 [ 2720.493540] __cleanup_mnt+0x16/0x20 [ 2720.496123] task_work_run+0xc6/0x110 [ 2720.500693] exit_to_usermode_loop+0x1dd/0x1f0 [ 2720.502200] do_syscall_64+0x3d6/0x440 [ 2720.504357] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2720.506579] RIP: 0033:0x7fbb019648fb [ 2720.509282] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2720.519412] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2720.525134] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2720.529603] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2720.534933] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2720.545816] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2720.550539] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2724.837305] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 366s! [ 2724.841372] Showing busy workqueues and worker pools: [ 2724.845719] workqueue events: flags=0x0 [ 2724.847840] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2724.851131] in-flight: 56293:free_work [libcfs] [ 2724.856903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2724.861735] in-flight: 22277:free_work [libcfs] [ 2724.864592] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2724.871796] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2724.877530] pending: nfs4_renew_state [nfsv4] [ 2724.880235] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2724.885589] in-flight: 47596:free_work [libcfs] [ 2724.889139] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2724.896859] workqueue events_unbound: flags=0x2 [ 2724.899511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2724.903665] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2724.908708] workqueue events_power_efficient: flags=0x80 [ 2724.911184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2724.914451] pending: neigh_periodic_work, do_cache_clean [ 2724.920866] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2724.924019] pending: check_lifetime [ 2724.925008] workqueue mm_percpu_wq: flags=0x8 [ 2724.930556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2724.936908] pending: vmstat_update [ 2724.938455] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2724.949407] pending: vmstat_update [ 2724.952393] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2724.959438] pending: vmstat_update [ 2724.960916] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2724.963281] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2724.970669] pending: cgroup_pidlist_destroy_work_fn [ 2724.974763] workqueue writeback: flags=0x4a [ 2724.977251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2724.982060] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2724.986108] workqueue xprtiod: flags=0xa [ 2724.989173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2724.994004] pending: xprt_autoclose [ 2724.999791] workqueue ipv6_addrconf: flags=0x40008 [ 2725.004135] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2725.006846] pending: addrconf_verify_work [ 2725.009363] workqueue knbd0-recv: flags=0x1a [ 2725.014085] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2725.020099] in-flight: 655:recv_work [ 2725.024457] workqueue ptlrpc_pinger: flags=0x2 [ 2725.026133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2725.031674] pending: ptlrpc_pinger_main [ptlrpc] [ 2725.036344] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=367s workers=5 idle: 53684 27131 54850 46584 [ 2725.044096] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=367s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2725.052798] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=367s workers=6 idle: 53660 37674 46417 53664 46418 [ 2725.066132] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2714s workers=2 idle: 657 [ 2736.059904] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 2736.063331] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2736.093543] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2736.097246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2736.101113] Workqueue: events free_work [libcfs] [ 2736.103550] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2736.105801] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2736.114561] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2736.118670] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2736.122148] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2736.126181] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2736.129937] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2736.135910] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2736.141111] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2736.147699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2736.150443] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2736.154680] Call Trace: [ 2736.156179] [ 2736.157897] ? show_regs.cold.9+0x22/0x2f [ 2736.161900] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2736.166530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2736.169812] ? watchdog+0x40/0x40 [ 2736.172927] ? hrtimer_interrupt+0x128/0x2c0 [ 2736.175373] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2736.177387] ? apic_timer_interrupt+0xf/0x20 [ 2736.179277] [ 2736.180123] ? on_each_cpu+0x51/0x120 [ 2736.181679] ? smp_call_function_many_cond+0x3db/0x540 [ 2736.183545] ? smp_call_function_many_cond+0x3a8/0x540 [ 2736.185910] ? load_new_mm_cr3+0x120/0x120 [ 2736.187672] ? load_new_mm_cr3+0x120/0x120 [ 2736.189159] on_each_cpu+0x51/0x120 [ 2736.190630] flush_tlb_kernel_range+0x93/0xb0 [ 2736.192606] free_unmap_vmap_area+0x63/0x70 [ 2736.194344] remove_vm_area+0xe7/0x110 [ 2736.195681] __vunmap+0x9c/0x400 [ 2736.196892] vfree+0x4d/0x100 [ 2736.198077] free_work+0x34/0x50 [libcfs] [ 2736.199863] process_one_work+0x2c8/0x700 [ 2736.201028] worker_thread+0x296/0x6e0 [ 2736.201993] ? rescuer_thread+0x570/0x570 [ 2736.204166] kthread+0x1d1/0x200 [ 2736.205415] ? set_kthread_struct+0x70/0x70 [ 2736.207027] ret_from_fork+0x1f/0x30 [ 2748.059623] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 2748.066642] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2748.103355] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2748.111993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2748.118676] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2748.124078] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2748.142026] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2748.145990] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2748.149539] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2748.153448] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2748.157641] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2748.161754] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2748.165254] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2748.169152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2748.172259] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2748.176038] Call Trace: [ 2748.177519] [ 2748.178524] ? show_regs.cold.9+0x22/0x2f [ 2748.180536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2748.183034] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2748.185209] ? watchdog+0x40/0x40 [ 2748.186731] ? hrtimer_interrupt+0x128/0x2c0 [ 2748.188275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2748.191315] ? apic_timer_interrupt+0xf/0x20 [ 2748.194204] [ 2748.195121] ? on_each_cpu_cond_mask+0x46/0x150 [ 2748.199509] ? __x64_sys_fspick+0x210/0x210 [ 2748.203089] ? smp_call_function_many_cond+0x3db/0x540 [ 2748.207164] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2748.208660] ? __x64_sys_fspick+0x210/0x210 [ 2748.211355] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2748.213538] on_each_cpu_cond_mask+0x46/0x150 [ 2748.215235] on_each_cpu_cond+0x1c/0x30 [ 2748.220594] invalidate_bh_lrus+0x27/0x30 [ 2748.222150] invalidate_bdev+0x5a/0x90 [ 2748.225303] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2748.227967] generic_shutdown_super+0xb1/0x1b0 [ 2748.229121] kill_block_super+0x29/0x80 [ 2748.231151] deactivate_locked_super+0x52/0xd0 [ 2748.233258] deactivate_super+0x83/0x90 [ 2748.235423] cleanup_mnt+0x5f/0xe0 [ 2748.237304] __cleanup_mnt+0x16/0x20 [ 2748.239402] task_work_run+0xc6/0x110 [ 2748.241266] exit_to_usermode_loop+0x1dd/0x1f0 [ 2748.242882] do_syscall_64+0x3d6/0x440 [ 2748.244140] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2748.246749] RIP: 0033:0x7fbb019648fb [ 2748.248661] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2748.260440] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2748.266085] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2748.270066] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2748.274721] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2748.279681] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2748.285312] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2752.055251] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 2752.062414] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2752.090340] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2752.094203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2752.097103] Workqueue: events free_work [libcfs] [ 2752.098752] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2752.100972] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2752.107242] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2752.110297] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2752.113594] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2752.117415] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2752.120111] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2752.123071] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2752.126059] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2752.129035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2752.130824] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2752.135312] Call Trace: [ 2752.136383] [ 2752.138167] ? show_regs.cold.9+0x22/0x2f [ 2752.140290] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2752.142951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2752.145117] ? watchdog+0x40/0x40 [ 2752.146331] ? hrtimer_interrupt+0x128/0x2c0 [ 2752.150988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2752.157298] ? apic_timer_interrupt+0xf/0x20 [ 2752.160144] [ 2752.165620] ? on_each_cpu+0x51/0x120 [ 2752.172926] ? smp_call_function_many_cond+0x3db/0x540 [ 2752.175974] ? smp_call_function_many_cond+0x3a8/0x540 [ 2752.179223] ? load_new_mm_cr3+0x120/0x120 [ 2752.181976] ? load_new_mm_cr3+0x120/0x120 [ 2752.183371] on_each_cpu+0x51/0x120 [ 2752.185136] flush_tlb_kernel_range+0x93/0xb0 [ 2752.186558] free_unmap_vmap_area+0x63/0x70 [ 2752.188178] remove_vm_area+0xe7/0x110 [ 2752.189943] __vunmap+0x9c/0x400 [ 2752.191232] vfree+0x4d/0x100 [ 2752.193174] free_work+0x34/0x50 [libcfs] [ 2752.195256] process_one_work+0x2c8/0x700 [ 2752.197520] worker_thread+0x296/0x6e0 [ 2752.199289] ? rescuer_thread+0x570/0x570 [ 2752.201759] kthread+0x1d1/0x200 [ 2752.203188] ? set_kthread_struct+0x70/0x70 [ 2752.205645] ret_from_fork+0x1f/0x30 [ 2755.552064] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 397s! [ 2755.555838] Showing busy workqueues and worker pools: [ 2755.558352] workqueue events: flags=0x0 [ 2755.560231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2755.563416] in-flight: 56293:free_work [libcfs] [ 2755.566205] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2755.569210] in-flight: 22277:free_work [libcfs] [ 2755.571214] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2755.575784] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2755.579294] pending: nfs4_renew_state [nfsv4] [ 2755.581823] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2755.584321] in-flight: 47596:free_work [libcfs] [ 2755.586822] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2755.591449] workqueue events_unbound: flags=0x2 [ 2755.593587] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2755.596351] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2755.600239] workqueue events_power_efficient: flags=0x80 [ 2755.602746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2755.605781] pending: neigh_periodic_work, do_cache_clean [ 2755.608464] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2755.613031] pending: check_lifetime [ 2755.615191] workqueue mm_percpu_wq: flags=0x8 [ 2755.620083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2755.627319] pending: vmstat_update [ 2755.628731] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2755.633358] pending: vmstat_update [ 2755.636149] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2755.642046] pending: vmstat_update [ 2755.645530] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2755.649613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2755.652982] pending: cgroup_pidlist_destroy_work_fn [ 2755.656597] workqueue writeback: flags=0x4a [ 2755.662157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2755.665340] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2755.667440] workqueue xprtiod: flags=0xa [ 2755.668686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2755.671471] pending: xprt_autoclose [ 2755.673430] workqueue ipv6_addrconf: flags=0x40008 [ 2755.675544] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2755.679603] pending: addrconf_verify_work [ 2755.682809] workqueue knbd0-recv: flags=0x1a [ 2755.688516] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2755.693135] in-flight: 655:recv_work [ 2755.695084] workqueue ptlrpc_pinger: flags=0x2 [ 2755.697179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2755.700212] pending: ptlrpc_pinger_main [ptlrpc] [ 2755.703036] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=397s workers=5 idle: 53684 27131 54850 46584 [ 2755.708370] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=397s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2755.715018] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=397s workers=6 idle: 53660 37674 46417 53664 46418 [ 2755.722425] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2745s workers=2 idle: 657 [ 2764.059025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 2764.066408] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2764.122809] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2764.129618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2764.133666] Workqueue: events free_work [libcfs] [ 2764.135698] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2764.137944] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2764.147357] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2764.151668] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2764.154690] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2764.157746] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2764.160844] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2764.164062] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2764.168652] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2764.172835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2764.175736] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2764.180384] Call Trace: [ 2764.181434] [ 2764.182605] ? show_regs.cold.9+0x22/0x2f [ 2764.186074] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2764.190172] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2764.193234] ? watchdog+0x40/0x40 [ 2764.195051] ? hrtimer_interrupt+0x128/0x2c0 [ 2764.197205] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2764.199931] ? apic_timer_interrupt+0xf/0x20 [ 2764.202197] [ 2764.203174] ? on_each_cpu+0x51/0x120 [ 2764.204724] ? smp_call_function_many_cond+0x3db/0x540 [ 2764.206865] ? smp_call_function_many_cond+0x3a8/0x540 [ 2764.208897] ? load_new_mm_cr3+0x120/0x120 [ 2764.210401] ? load_new_mm_cr3+0x120/0x120 [ 2764.211911] on_each_cpu+0x51/0x120 [ 2764.213086] flush_tlb_kernel_range+0x93/0xb0 [ 2764.215116] free_unmap_vmap_area+0x63/0x70 [ 2764.216802] remove_vm_area+0xe7/0x110 [ 2764.218666] __vunmap+0x9c/0x400 [ 2764.220012] vfree+0x4d/0x100 [ 2764.221208] free_work+0x34/0x50 [libcfs] [ 2764.223068] process_one_work+0x2c8/0x700 [ 2764.224857] worker_thread+0x296/0x6e0 [ 2764.226531] ? rescuer_thread+0x570/0x570 [ 2764.228321] kthread+0x1d1/0x200 [ 2764.230357] ? set_kthread_struct+0x70/0x70 [ 2764.232222] ret_from_fork+0x1f/0x30 [ 2776.058558] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2776.063099] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2776.094039] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2776.098691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2776.104459] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2776.107575] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2776.123014] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2776.130578] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2776.134723] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2776.144636] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2776.151607] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2776.153392] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2776.160059] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2776.163712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2776.165945] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2776.168347] Call Trace: [ 2776.169464] [ 2776.170412] ? show_regs.cold.9+0x22/0x2f [ 2776.172311] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2776.174385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2776.176541] ? watchdog+0x40/0x40 [ 2776.178051] ? hrtimer_interrupt+0x128/0x2c0 [ 2776.179750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2776.181728] ? apic_timer_interrupt+0xf/0x20 [ 2776.183378] [ 2776.184318] ? on_each_cpu_cond_mask+0x46/0x150 [ 2776.186143] ? __x64_sys_fspick+0x210/0x210 [ 2776.187870] ? smp_call_function_many_cond+0x3db/0x540 [ 2776.189877] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2776.191329] ? __x64_sys_fspick+0x210/0x210 [ 2776.192751] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2776.194525] on_each_cpu_cond_mask+0x46/0x150 [ 2776.196340] on_each_cpu_cond+0x1c/0x30 [ 2776.197869] invalidate_bh_lrus+0x27/0x30 [ 2776.199581] invalidate_bdev+0x5a/0x90 [ 2776.201108] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2776.203355] generic_shutdown_super+0xb1/0x1b0 [ 2776.204865] kill_block_super+0x29/0x80 [ 2776.208322] deactivate_locked_super+0x52/0xd0 [ 2776.213721] deactivate_super+0x83/0x90 [ 2776.217549] cleanup_mnt+0x5f/0xe0 [ 2776.219841] __cleanup_mnt+0x16/0x20 [ 2776.224698] task_work_run+0xc6/0x110 [ 2776.230991] exit_to_usermode_loop+0x1dd/0x1f0 [ 2776.238856] do_syscall_64+0x3d6/0x440 [ 2776.241956] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2776.245925] RIP: 0033:0x7fbb019648fb [ 2776.249675] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2776.261770] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2776.265637] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2776.270804] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2776.278772] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2776.281777] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2776.286663] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2777.743129] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 2777.750277] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 2777.759760] (detected by 2, t=420013 jiffies, g=626849, q=1398) [ 2777.764596] Sending NMI from CPU 2 to CPUs 3: [ 2777.769003] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 2777.769013] rcu: rcu_sched kthread starved for 419655 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2777.786080] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2777.792718] rcu: RCU grace-period kthread stack dump: [ 2777.794782] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2777.804790] Call Trace: [ 2777.809633] __schedule+0x351/0xcb0 [ 2777.813830] schedule+0xc0/0x180 [ 2777.819652] schedule_timeout+0xb4/0x190 [ 2777.825146] ? do_raw_spin_unlock+0x75/0x190 [ 2777.831215] ? __next_timer_interrupt+0x160/0x160 [ 2777.837006] ? prepare_to_swait_event+0xf3/0x140 [ 2777.841029] rcu_gp_kthread+0x8dd/0x15b0 [ 2777.845292] ? force_qs_rnp+0x2c0/0x2c0 [ 2777.848844] kthread+0x1d1/0x200 [ 2777.850819] ? set_kthread_struct+0x70/0x70 [ 2777.854081] ret_from_fork+0x1f/0x30 [ 2777.856384] rcu: Stack dump where RCU GP kthread last ran: [ 2777.860328] NMI backtrace for cpu 2 [ 2777.864622] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2777.875475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2777.884704] Workqueue: events free_work [libcfs] [ 2777.889573] Call Trace: [ 2777.890242] [ 2777.891661] ? dump_stack+0xbb/0x10e [ 2777.899592] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2777.906577] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2777.910915] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2777.912849] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2777.917510] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2777.924083] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2777.928374] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 2777.931499] ? update_process_times+0x6d/0xc0 [ 2777.934295] ? tick_sched_handle.isra.18+0x27/0x90 [ 2777.937222] ? tick_sched_timer+0x53/0xb0 [ 2777.942076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2777.950373] ? tick_sched_handle.isra.18+0x90/0x90 [ 2777.952643] ? hrtimer_interrupt+0x128/0x2c0 [ 2777.954657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2777.957038] ? apic_timer_interrupt+0xf/0x20 [ 2777.959079] [ 2777.960340] ? on_each_cpu+0x51/0x120 [ 2777.963694] ? smp_call_function_many_cond+0x3db/0x540 [ 2777.966685] ? smp_call_function_many_cond+0x3a8/0x540 [ 2777.969884] ? load_new_mm_cr3+0x120/0x120 [ 2777.972560] ? load_new_mm_cr3+0x120/0x120 [ 2777.975667] ? on_each_cpu+0x51/0x120 [ 2777.977812] ? flush_tlb_kernel_range+0x93/0xb0 [ 2777.979519] ? free_unmap_vmap_area+0x63/0x70 [ 2777.982803] ? remove_vm_area+0xe7/0x110 [ 2777.985230] ? __vunmap+0x9c/0x400 [ 2777.987576] ? vfree+0x4d/0x100 [ 2777.991814] ? free_work+0x34/0x50 [libcfs] [ 2777.994392] ? process_one_work+0x2c8/0x700 [ 2777.997662] ? worker_thread+0x296/0x6e0 [ 2778.001372] ? rescuer_thread+0x570/0x570 [ 2778.004389] ? kthread+0x1d1/0x200 [ 2778.007227] ? set_kthread_struct+0x70/0x70 [ 2778.008464] ? ret_from_fork+0x1f/0x30 [ 2780.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 2780.058734] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2780.089478] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2780.101023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2780.108776] Workqueue: events free_work [libcfs] [ 2780.115897] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2780.120013] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2780.138783] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2780.144579] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2780.152539] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2780.157602] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2780.161747] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2780.169419] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2780.175859] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2780.185740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2780.190491] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2780.199140] Call Trace: [ 2780.200297] [ 2780.201109] ? show_regs.cold.9+0x22/0x2f [ 2780.208207] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2780.212191] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2780.218509] ? watchdog+0x40/0x40 [ 2780.220321] ? hrtimer_interrupt+0x128/0x2c0 [ 2780.221996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2780.225514] ? apic_timer_interrupt+0xf/0x20 [ 2780.227892] [ 2780.229478] ? on_each_cpu+0x51/0x120 [ 2780.231429] ? smp_call_function_many_cond+0x3d9/0x540 [ 2780.233752] ? smp_call_function_many_cond+0x3a8/0x540 [ 2780.236403] ? load_new_mm_cr3+0x120/0x120 [ 2780.238793] ? load_new_mm_cr3+0x120/0x120 [ 2780.241904] on_each_cpu+0x51/0x120 [ 2780.246970] flush_tlb_kernel_range+0x93/0xb0 [ 2780.249342] free_unmap_vmap_area+0x63/0x70 [ 2780.251519] remove_vm_area+0xe7/0x110 [ 2780.253617] __vunmap+0x9c/0x400 [ 2780.256257] vfree+0x4d/0x100 [ 2780.258264] free_work+0x34/0x50 [libcfs] [ 2780.260733] process_one_work+0x2c8/0x700 [ 2780.264958] worker_thread+0x296/0x6e0 [ 2780.269674] ? rescuer_thread+0x570/0x570 [ 2780.272074] kthread+0x1d1/0x200 [ 2780.274540] ? set_kthread_struct+0x70/0x70 [ 2780.277620] ret_from_fork+0x1f/0x30 [ 2785.760574] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 427s! [ 2785.767200] Showing busy workqueues and worker pools: [ 2785.770364] workqueue events: flags=0x0 [ 2785.772900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2785.777220] in-flight: 56293:free_work [libcfs] [ 2785.780364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2785.784604] in-flight: 22277:free_work [libcfs] [ 2785.787186] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2785.794475] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2785.798212] pending: nfs4_renew_state [nfsv4] [ 2785.800420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2785.804456] in-flight: 47596:free_work [libcfs] [ 2785.809400] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2785.817459] workqueue events_unbound: flags=0x2 [ 2785.821360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2785.823935] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2785.827296] workqueue events_power_efficient: flags=0x80 [ 2785.829499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2785.832664] pending: neigh_periodic_work, do_cache_clean [ 2785.835432] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2785.838576] pending: check_lifetime [ 2785.840858] workqueue mm_percpu_wq: flags=0x8 [ 2785.843434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2785.846853] pending: vmstat_update [ 2785.848371] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2785.852351] pending: vmstat_update [ 2785.854388] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2785.858408] pending: vmstat_update [ 2785.860441] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2785.863349] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2785.866293] pending: cgroup_pidlist_destroy_work_fn [ 2785.868956] workqueue writeback: flags=0x4a [ 2785.871204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2785.874370] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2785.877578] workqueue xprtiod: flags=0xa [ 2785.879427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2785.882353] pending: xprt_autoclose [ 2785.884216] workqueue ipv6_addrconf: flags=0x40008 [ 2785.888376] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2785.892413] pending: addrconf_verify_work [ 2785.895279] workqueue knbd0-recv: flags=0x1a [ 2785.897841] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2785.900133] in-flight: 655:recv_work [ 2785.902043] workqueue ptlrpc_pinger: flags=0x2 [ 2785.904235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2785.912391] pending: ptlrpc_pinger_main [ptlrpc] [ 2785.919914] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=428s workers=5 idle: 53684 27131 54850 46584 [ 2785.925786] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=428s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2785.931631] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=428s workers=6 idle: 53660 37674 46417 53664 46418 [ 2785.940245] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2775s workers=2 idle: 657 [ 2804.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2804.059006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2804.062284] Modules linked in: [ 2804.065557] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2804.065778] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2804.065784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2804.065791] Workqueue: events free_work [libcfs] [ 2804.068064] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2804.068140] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2804.068146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2804.068149] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2804.068160] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2804.068164] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2804.068168] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2804.068170] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2804.068172] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2804.068174] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2804.068176] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2804.068179] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2804.068182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2804.068183] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2804.068191] Call Trace: [ 2804.068195] [ 2804.068200] ? show_regs.cold.9+0x22/0x2f [ 2804.068208] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2804.068214] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2804.095495] [ 2804.095502] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2804.095516] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2804.095520] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2804.095524] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2804.095526] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2804.095528] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2804.095530] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2804.095532] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2804.095535] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2804.095537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2804.095540] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2804.095547] Call Trace: [ 2804.095551] [ 2804.095555] ? show_regs.cold.9+0x22/0x2f [ 2804.095563] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2804.095568] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2804.095583] ? watchdog+0x40/0x40 [ 2804.095587] ? hrtimer_interrupt+0x128/0x2c0 [ 2804.095590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2804.095597] ? apic_timer_interrupt+0xf/0x20 [ 2804.095601] [ 2804.095603] ? on_each_cpu+0x51/0x120 [ 2804.095607] ? smp_call_function_many_cond+0x3db/0x540 [ 2804.095610] ? smp_call_function_many_cond+0x3a8/0x540 [ 2804.101837] ? watchdog+0x40/0x40 [ 2804.101854] ? hrtimer_interrupt+0x128/0x2c0 [ 2804.101858] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2804.101865] ? apic_timer_interrupt+0xf/0x20 [ 2804.101869] [ 2804.101873] ? on_each_cpu_cond_mask+0x46/0x150 [ 2804.101877] ? __x64_sys_fspick+0x210/0x210 [ 2804.101884] ? smp_call_function_many_cond+0x3d9/0x540 [ 2804.101887] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2804.101890] ? __x64_sys_fspick+0x210/0x210 [ 2804.101894] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2804.101897] on_each_cpu_cond_mask+0x46/0x150 [ 2804.101902] on_each_cpu_cond+0x1c/0x30 [ 2804.101904] invalidate_bh_lrus+0x27/0x30 [ 2804.101907] invalidate_bdev+0x5a/0x90 [ 2804.101915] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2804.102030] generic_shutdown_super+0xb1/0x1b0 [ 2804.102038] kill_block_super+0x29/0x80 [ 2804.102043] deactivate_locked_super+0x52/0xd0 [ 2804.102047] deactivate_super+0x83/0x90 [ 2804.102051] cleanup_mnt+0x5f/0xe0 [ 2804.102057] __cleanup_mnt+0x16/0x20 [ 2804.102060] task_work_run+0xc6/0x110 [ 2804.102067] exit_to_usermode_loop+0x1dd/0x1f0 [ 2804.102074] do_syscall_64+0x3d6/0x440 [ 2804.102078] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2804.102084] RIP: 0033:0x7fbb019648fb [ 2804.102092] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2804.102096] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2804.102101] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2804.102103] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2804.102105] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2804.102106] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2804.102109] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2804.318192] ? load_new_mm_cr3+0x120/0x120 [ 2804.320351] ? load_new_mm_cr3+0x120/0x120 [ 2804.322526] on_each_cpu+0x51/0x120 [ 2804.324874] flush_tlb_kernel_range+0x93/0xb0 [ 2804.326273] free_unmap_vmap_area+0x63/0x70 [ 2804.327418] remove_vm_area+0xe7/0x110 [ 2804.329541] __vunmap+0x9c/0x400 [ 2804.331530] vfree+0x4d/0x100 [ 2804.333468] free_work+0x34/0x50 [libcfs] [ 2804.337743] process_one_work+0x2c8/0x700 [ 2804.338716] worker_thread+0x296/0x6e0 [ 2804.339960] ? rescuer_thread+0x570/0x570 [ 2804.341066] kthread+0x1d1/0x200 [ 2804.341978] ? set_kthread_struct+0x70/0x70 [ 2804.343111] ret_from_fork+0x1f/0x30 [ 2812.055544] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2812.063180] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2812.112315] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2812.125893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2812.137434] Workqueue: events free_work [libcfs] [ 2812.139236] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2812.148101] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2812.165280] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2812.172738] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2812.179793] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2812.184702] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2812.189538] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2812.196587] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2812.201091] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2812.207537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2812.211185] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2812.219047] Call Trace: [ 2812.220136] [ 2812.221282] ? show_regs.cold.9+0x22/0x2f [ 2812.223695] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2812.227023] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2812.228865] ? watchdog+0x40/0x40 [ 2812.231150] ? hrtimer_interrupt+0x128/0x2c0 [ 2812.236893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2812.239376] ? apic_timer_interrupt+0xf/0x20 [ 2812.243172] [ 2812.245742] ? on_each_cpu+0x51/0x120 [ 2812.250251] ? smp_call_function_many_cond+0x3d9/0x540 [ 2812.258723] ? smp_call_function_many_cond+0x3a8/0x540 [ 2812.265936] ? load_new_mm_cr3+0x120/0x120 [ 2812.267580] ? load_new_mm_cr3+0x120/0x120 [ 2812.270261] on_each_cpu+0x51/0x120 [ 2812.273932] flush_tlb_kernel_range+0x93/0xb0 [ 2812.276863] free_unmap_vmap_area+0x63/0x70 [ 2812.279504] remove_vm_area+0xe7/0x110 [ 2812.281547] __vunmap+0x9c/0x400 [ 2812.283517] vfree+0x4d/0x100 [ 2812.285554] free_work+0x34/0x50 [libcfs] [ 2812.290821] process_one_work+0x2c8/0x700 [ 2812.295286] worker_thread+0x296/0x6e0 [ 2812.296879] ? rescuer_thread+0x570/0x570 [ 2812.298508] kthread+0x1d1/0x200 [ 2812.301660] ? set_kthread_struct+0x70/0x70 [ 2812.303266] ret_from_fork+0x1f/0x30 [ 2815.967073] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 457s! [ 2815.971040] Showing busy workqueues and worker pools: [ 2815.975114] workqueue events: flags=0x0 [ 2815.976921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2815.981817] in-flight: 56293:free_work [libcfs] [ 2815.984613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2815.988206] in-flight: 22277:free_work [libcfs] [ 2815.990342] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2815.997294] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2816.002685] pending: nfs4_renew_state [nfsv4] [ 2816.005657] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2816.011783] in-flight: 47596:free_work [libcfs] [ 2816.018044] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2816.024941] workqueue events_unbound: flags=0x2 [ 2816.028716] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2816.032928] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2816.040724] workqueue events_power_efficient: flags=0x80 [ 2816.047470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2816.051348] pending: neigh_periodic_work, do_cache_clean [ 2816.056276] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2816.066667] pending: check_lifetime [ 2816.069867] workqueue mm_percpu_wq: flags=0x8 [ 2816.072812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2816.077079] pending: vmstat_update [ 2816.079453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2816.083128] pending: vmstat_update [ 2816.085296] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2816.091595] pending: vmstat_update [ 2816.095183] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2816.098495] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2816.102872] pending: cgroup_pidlist_destroy_work_fn [ 2816.107135] workqueue writeback: flags=0x4a [ 2816.113967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2816.118578] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2816.122445] workqueue xprtiod: flags=0xa [ 2816.126025] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2816.130085] pending: xprt_autoclose [ 2816.132737] workqueue ipv6_addrconf: flags=0x40008 [ 2816.137262] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2816.141477] pending: addrconf_verify_work [ 2816.148657] workqueue knbd0-recv: flags=0x1a [ 2816.151872] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2816.156224] in-flight: 655:recv_work [ 2816.157951] workqueue ptlrpc_pinger: flags=0x2 [ 2816.160844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2816.165515] pending: ptlrpc_pinger_main [ptlrpc] [ 2816.170747] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=458s workers=5 idle: 53684 27131 54850 46584 [ 2816.177151] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=458s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2816.189763] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=458s workers=6 idle: 53660 37674 46417 53664 46418 [ 2816.195331] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2806s workers=2 idle: 657 [ 2832.058767] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2832.059008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2832.064020] Modules linked in: [ 2832.070614] Modules linked in: zfs(O) spl(O) [ 2832.073425] zfs(O) spl(O) lustre(O) [ 2832.076871] lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 2832.079287] osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2832.079357] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2832.079363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2832.079366] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2832.079376] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2832.079381] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2832.086787] osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2832.086843] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2832.086850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2832.086856] Workqueue: events free_work [libcfs] [ 2832.086896] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2832.086905] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2832.086909] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2832.086913] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2832.086916] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2832.086917] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2832.086919] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2832.086921] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2832.086924] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2832.086926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2832.086928] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2832.086936] Call Trace: [ 2832.086941] [ 2832.086946] ? show_regs.cold.9+0x22/0x2f [ 2832.086955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2832.086961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2832.086965] ? watchdog+0x40/0x40 [ 2832.113866] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2832.113876] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2832.113878] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2832.113880] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2832.113883] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2832.113885] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2832.113888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2832.113890] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2832.113898] Call Trace: [ 2832.113904] [ 2832.113910] ? show_regs.cold.9+0x22/0x2f [ 2832.113923] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2832.113930] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2832.113934] ? watchdog+0x40/0x40 [ 2832.113938] ? hrtimer_interrupt+0x128/0x2c0 [ 2832.113941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2832.113949] ? apic_timer_interrupt+0xf/0x20 [ 2832.113954] [ 2832.113956] ? on_each_cpu_cond_mask+0x46/0x150 [ 2832.113961] ? __x64_sys_fspick+0x210/0x210 [ 2832.113980] ? smp_call_function_many_cond+0x3db/0x540 [ 2832.113985] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2832.113988] ? __x64_sys_fspick+0x210/0x210 [ 2832.113993] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2832.113995] on_each_cpu_cond_mask+0x46/0x150 [ 2832.113999] on_each_cpu_cond+0x1c/0x30 [ 2832.114003] invalidate_bh_lrus+0x27/0x30 [ 2832.114006] invalidate_bdev+0x5a/0x90 [ 2832.114014] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2832.114135] generic_shutdown_super+0xb1/0x1b0 [ 2832.114156] kill_block_super+0x29/0x80 [ 2832.114161] deactivate_locked_super+0x52/0xd0 [ 2832.114166] deactivate_super+0x83/0x90 [ 2832.114171] cleanup_mnt+0x5f/0xe0 [ 2832.114177] __cleanup_mnt+0x16/0x20 [ 2832.114180] task_work_run+0xc6/0x110 [ 2832.114197] exit_to_usermode_loop+0x1dd/0x1f0 [ 2832.114205] do_syscall_64+0x3d6/0x440 [ 2832.114211] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2832.114217] RIP: 0033:0x7fbb019648fb [ 2832.114223] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2832.114227] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2832.114234] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2832.114236] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2832.114237] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2832.114239] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2832.114241] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2832.388729] ? hrtimer_interrupt+0x128/0x2c0 [ 2832.390710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2832.394285] ? apic_timer_interrupt+0xf/0x20 [ 2832.397300] [ 2832.398082] ? on_each_cpu+0x51/0x120 [ 2832.400251] ? smp_call_function_many_cond+0x3db/0x540 [ 2832.402719] ? smp_call_function_many_cond+0x3a8/0x540 [ 2832.405342] ? load_new_mm_cr3+0x120/0x120 [ 2832.407335] ? load_new_mm_cr3+0x120/0x120 [ 2832.408367] on_each_cpu+0x51/0x120 [ 2832.409743] flush_tlb_kernel_range+0x93/0xb0 [ 2832.411326] free_unmap_vmap_area+0x63/0x70 [ 2832.413080] remove_vm_area+0xe7/0x110 [ 2832.414420] __vunmap+0x9c/0x400 [ 2832.419876] vfree+0x4d/0x100 [ 2832.422721] free_work+0x34/0x50 [libcfs] [ 2832.425783] process_one_work+0x2c8/0x700 [ 2832.428322] worker_thread+0x296/0x6e0 [ 2832.430642] ? rescuer_thread+0x570/0x570 [ 2832.433251] kthread+0x1d1/0x200 [ 2832.436880] ? set_kthread_struct+0x70/0x70 [ 2832.439460] ret_from_fork+0x1f/0x30 [ 2844.058971] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2844.065758] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2844.101124] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2844.108414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2844.113355] Workqueue: events free_work [libcfs] [ 2844.116187] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2844.121428] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2844.135847] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2844.141364] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2844.152031] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2844.161346] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2844.167886] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2844.173613] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2844.177126] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2844.184289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2844.192586] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2844.200521] Call Trace: [ 2844.204558] [ 2844.205743] ? show_regs.cold.9+0x22/0x2f [ 2844.213378] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2844.220365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2844.224683] ? watchdog+0x40/0x40 [ 2844.227544] ? hrtimer_interrupt+0x128/0x2c0 [ 2844.231381] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2844.235589] ? apic_timer_interrupt+0xf/0x20 [ 2844.237363] [ 2844.239350] ? on_each_cpu+0x51/0x120 [ 2844.242508] ? smp_call_function_many_cond+0x3db/0x540 [ 2844.247509] ? smp_call_function_many_cond+0x3a8/0x540 [ 2844.255528] ? load_new_mm_cr3+0x120/0x120 [ 2844.262630] ? load_new_mm_cr3+0x120/0x120 [ 2844.267492] on_each_cpu+0x51/0x120 [ 2844.268907] flush_tlb_kernel_range+0x93/0xb0 [ 2844.280592] free_unmap_vmap_area+0x63/0x70 [ 2844.284853] remove_vm_area+0xe7/0x110 [ 2844.286426] __vunmap+0x9c/0x400 [ 2844.287748] vfree+0x4d/0x100 [ 2844.291065] free_work+0x34/0x50 [libcfs] [ 2844.293628] process_one_work+0x2c8/0x700 [ 2844.302163] worker_thread+0x296/0x6e0 [ 2844.303625] ? rescuer_thread+0x570/0x570 [ 2844.311568] kthread+0x1d1/0x200 [ 2844.315530] ? set_kthread_struct+0x70/0x70 [ 2844.319028] ret_from_fork+0x1f/0x30 [ 2846.688734] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 488s! [ 2846.695647] Showing busy workqueues and worker pools: [ 2846.696980] workqueue events: flags=0x0 [ 2846.698013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2846.701892] in-flight: 56293:free_work [libcfs] [ 2846.703590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2846.707300] in-flight: 22277:free_work [libcfs] [ 2846.711166] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2846.718572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2846.726198] pending: nfs4_renew_state [nfsv4] [ 2846.730445] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2846.733619] in-flight: 47596:free_work [libcfs] [ 2846.739549] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2846.744591] workqueue events_unbound: flags=0x2 [ 2846.748353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2846.752427] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2846.757322] workqueue events_power_efficient: flags=0x80 [ 2846.760798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2846.764703] pending: neigh_periodic_work, do_cache_clean [ 2846.777452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2846.782222] pending: check_lifetime [ 2846.786009] workqueue mm_percpu_wq: flags=0x8 [ 2846.792416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2846.801946] pending: vmstat_update [ 2846.805385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2846.809448] pending: vmstat_update [ 2846.812265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2846.816881] pending: vmstat_update [ 2846.821313] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2846.825208] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2846.830087] pending: cgroup_pidlist_destroy_work_fn [ 2846.833564] workqueue writeback: flags=0x4a [ 2846.836321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2846.844063] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2846.848533] workqueue xprtiod: flags=0xa [ 2846.852118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2846.855080] pending: xprt_autoclose [ 2846.856552] workqueue ipv6_addrconf: flags=0x40008 [ 2846.860470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2846.862417] pending: addrconf_verify_work [ 2846.871295] workqueue knbd0-recv: flags=0x1a [ 2846.880049] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2846.884271] in-flight: 655:recv_work [ 2846.889619] workqueue ptlrpc_pinger: flags=0x2 [ 2846.892976] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2846.897042] pending: ptlrpc_pinger_main [ptlrpc] [ 2846.905602] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=489s workers=5 idle: 53684 27131 54850 46584 [ 2846.912561] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=489s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2846.922356] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=489s workers=6 idle: 53660 37674 46417 53664 46418 [ 2846.928188] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2836s workers=2 idle: 657 [ 2860.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2860.059007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2860.063515] Modules linked in: [ 2860.066342] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) [ 2860.068710] zfs(O) spl(O) [ 2860.071299] mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 2860.073927] lustre(O) osp(O) ofd(O) [ 2860.086338] sha512_generic [ 2860.086351] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2860.086358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2860.086434] Workqueue: events free_work [libcfs] [ 2860.088815] lod(O) [ 2860.090025] [ 2860.100210] ost(O) [ 2860.102854] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2860.104665] mdt(O) [ 2860.105269] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2860.105904] mdd(O) [ 2860.106582] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 [ 2860.108695] mgs(O) [ 2860.109354] ORIG_RAX: ffffffffffffff13 [ 2860.115196] osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 [ 2860.116086] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2860.117683] auth_rpcgss [ 2860.118748] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2860.119922] nfsv4 [ 2860.126866] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2860.129723] dns_resolver [ 2860.130825] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2860.133955] intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2860.133992] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2860.133999] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2860.134001] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2860.134013] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2860.134018] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2860.134022] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2860.134024] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2860.134026] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2860.134028] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2860.134030] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2860.134033] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2860.134035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2860.134037] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2860.134045] Call Trace: [ 2860.134050] [ 2860.134055] ? show_regs.cold.9+0x22/0x2f [ 2860.134066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2860.134758] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2860.136945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2860.138009] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2860.140410] ? watchdog+0x40/0x40 [ 2860.150084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2860.153333] ? hrtimer_interrupt+0x128/0x2c0 [ 2860.156681] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2860.158747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2860.166239] Call Trace: [ 2860.168942] ? apic_timer_interrupt+0xf/0x20 [ 2860.171564] [ 2860.173702] [ 2860.176256] ? show_regs.cold.9+0x22/0x2f [ 2860.178581] ? on_each_cpu_cond_mask+0x46/0x150 [ 2860.180974] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2860.183960] ? __x64_sys_fspick+0x210/0x210 [ 2860.186361] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2860.188691] ? smp_call_function_many_cond+0x3db/0x540 [ 2860.189652] ? watchdog+0x40/0x40 [ 2860.190203] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2860.191893] ? hrtimer_interrupt+0x128/0x2c0 [ 2860.193304] ? __x64_sys_fspick+0x210/0x210 [ 2860.196053] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2860.197516] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2860.201014] ? apic_timer_interrupt+0xf/0x20 [ 2860.202052] on_each_cpu_cond_mask+0x46/0x150 [ 2860.204389] [ 2860.204399] ? on_each_cpu+0x51/0x120 [ 2860.204410] ? smp_call_function_many_cond+0x3db/0x540 [ 2860.204414] ? smp_call_function_many_cond+0x3a8/0x540 [ 2860.204418] ? load_new_mm_cr3+0x120/0x120 [ 2860.204422] ? load_new_mm_cr3+0x120/0x120 [ 2860.204426] on_each_cpu+0x51/0x120 [ 2860.204431] flush_tlb_kernel_range+0x93/0xb0 [ 2860.205771] on_each_cpu_cond+0x1c/0x30 [ 2860.208506] free_unmap_vmap_area+0x63/0x70 [ 2860.209980] invalidate_bh_lrus+0x27/0x30 [ 2860.211000] remove_vm_area+0xe7/0x110 [ 2860.212253] invalidate_bdev+0x5a/0x90 [ 2860.213146] __vunmap+0x9c/0x400 [ 2860.213879] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2860.215479] vfree+0x4d/0x100 [ 2860.216796] generic_shutdown_super+0xb1/0x1b0 [ 2860.218673] free_work+0x34/0x50 [libcfs] [ 2860.219974] kill_block_super+0x29/0x80 [ 2860.221838] process_one_work+0x2c8/0x700 [ 2860.223440] deactivate_locked_super+0x52/0xd0 [ 2860.224903] worker_thread+0x296/0x6e0 [ 2860.226270] deactivate_super+0x83/0x90 [ 2860.228188] ? rescuer_thread+0x570/0x570 [ 2860.229543] cleanup_mnt+0x5f/0xe0 [ 2860.231240] kthread+0x1d1/0x200 [ 2860.232596] __cleanup_mnt+0x16/0x20 [ 2860.234037] ? set_kthread_struct+0x70/0x70 [ 2860.235305] task_work_run+0xc6/0x110 [ 2860.236217] ret_from_fork+0x1f/0x30 [ 2860.237443] exit_to_usermode_loop+0x1dd/0x1f0 [ 2860.273970] do_syscall_64+0x3d6/0x440 [ 2860.275101] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2860.276690] RIP: 0033:0x7fbb019648fb [ 2860.277841] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2860.283268] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2860.285526] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2860.288615] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2860.297483] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2860.302481] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2860.306456] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2872.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2872.063551] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2872.123303] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2872.131915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2872.139012] Workqueue: events free_work [libcfs] [ 2872.145567] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2872.148857] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2872.170375] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2872.175741] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2872.183405] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2872.188722] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2872.194717] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2872.204895] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2872.209222] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2872.214009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2872.218934] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2872.224015] Call Trace: [ 2872.225676] [ 2872.226939] ? show_regs.cold.9+0x22/0x2f [ 2872.230256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2872.234942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2872.238940] ? watchdog+0x40/0x40 [ 2872.240869] ? hrtimer_interrupt+0x128/0x2c0 [ 2872.246627] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2872.257391] ? apic_timer_interrupt+0xf/0x20 [ 2872.260675] [ 2872.262388] ? on_each_cpu+0x51/0x120 [ 2872.265068] ? smp_call_function_many_cond+0x3db/0x540 [ 2872.268495] ? smp_call_function_many_cond+0x3a8/0x540 [ 2872.273046] ? load_new_mm_cr3+0x120/0x120 [ 2872.276094] ? load_new_mm_cr3+0x120/0x120 [ 2872.278664] on_each_cpu+0x51/0x120 [ 2872.280728] flush_tlb_kernel_range+0x93/0xb0 [ 2872.283779] free_unmap_vmap_area+0x63/0x70 [ 2872.284961] remove_vm_area+0xe7/0x110 [ 2872.287425] __vunmap+0x9c/0x400 [ 2872.288866] vfree+0x4d/0x100 [ 2872.290082] free_work+0x34/0x50 [libcfs] [ 2872.298120] process_one_work+0x2c8/0x700 [ 2872.300193] worker_thread+0x296/0x6e0 [ 2872.301759] ? rescuer_thread+0x570/0x570 [ 2872.304677] kthread+0x1d1/0x200 [ 2872.306453] ? set_kthread_struct+0x70/0x70 [ 2872.309252] ret_from_fork+0x1f/0x30 [ 2877.407024] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 519s! [ 2877.409259] Showing busy workqueues and worker pools: [ 2877.410508] workqueue events: flags=0x0 [ 2877.411488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2877.413306] in-flight: 56293:free_work [libcfs] [ 2877.414620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2877.416576] in-flight: 22277:free_work [libcfs] [ 2877.418072] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2877.420844] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2877.423793] pending: nfs4_renew_state [nfsv4] [ 2877.425499] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2877.427392] in-flight: 47596:free_work [libcfs] [ 2877.428741] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2877.434317] workqueue events_unbound: flags=0x2 [ 2877.436179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2877.437763] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2877.440543] workqueue events_power_efficient: flags=0x80 [ 2877.442170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2877.444241] pending: neigh_periodic_work, do_cache_clean [ 2877.445864] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2877.447750] pending: check_lifetime [ 2877.448766] workqueue mm_percpu_wq: flags=0x8 [ 2877.449857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2877.451574] pending: vmstat_update [ 2877.452615] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2877.454567] pending: vmstat_update [ 2877.455722] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2877.458169] pending: vmstat_update [ 2877.459202] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2877.460741] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2877.462400] pending: cgroup_pidlist_destroy_work_fn [ 2877.463807] workqueue writeback: flags=0x4a [ 2877.464900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2877.466820] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2877.468598] workqueue xprtiod: flags=0xa [ 2877.469617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2877.471556] pending: xprt_autoclose [ 2877.472823] workqueue ipv6_addrconf: flags=0x40008 [ 2877.474154] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2877.475775] pending: addrconf_verify_work [ 2877.476927] workqueue knbd0-recv: flags=0x1a [ 2877.478131] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2877.479814] in-flight: 655:recv_work [ 2877.480851] workqueue ptlrpc_pinger: flags=0x2 [ 2877.482262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2877.483972] pending: ptlrpc_pinger_main [ptlrpc] [ 2877.485885] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=519s workers=5 idle: 53684 27131 54850 46584 [ 2877.488298] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=519s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2877.491256] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=519s workers=6 idle: 53660 37674 46417 53664 46418 [ 2877.493828] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2867s workers=2 idle: 657 [ 2888.058015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2888.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2888.061751] Modules linked in: zfs(O) [ 2888.069569] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [ 2888.071427] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2888.071598] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2888.071604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2888.071607] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2888.071620] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2888.071624] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2888.071629] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2888.071631] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2888.071633] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2888.071635] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2888.071637] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2888.071640] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2888.071642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2888.071644] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2888.071653] Call Trace: [ 2888.071658] [ 2888.071663] ? show_regs.cold.9+0x22/0x2f [ 2888.071673] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2888.071679] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2888.071683] ? watchdog+0x40/0x40 [ 2888.071687] ? hrtimer_interrupt+0x128/0x2c0 [ 2888.071691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2888.071699] ? apic_timer_interrupt+0xf/0x20 [ 2888.071704] [ 2888.071706] ? on_each_cpu_cond_mask+0x46/0x150 [ 2888.071710] ? __x64_sys_fspick+0x210/0x210 [ 2888.071717] ? smp_call_function_many_cond+0x3db/0x540 [ 2888.071721] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2888.071724] ? __x64_sys_fspick+0x210/0x210 [ 2888.071728] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2888.071731] on_each_cpu_cond_mask+0x46/0x150 [ 2888.071735] on_each_cpu_cond+0x1c/0x30 [ 2888.071739] invalidate_bh_lrus+0x27/0x30 [ 2888.071742] invalidate_bdev+0x5a/0x90 [ 2888.071749] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2888.072025] generic_shutdown_super+0xb1/0x1b0 [ 2888.072033] kill_block_super+0x29/0x80 [ 2888.072039] deactivate_locked_super+0x52/0xd0 [ 2888.080429] lov(O) [ 2888.103973] deactivate_super+0x83/0x90 [ 2888.107175] osc(O) [ 2888.110348] cleanup_mnt+0x5f/0xe0 [ 2888.112406] lmv(O) [ 2888.118709] __cleanup_mnt+0x16/0x20 [ 2888.121664] fid(O) [ 2888.124130] task_work_run+0xc6/0x110 [ 2888.126393] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2888.126451] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2888.126459] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2888.126466] Workqueue: events free_work [libcfs] [ 2888.126506] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2888.126514] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2888.126518] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2888.126523] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2888.126526] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2888.126527] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2888.126530] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2888.126532] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2888.126535] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2888.126538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2888.126541] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2888.126548] Call Trace: [ 2888.126553] [ 2888.126558] ? show_regs.cold.9+0x22/0x2f [ 2888.126567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2888.126573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2888.126577] ? watchdog+0x40/0x40 [ 2888.126581] ? hrtimer_interrupt+0x128/0x2c0 [ 2888.126585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2888.126593] ? apic_timer_interrupt+0xf/0x20 [ 2888.126598] [ 2888.126600] ? on_each_cpu+0x51/0x120 [ 2888.126605] ? smp_call_function_many_cond+0x3d9/0x540 [ 2888.126608] ? smp_call_function_many_cond+0x3a8/0x540 [ 2888.126612] ? load_new_mm_cr3+0x120/0x120 [ 2888.126617] ? load_new_mm_cr3+0x120/0x120 [ 2888.126621] on_each_cpu+0x51/0x120 [ 2888.126625] flush_tlb_kernel_range+0x93/0xb0 [ 2888.126630] free_unmap_vmap_area+0x63/0x70 [ 2888.126636] remove_vm_area+0xe7/0x110 [ 2888.126640] __vunmap+0x9c/0x400 [ 2888.126643] vfree+0x4d/0x100 [ 2888.126647] free_work+0x34/0x50 [libcfs] [ 2888.126673] process_one_work+0x2c8/0x700 [ 2888.126680] worker_thread+0x296/0x6e0 [ 2888.126685] ? rescuer_thread+0x570/0x570 [ 2888.126689] kthread+0x1d1/0x200 [ 2888.126694] ? set_kthread_struct+0x70/0x70 [ 2888.126698] ret_from_fork+0x1f/0x30 [ 2888.409463] exit_to_usermode_loop+0x1dd/0x1f0 [ 2888.413128] do_syscall_64+0x3d6/0x440 [ 2888.414514] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2888.415776] RIP: 0033:0x7fbb019648fb [ 2888.418710] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2888.433752] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2888.442622] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2888.449064] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2888.455409] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2888.460598] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2888.466677] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2904.055006] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2904.065348] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2904.112164] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2904.120836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2904.125805] Workqueue: events free_work [libcfs] [ 2904.128144] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2904.132048] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2904.149621] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2904.161753] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2904.165024] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2904.170461] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2904.175079] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2904.183338] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2904.188689] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2904.191798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2904.195350] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2904.202659] Call Trace: [ 2904.205066] [ 2904.206221] ? show_regs.cold.9+0x22/0x2f [ 2904.208132] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2904.211951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2904.216879] ? watchdog+0x40/0x40 [ 2904.222125] ? hrtimer_interrupt+0x128/0x2c0 [ 2904.224691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2904.227099] ? apic_timer_interrupt+0xf/0x20 [ 2904.228706] [ 2904.229193] ? on_each_cpu+0x51/0x120 [ 2904.233248] ? smp_call_function_many_cond+0x3db/0x540 [ 2904.238255] ? smp_call_function_many_cond+0x3a8/0x540 [ 2904.242162] ? load_new_mm_cr3+0x120/0x120 [ 2904.247287] ? load_new_mm_cr3+0x120/0x120 [ 2904.249919] on_each_cpu+0x51/0x120 [ 2904.251803] flush_tlb_kernel_range+0x93/0xb0 [ 2904.253901] free_unmap_vmap_area+0x63/0x70 [ 2904.255515] remove_vm_area+0xe7/0x110 [ 2904.257382] __vunmap+0x9c/0x400 [ 2904.259008] vfree+0x4d/0x100 [ 2904.260798] free_work+0x34/0x50 [libcfs] [ 2904.262946] process_one_work+0x2c8/0x700 [ 2904.264830] worker_thread+0x296/0x6e0 [ 2904.266683] ? rescuer_thread+0x570/0x570 [ 2904.268819] kthread+0x1d1/0x200 [ 2904.270668] ? set_kthread_struct+0x70/0x70 [ 2904.273667] ret_from_fork+0x1f/0x30 [ 2907.616076] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 549s! [ 2907.630044] Showing busy workqueues and worker pools: [ 2907.635057] workqueue events: flags=0x0 [ 2907.638099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2907.644943] in-flight: 56293:free_work [libcfs] [ 2907.650212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2907.655296] in-flight: 22277:free_work [libcfs] [ 2907.657085] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2907.664070] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2907.669291] pending: nfs4_renew_state [nfsv4] [ 2907.671863] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2907.680190] in-flight: 47596:free_work [libcfs] [ 2907.681644] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2907.693777] workqueue events_unbound: flags=0x2 [ 2907.697700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2907.700229] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2907.711569] workqueue events_power_efficient: flags=0x80 [ 2907.721467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2907.728212] pending: neigh_periodic_work, do_cache_clean [ 2907.734519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2907.739685] pending: check_lifetime [ 2907.745236] workqueue mm_percpu_wq: flags=0x8 [ 2907.750495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2907.754522] pending: vmstat_update [ 2907.755894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2907.758839] pending: vmstat_update [ 2907.764174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2907.768477] pending: vmstat_update [ 2907.775852] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2907.778972] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2907.782978] pending: cgroup_pidlist_destroy_work_fn [ 2907.786092] workqueue writeback: flags=0x4a [ 2907.788177] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2907.790269] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2907.794684] workqueue xprtiod: flags=0xa [ 2907.796993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2907.800813] pending: xprt_autoclose [ 2907.804268] workqueue ipv6_addrconf: flags=0x40008 [ 2907.809184] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2907.814099] pending: addrconf_verify_work [ 2907.817235] workqueue knbd0-recv: flags=0x1a [ 2907.819748] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2907.821408] in-flight: 655:recv_work [ 2907.824596] workqueue ptlrpc_pinger: flags=0x2 [ 2907.827252] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2907.830686] pending: ptlrpc_pinger_main [ptlrpc] [ 2907.834997] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=550s workers=5 idle: 53684 27131 54850 46584 [ 2907.843208] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=550s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2907.849619] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=550s workers=6 idle: 53660 37674 46417 53664 46418 [ 2907.855740] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2897s workers=2 idle: 657 [ 2916.058014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2916.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2916.066662] Modules linked in: zfs(O) [ 2916.070111] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log [ 2916.072361] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2916.072448] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2916.072461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2916.072464] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2916.072476] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2916.072480] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2916.072485] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2916.072487] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2916.072489] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2916.072491] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2916.072493] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2916.072495] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2916.072498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2916.072500] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2916.072508] Call Trace: [ 2916.072513] [ 2916.072519] ? show_regs.cold.9+0x22/0x2f [ 2916.072529] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2916.072535] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2916.072538] ? watchdog+0x40/0x40 [ 2916.072542] ? hrtimer_interrupt+0x128/0x2c0 [ 2916.072546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2916.072554] ? apic_timer_interrupt+0xf/0x20 [ 2916.072559] [ 2916.072561] ? on_each_cpu_cond_mask+0x46/0x150 [ 2916.072565] ? __x64_sys_fspick+0x210/0x210 [ 2916.072573] ? smp_call_function_many_cond+0x3db/0x540 [ 2916.072577] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2916.072580] ? __x64_sys_fspick+0x210/0x210 [ 2916.072584] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2916.072587] on_each_cpu_cond_mask+0x46/0x150 [ 2916.072591] on_each_cpu_cond+0x1c/0x30 [ 2916.072594] invalidate_bh_lrus+0x27/0x30 [ 2916.072598] invalidate_bdev+0x5a/0x90 [ 2916.072604] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2916.072719] generic_shutdown_super+0xb1/0x1b0 [ 2916.072726] kill_block_super+0x29/0x80 [ 2916.072731] deactivate_locked_super+0x52/0xd0 [ 2916.072736] deactivate_super+0x83/0x90 [ 2916.072741] cleanup_mnt+0x5f/0xe0 [ 2916.072747] __cleanup_mnt+0x16/0x20 [ 2916.072751] task_work_run+0xc6/0x110 [ 2916.072758] exit_to_usermode_loop+0x1dd/0x1f0 [ 2916.072766] do_syscall_64+0x3d6/0x440 [ 2916.072771] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2916.072777] RIP: 0033:0x7fbb019648fb [ 2916.072782] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2916.072786] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2916.072790] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2916.072793] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2916.072795] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2916.072797] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2916.072798] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2916.272853] dm_mod sha512_ssse3 sha512_generic [ 2916.274776] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2916.281580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2916.283934] Workqueue: events free_work [libcfs] [ 2916.286833] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2916.288718] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2916.295555] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2916.297350] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2916.301278] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2916.303907] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2916.309619] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2916.322796] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2916.329117] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2916.338347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2916.347393] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2916.359778] Call Trace: [ 2916.360482] [ 2916.365877] ? show_regs.cold.9+0x22/0x2f [ 2916.371570] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2916.372940] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2916.379083] ? watchdog+0x40/0x40 [ 2916.381845] ? hrtimer_interrupt+0x128/0x2c0 [ 2916.389577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2916.392435] ? apic_timer_interrupt+0xf/0x20 [ 2916.396073] [ 2916.398739] ? on_each_cpu+0x51/0x120 [ 2916.404589] ? smp_call_function_many_cond+0x3db/0x540 [ 2916.407895] ? smp_call_function_many_cond+0x3a8/0x540 [ 2916.412280] ? load_new_mm_cr3+0x120/0x120 [ 2916.414814] ? load_new_mm_cr3+0x120/0x120 [ 2916.417140] on_each_cpu+0x51/0x120 [ 2916.426381] flush_tlb_kernel_range+0x93/0xb0 [ 2916.427896] free_unmap_vmap_area+0x63/0x70 [ 2916.429292] remove_vm_area+0xe7/0x110 [ 2916.434840] __vunmap+0x9c/0x400 [ 2916.442273] vfree+0x4d/0x100 [ 2916.444119] free_work+0x34/0x50 [libcfs] [ 2916.448934] process_one_work+0x2c8/0x700 [ 2916.450468] worker_thread+0x296/0x6e0 [ 2916.451978] ? rescuer_thread+0x570/0x570 [ 2916.455680] kthread+0x1d1/0x200 [ 2916.459675] ? set_kthread_struct+0x70/0x70 [ 2916.462117] ret_from_fork+0x1f/0x30 [ 2932.058058] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2932.065837] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2932.105735] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2932.113298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2932.119909] Workqueue: events free_work [libcfs] [ 2932.121864] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2932.125471] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2932.142241] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2932.146293] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2932.150468] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2932.155450] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2932.158469] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2932.166789] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2932.168584] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2932.174240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2932.177734] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2932.181713] Call Trace: [ 2932.182651] [ 2932.185352] ? show_regs.cold.9+0x22/0x2f [ 2932.189866] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2932.193804] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2932.197629] ? watchdog+0x40/0x40 [ 2932.199041] ? hrtimer_interrupt+0x128/0x2c0 [ 2932.200692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2932.203763] ? apic_timer_interrupt+0xf/0x20 [ 2932.207293] [ 2932.208386] ? on_each_cpu+0x51/0x120 [ 2932.213411] ? smp_call_function_many_cond+0x3db/0x540 [ 2932.215671] ? smp_call_function_many_cond+0x3a8/0x540 [ 2932.218520] ? load_new_mm_cr3+0x120/0x120 [ 2932.221188] ? load_new_mm_cr3+0x120/0x120 [ 2932.222749] on_each_cpu+0x51/0x120 [ 2932.228091] flush_tlb_kernel_range+0x93/0xb0 [ 2932.230498] free_unmap_vmap_area+0x63/0x70 [ 2932.234432] remove_vm_area+0xe7/0x110 [ 2932.237102] __vunmap+0x9c/0x400 [ 2932.238223] vfree+0x4d/0x100 [ 2932.239293] free_work+0x34/0x50 [libcfs] [ 2932.242220] process_one_work+0x2c8/0x700 [ 2932.245275] worker_thread+0x296/0x6e0 [ 2932.248296] ? rescuer_thread+0x570/0x570 [ 2932.251563] kthread+0x1d1/0x200 [ 2932.252930] ? set_kthread_struct+0x70/0x70 [ 2932.257458] ret_from_fork+0x1f/0x30 [ 2938.335025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 580s! [ 2938.338013] Showing busy workqueues and worker pools: [ 2938.340122] workqueue events: flags=0x0 [ 2938.341260] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2938.343065] in-flight: 56293:free_work [libcfs] [ 2938.344849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2938.347355] in-flight: 22277:free_work [libcfs] [ 2938.349157] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2938.352261] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2938.354635] pending: nfs4_renew_state [nfsv4] [ 2938.356322] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2938.358787] in-flight: 47596:free_work [libcfs] [ 2938.360310] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2938.364040] workqueue events_unbound: flags=0x2 [ 2938.367010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2938.371799] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2938.375138] workqueue events_power_efficient: flags=0x80 [ 2938.377313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2938.379952] pending: neigh_periodic_work, do_cache_clean [ 2938.382095] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2938.383796] pending: check_lifetime [ 2938.385635] workqueue mm_percpu_wq: flags=0x8 [ 2938.387272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2938.390280] pending: vmstat_update [ 2938.391919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2938.394268] pending: vmstat_update [ 2938.395962] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2938.398323] pending: vmstat_update [ 2938.399916] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2938.401932] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2938.404229] pending: cgroup_pidlist_destroy_work_fn [ 2938.406976] workqueue writeback: flags=0x4a [ 2938.408291] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2938.410636] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2938.413073] workqueue xprtiod: flags=0xa [ 2938.414511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2938.416959] pending: xprt_autoclose [ 2938.418241] workqueue ipv6_addrconf: flags=0x40008 [ 2938.420142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2938.422387] pending: addrconf_verify_work [ 2938.424173] workqueue knbd0-recv: flags=0x1a [ 2938.425781] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2938.428742] in-flight: 655:recv_work [ 2938.429791] workqueue ptlrpc_pinger: flags=0x2 [ 2938.431809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2938.434042] pending: ptlrpc_pinger_main [ptlrpc] [ 2938.437258] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=580s workers=5 idle: 53684 27131 54850 46584 [ 2938.440631] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=580s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2938.446308] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=580s workers=6 idle: 53660 37674 46417 53664 46418 [ 2938.452168] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2928s workers=2 idle: 657 [ 2944.059037] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 2944.061468] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2944.063760] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2944.063852] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2944.063859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2944.063865] Workqueue: events free_work [libcfs] [ 2944.063902] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2944.063911] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2944.063915] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2944.063919] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2944.063921] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2944.063923] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2944.063926] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2944.063927] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2944.063930] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2944.063933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2944.063935] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2944.063944] Call Trace: [ 2944.063950] [ 2944.063955] ? show_regs.cold.9+0x22/0x2f [ 2944.063964] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2944.063970] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2944.063974] ? watchdog+0x40/0x40 [ 2944.063978] ? hrtimer_interrupt+0x128/0x2c0 [ 2944.063981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2944.063989] ? apic_timer_interrupt+0xf/0x20 [ 2944.063994] [ 2944.063996] ? on_each_cpu+0x51/0x120 [ 2944.064000] ? smp_call_function_many_cond+0x3db/0x540 [ 2944.064004] ? smp_call_function_many_cond+0x3a8/0x540 [ 2944.064008] ? load_new_mm_cr3+0x120/0x120 [ 2944.064013] ? load_new_mm_cr3+0x120/0x120 [ 2944.064017] on_each_cpu+0x51/0x120 [ 2944.064021] flush_tlb_kernel_range+0x93/0xb0 [ 2944.064026] free_unmap_vmap_area+0x63/0x70 [ 2944.064031] remove_vm_area+0xe7/0x110 [ 2944.064034] __vunmap+0x9c/0x400 [ 2944.064038] vfree+0x4d/0x100 [ 2944.064041] free_work+0x34/0x50 [libcfs] [ 2944.064067] process_one_work+0x2c8/0x700 [ 2944.064075] worker_thread+0x296/0x6e0 [ 2944.064079] ? rescuer_thread+0x570/0x570 [ 2944.064082] kthread+0x1d1/0x200 [ 2944.064086] ? set_kthread_struct+0x70/0x70 [ 2944.064089] ret_from_fork+0x1f/0x30 [ 2944.344745] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2944.397968] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2944.411028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2944.415322] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2944.418446] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2944.430922] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2944.437178] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2944.442103] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2944.446653] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2944.453377] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2944.461968] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2944.466743] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2944.475373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2944.479601] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2944.490691] Call Trace: [ 2944.494240] [ 2944.495934] ? show_regs.cold.9+0x22/0x2f [ 2944.498363] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2944.501095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2944.505130] ? watchdog+0x40/0x40 [ 2944.507158] ? hrtimer_interrupt+0x128/0x2c0 [ 2944.511118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2944.514234] ? apic_timer_interrupt+0xf/0x20 [ 2944.519814] [ 2944.521044] ? on_each_cpu_cond_mask+0x46/0x150 [ 2944.523950] ? __x64_sys_fspick+0x210/0x210 [ 2944.528646] ? smp_call_function_many_cond+0x3db/0x540 [ 2944.536374] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2944.538982] ? __x64_sys_fspick+0x210/0x210 [ 2944.540679] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2944.548616] on_each_cpu_cond_mask+0x46/0x150 [ 2944.551078] on_each_cpu_cond+0x1c/0x30 [ 2944.553347] invalidate_bh_lrus+0x27/0x30 [ 2944.556340] invalidate_bdev+0x5a/0x90 [ 2944.558672] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2944.564125] generic_shutdown_super+0xb1/0x1b0 [ 2944.567379] kill_block_super+0x29/0x80 [ 2944.573301] deactivate_locked_super+0x52/0xd0 [ 2944.578600] deactivate_super+0x83/0x90 [ 2944.582274] cleanup_mnt+0x5f/0xe0 [ 2944.584201] __cleanup_mnt+0x16/0x20 [ 2944.585956] task_work_run+0xc6/0x110 [ 2944.590096] exit_to_usermode_loop+0x1dd/0x1f0 [ 2944.591251] do_syscall_64+0x3d6/0x440 [ 2944.592157] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2944.598032] RIP: 0033:0x7fbb019648fb [ 2944.601263] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2944.615318] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2944.622935] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2944.631479] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2944.642411] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2944.650489] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2944.658560] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2957.748113] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 2957.752177] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 2957.759229] (detected by 2, t=600018 jiffies, g=626849, q=1398) [ 2957.764208] Sending NMI from CPU 2 to CPUs 3: [ 2957.768396] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 2957.768405] rcu: rcu_sched kthread starved for 599660 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2957.785571] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2957.790669] rcu: RCU grace-period kthread stack dump: [ 2957.793927] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2957.800616] Call Trace: [ 2957.802041] __schedule+0x351/0xcb0 [ 2957.806077] schedule+0xc0/0x180 [ 2957.807972] schedule_timeout+0xb4/0x190 [ 2957.811451] ? do_raw_spin_unlock+0x75/0x190 [ 2957.813169] ? __next_timer_interrupt+0x160/0x160 [ 2957.815868] ? prepare_to_swait_event+0xf3/0x140 [ 2957.817989] rcu_gp_kthread+0x8dd/0x15b0 [ 2957.820019] ? force_qs_rnp+0x2c0/0x2c0 [ 2957.822058] kthread+0x1d1/0x200 [ 2957.823954] ? set_kthread_struct+0x70/0x70 [ 2957.826201] ret_from_fork+0x1f/0x30 [ 2957.828247] rcu: Stack dump where RCU GP kthread last ran: [ 2957.831189] NMI backtrace for cpu 2 [ 2957.833029] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2957.839404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2957.844665] Workqueue: events free_work [libcfs] [ 2957.847131] Call Trace: [ 2957.848664] [ 2957.849210] ? dump_stack+0xbb/0x10e [ 2957.850575] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2957.852221] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2957.854148] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2957.856358] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2957.858620] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2957.860514] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2957.862695] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 2957.864384] ? update_process_times+0x6d/0xc0 [ 2957.865603] ? tick_sched_handle.isra.18+0x27/0x90 [ 2957.867028] ? tick_sched_timer+0x53/0xb0 [ 2957.868170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2957.869486] ? tick_sched_handle.isra.18+0x90/0x90 [ 2957.870719] ? hrtimer_interrupt+0x128/0x2c0 [ 2957.872232] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2957.873673] ? apic_timer_interrupt+0xf/0x20 [ 2957.874837] [ 2957.875575] ? on_each_cpu+0x51/0x120 [ 2957.876565] ? smp_call_function_many_cond+0x3db/0x540 [ 2957.878051] ? smp_call_function_many_cond+0x3a8/0x540 [ 2957.879570] ? load_new_mm_cr3+0x120/0x120 [ 2957.880699] ? load_new_mm_cr3+0x120/0x120 [ 2957.881829] ? on_each_cpu+0x51/0x120 [ 2957.882875] ? flush_tlb_kernel_range+0x93/0xb0 [ 2957.884228] ? free_unmap_vmap_area+0x63/0x70 [ 2957.885588] ? remove_vm_area+0xe7/0x110 [ 2957.886694] ? __vunmap+0x9c/0x400 [ 2957.887655] ? vfree+0x4d/0x100 [ 2957.888604] ? free_work+0x34/0x50 [libcfs] [ 2957.889858] ? process_one_work+0x2c8/0x700 [ 2957.891002] ? worker_thread+0x296/0x6e0 [ 2957.892166] ? rescuer_thread+0x570/0x570 [ 2957.893384] ? kthread+0x1d1/0x200 [ 2957.894382] ? set_kthread_struct+0x70/0x70 [ 2957.895544] ? ret_from_fork+0x1f/0x30 [ 2964.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 2964.062001] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2964.094561] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2964.103775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2964.108456] Workqueue: events free_work [libcfs] [ 2964.110567] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2964.112952] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2964.121380] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2964.124489] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2964.128067] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2964.134417] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2964.138698] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2964.145315] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2964.149274] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2964.153112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2964.159661] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2964.162972] Call Trace: [ 2964.163978] [ 2964.164879] ? show_regs.cold.9+0x22/0x2f [ 2964.167065] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2964.169619] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2964.172212] ? watchdog+0x40/0x40 [ 2964.173765] ? hrtimer_interrupt+0x128/0x2c0 [ 2964.177512] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2964.179883] ? apic_timer_interrupt+0xf/0x20 [ 2964.183257] [ 2964.186181] ? on_each_cpu+0x51/0x120 [ 2964.187327] ? smp_call_function_many_cond+0x3db/0x540 [ 2964.195800] ? smp_call_function_many_cond+0x3a8/0x540 [ 2964.198698] ? load_new_mm_cr3+0x120/0x120 [ 2964.205622] ? load_new_mm_cr3+0x120/0x120 [ 2964.207795] on_each_cpu+0x51/0x120 [ 2964.209893] flush_tlb_kernel_range+0x93/0xb0 [ 2964.215325] free_unmap_vmap_area+0x63/0x70 [ 2964.217503] remove_vm_area+0xe7/0x110 [ 2964.219600] __vunmap+0x9c/0x400 [ 2964.220876] vfree+0x4d/0x100 [ 2964.223061] free_work+0x34/0x50 [libcfs] [ 2964.229559] process_one_work+0x2c8/0x700 [ 2964.233774] worker_thread+0x296/0x6e0 [ 2964.236118] ? rescuer_thread+0x570/0x570 [ 2964.239651] kthread+0x1d1/0x200 [ 2964.241237] ? set_kthread_struct+0x70/0x70 [ 2964.243242] ret_from_fork+0x1f/0x30 [ 2968.548402] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 610s! [ 2968.553315] Showing busy workqueues and worker pools: [ 2968.556478] workqueue events: flags=0x0 [ 2968.557903] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2968.565340] in-flight: 56293:free_work [libcfs] [ 2968.569506] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2968.578388] in-flight: 22277:free_work [libcfs] [ 2968.588836] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2968.599208] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2968.608356] pending: nfs4_renew_state [nfsv4] [ 2968.609692] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2968.615545] in-flight: 47596:free_work [libcfs] [ 2968.621516] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2968.633823] workqueue events_unbound: flags=0x2 [ 2968.636386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2968.644599] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2968.652765] workqueue events_power_efficient: flags=0x80 [ 2968.657669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2968.660700] pending: neigh_periodic_work, do_cache_clean [ 2968.662978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2968.666502] pending: check_lifetime [ 2968.668040] workqueue mm_percpu_wq: flags=0x8 [ 2968.671140] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2968.675511] pending: vmstat_update [ 2968.677023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2968.680918] pending: vmstat_update [ 2968.685901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2968.691208] pending: vmstat_update [ 2968.693612] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2968.698951] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2968.703489] pending: cgroup_pidlist_destroy_work_fn [ 2968.706251] workqueue writeback: flags=0x4a [ 2968.708092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2968.710922] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2968.715331] workqueue xprtiod: flags=0xa [ 2968.717141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2968.719779] pending: xprt_autoclose [ 2968.720969] workqueue ipv6_addrconf: flags=0x40008 [ 2968.723198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2968.729147] pending: addrconf_verify_work [ 2968.732590] workqueue knbd0-recv: flags=0x1a [ 2968.734184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2968.740481] in-flight: 655:recv_work [ 2968.741956] workqueue ptlrpc_pinger: flags=0x2 [ 2968.744196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2968.747971] pending: ptlrpc_pinger_main [ptlrpc] [ 2968.750021] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=610s workers=5 idle: 53684 27131 54850 46584 [ 2968.764238] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=611s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2968.775043] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=611s workers=6 idle: 53660 37674 46417 53664 46418 [ 2968.781220] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2958s workers=2 idle: 657 [ 2972.058377] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 2972.063509] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2972.089282] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2972.093557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2972.097880] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2972.100643] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2972.108734] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2972.112631] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 2972.116184] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 2972.119548] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 2972.123547] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 2972.128987] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 2972.133423] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 2972.136971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2972.142143] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 2972.146429] Call Trace: [ 2972.149255] [ 2972.150661] ? show_regs.cold.9+0x22/0x2f [ 2972.155304] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2972.157683] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2972.160862] ? watchdog+0x40/0x40 [ 2972.162083] ? hrtimer_interrupt+0x128/0x2c0 [ 2972.163434] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2972.165775] ? apic_timer_interrupt+0xf/0x20 [ 2972.168782] [ 2972.170583] ? on_each_cpu_cond_mask+0x46/0x150 [ 2972.173517] ? __x64_sys_fspick+0x210/0x210 [ 2972.175479] ? smp_call_function_many_cond+0x3db/0x540 [ 2972.177801] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2972.179883] ? __x64_sys_fspick+0x210/0x210 [ 2972.181906] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 2972.184709] on_each_cpu_cond_mask+0x46/0x150 [ 2972.187433] on_each_cpu_cond+0x1c/0x30 [ 2972.189871] invalidate_bh_lrus+0x27/0x30 [ 2972.192735] invalidate_bdev+0x5a/0x90 [ 2972.196032] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 2972.198546] generic_shutdown_super+0xb1/0x1b0 [ 2972.200735] kill_block_super+0x29/0x80 [ 2972.202924] deactivate_locked_super+0x52/0xd0 [ 2972.205156] deactivate_super+0x83/0x90 [ 2972.206839] cleanup_mnt+0x5f/0xe0 [ 2972.208585] __cleanup_mnt+0x16/0x20 [ 2972.210236] task_work_run+0xc6/0x110 [ 2972.211909] exit_to_usermode_loop+0x1dd/0x1f0 [ 2972.214186] do_syscall_64+0x3d6/0x440 [ 2972.217770] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2972.220910] RIP: 0033:0x7fbb019648fb [ 2972.223709] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 2972.232856] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 2972.236805] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 2972.240961] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 2972.245052] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 2972.250800] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 2972.257378] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 2984.060702] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 2984.067778] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2984.097559] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2984.106279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2984.118633] Workqueue: events free_work [libcfs] [ 2984.122901] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 2984.124804] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 2984.136054] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2984.139869] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 2984.142655] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 2984.145196] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 2984.148876] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2984.153426] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 2984.157361] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 2984.164027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2984.166636] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 2984.172010] Call Trace: [ 2984.174518] [ 2984.175600] ? show_regs.cold.9+0x22/0x2f [ 2984.176690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2984.179988] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2984.182151] ? watchdog+0x40/0x40 [ 2984.184470] ? hrtimer_interrupt+0x128/0x2c0 [ 2984.186530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2984.189560] ? apic_timer_interrupt+0xf/0x20 [ 2984.192129] [ 2984.193430] ? on_each_cpu+0x51/0x120 [ 2984.197261] ? smp_call_function_many_cond+0x3db/0x540 [ 2984.200453] ? smp_call_function_many_cond+0x3a8/0x540 [ 2984.203522] ? load_new_mm_cr3+0x120/0x120 [ 2984.205616] ? load_new_mm_cr3+0x120/0x120 [ 2984.208085] on_each_cpu+0x51/0x120 [ 2984.209061] flush_tlb_kernel_range+0x93/0xb0 [ 2984.210137] free_unmap_vmap_area+0x63/0x70 [ 2984.212363] remove_vm_area+0xe7/0x110 [ 2984.214411] __vunmap+0x9c/0x400 [ 2984.217267] vfree+0x4d/0x100 [ 2984.219823] free_work+0x34/0x50 [libcfs] [ 2984.221516] process_one_work+0x2c8/0x700 [ 2984.223388] worker_thread+0x296/0x6e0 [ 2984.225213] ? rescuer_thread+0x570/0x570 [ 2984.227572] kthread+0x1d1/0x200 [ 2984.230147] ? set_kthread_struct+0x70/0x70 [ 2984.231654] ret_from_fork+0x1f/0x30 [ 2996.055014] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 2996.060310] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2996.083796] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2996.089827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2996.092633] Workqueue: events free_work [libcfs] [ 2996.094320] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 2996.096984] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 2996.104564] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2996.107979] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 2996.110346] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 2996.113620] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 2996.117688] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 2996.120417] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 2996.123177] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 2996.127426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2996.130584] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 2996.134196] Call Trace: [ 2996.135644] [ 2996.136985] ? show_regs.cold.9+0x22/0x2f [ 2996.139000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2996.140807] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2996.142989] ? watchdog+0x40/0x40 [ 2996.144671] ? hrtimer_interrupt+0x128/0x2c0 [ 2996.146628] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2996.148357] ? apic_timer_interrupt+0xf/0x20 [ 2996.150249] [ 2996.151192] ? on_each_cpu+0x51/0x120 [ 2996.152868] ? smp_call_function_many_cond+0x3d9/0x540 [ 2996.155086] ? smp_call_function_many_cond+0x3a8/0x540 [ 2996.157332] ? load_new_mm_cr3+0x120/0x120 [ 2996.159138] ? load_new_mm_cr3+0x120/0x120 [ 2996.160909] on_each_cpu+0x51/0x120 [ 2996.162407] flush_tlb_kernel_range+0x93/0xb0 [ 2996.164304] free_unmap_vmap_area+0x63/0x70 [ 2996.166169] remove_vm_area+0xe7/0x110 [ 2996.167870] __vunmap+0x9c/0x400 [ 2996.169249] vfree+0x4d/0x100 [ 2996.170673] free_work+0x34/0x50 [libcfs] [ 2996.172397] process_one_work+0x2c8/0x700 [ 2996.174198] worker_thread+0x296/0x6e0 [ 2996.175841] ? rescuer_thread+0x570/0x570 [ 2996.177623] kthread+0x1d1/0x200 [ 2996.178866] ? set_kthread_struct+0x70/0x70 [ 2996.180732] ret_from_fork+0x1f/0x30 [ 2999.263149] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 640s! [ 2999.268732] Showing busy workqueues and worker pools: [ 2999.272195] workqueue events: flags=0x0 [ 2999.274686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2999.286921] in-flight: 56293:free_work [libcfs] [ 2999.294174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2999.303532] in-flight: 22277:free_work [libcfs] [ 2999.308829] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 2999.319541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2999.328921] pending: nfs4_renew_state [nfsv4] [ 2999.333696] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2999.338540] in-flight: 47596:free_work [libcfs] [ 2999.340792] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2999.347159] workqueue events_unbound: flags=0x2 [ 2999.350155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2999.354354] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2999.360123] workqueue events_power_efficient: flags=0x80 [ 2999.363929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2999.368720] pending: neigh_periodic_work, do_cache_clean [ 2999.372695] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2999.376498] pending: check_lifetime [ 2999.378886] workqueue mm_percpu_wq: flags=0x8 [ 2999.382153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2999.386884] pending: vmstat_update [ 2999.388974] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2999.393724] pending: vmstat_update [ 2999.396129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2999.399538] pending: vmstat_update [ 2999.401961] workqueue cgroup_pidlist_destroy: flags=0x0 [ 2999.405980] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2999.411534] pending: cgroup_pidlist_destroy_work_fn [ 2999.415632] workqueue writeback: flags=0x4a [ 2999.418237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2999.422845] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 2999.426084] workqueue xprtiod: flags=0xa [ 2999.428945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2999.433619] pending: xprt_autoclose [ 2999.436675] workqueue ipv6_addrconf: flags=0x40008 [ 2999.439336] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2999.443808] pending: addrconf_verify_work [ 2999.447173] workqueue knbd0-recv: flags=0x1a [ 2999.449673] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2999.453109] in-flight: 655:recv_work [ 2999.455208] workqueue ptlrpc_pinger: flags=0x2 [ 2999.457694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2999.460834] pending: ptlrpc_pinger_main [ptlrpc] [ 2999.464104] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=641s workers=5 idle: 53684 27131 54850 46584 [ 2999.468820] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=641s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 2999.475205] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=641s workers=6 idle: 53660 37674 46417 53664 46418 [ 2999.479796] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2989s workers=2 idle: 657 [ 3000.058314] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3000.070523] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3000.124684] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3000.132927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3000.144080] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3000.151580] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3000.163129] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3000.167066] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3000.170089] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3000.173298] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3000.177953] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3000.182562] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3000.187067] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3000.192799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3000.196654] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3000.209464] Call Trace: [ 3000.212092] [ 3000.212886] ? show_regs.cold.9+0x22/0x2f [ 3000.223596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3000.228485] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3000.231106] ? watchdog+0x40/0x40 [ 3000.233118] ? hrtimer_interrupt+0x128/0x2c0 [ 3000.235236] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3000.237986] ? apic_timer_interrupt+0xf/0x20 [ 3000.240046] [ 3000.241169] ? on_each_cpu_cond_mask+0x46/0x150 [ 3000.244090] ? __x64_sys_fspick+0x210/0x210 [ 3000.246782] ? smp_call_function_many_cond+0x3d9/0x540 [ 3000.249620] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3000.251754] ? __x64_sys_fspick+0x210/0x210 [ 3000.254357] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3000.257114] on_each_cpu_cond_mask+0x46/0x150 [ 3000.260343] on_each_cpu_cond+0x1c/0x30 [ 3000.262274] invalidate_bh_lrus+0x27/0x30 [ 3000.265317] invalidate_bdev+0x5a/0x90 [ 3000.269921] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3000.274594] generic_shutdown_super+0xb1/0x1b0 [ 3000.281713] kill_block_super+0x29/0x80 [ 3000.283854] deactivate_locked_super+0x52/0xd0 [ 3000.289240] deactivate_super+0x83/0x90 [ 3000.291305] cleanup_mnt+0x5f/0xe0 [ 3000.294130] __cleanup_mnt+0x16/0x20 [ 3000.297956] task_work_run+0xc6/0x110 [ 3000.300031] exit_to_usermode_loop+0x1dd/0x1f0 [ 3000.302026] do_syscall_64+0x3d6/0x440 [ 3000.304083] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3000.306120] RIP: 0033:0x7fbb019648fb [ 3000.308253] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3000.328973] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3000.339910] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3000.343963] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3000.347273] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3000.352591] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3000.356585] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3012.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3012.062075] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3012.104039] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3012.114769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3012.124649] Workqueue: events free_work [libcfs] [ 3012.127813] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3012.131255] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3012.142419] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3012.146506] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3012.150681] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3012.154751] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3012.158090] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3012.163118] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3012.167585] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3012.174418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3012.177545] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3012.182870] Call Trace: [ 3012.184194] [ 3012.185174] ? show_regs.cold.9+0x22/0x2f [ 3012.187025] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3012.188935] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3012.191424] ? watchdog+0x40/0x40 [ 3012.193059] ? hrtimer_interrupt+0x128/0x2c0 [ 3012.195078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3012.197753] ? apic_timer_interrupt+0xf/0x20 [ 3012.200188] [ 3012.201915] ? on_each_cpu+0x51/0x120 [ 3012.204919] ? smp_call_function_many_cond+0x3db/0x540 [ 3012.209349] ? smp_call_function_many_cond+0x3a8/0x540 [ 3012.212755] ? load_new_mm_cr3+0x120/0x120 [ 3012.215024] ? load_new_mm_cr3+0x120/0x120 [ 3012.216620] on_each_cpu+0x51/0x120 [ 3012.219673] flush_tlb_kernel_range+0x93/0xb0 [ 3012.222021] free_unmap_vmap_area+0x63/0x70 [ 3012.225809] remove_vm_area+0xe7/0x110 [ 3012.227801] __vunmap+0x9c/0x400 [ 3012.229683] vfree+0x4d/0x100 [ 3012.231576] free_work+0x34/0x50 [libcfs] [ 3012.233682] process_one_work+0x2c8/0x700 [ 3012.235779] worker_thread+0x296/0x6e0 [ 3012.237932] ? rescuer_thread+0x570/0x570 [ 3012.240080] kthread+0x1d1/0x200 [ 3012.241818] ? set_kthread_struct+0x70/0x70 [ 3012.243672] ret_from_fork+0x1f/0x30 [ 3024.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3024.058342] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3024.084565] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3024.092387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3024.095770] Workqueue: events free_work [libcfs] [ 3024.097796] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3024.100540] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3024.109329] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3024.113704] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3024.117397] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3024.121038] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3024.124364] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3024.128781] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3024.134265] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3024.139628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3024.144639] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3024.150701] Call Trace: [ 3024.152101] [ 3024.153189] ? show_regs.cold.9+0x22/0x2f [ 3024.155345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3024.157724] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3024.161543] ? watchdog+0x40/0x40 [ 3024.163303] ? hrtimer_interrupt+0x128/0x2c0 [ 3024.165443] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3024.167577] ? apic_timer_interrupt+0xf/0x20 [ 3024.169763] [ 3024.171070] ? on_each_cpu+0x51/0x120 [ 3024.172773] ? smp_call_function_many_cond+0x3d9/0x540 [ 3024.175285] ? smp_call_function_many_cond+0x3a8/0x540 [ 3024.177673] ? load_new_mm_cr3+0x120/0x120 [ 3024.180133] ? load_new_mm_cr3+0x120/0x120 [ 3024.182184] on_each_cpu+0x51/0x120 [ 3024.183749] flush_tlb_kernel_range+0x93/0xb0 [ 3024.186237] free_unmap_vmap_area+0x63/0x70 [ 3024.187781] remove_vm_area+0xe7/0x110 [ 3024.189467] __vunmap+0x9c/0x400 [ 3024.191157] vfree+0x4d/0x100 [ 3024.192586] free_work+0x34/0x50 [libcfs] [ 3024.194553] process_one_work+0x2c8/0x700 [ 3024.196563] worker_thread+0x296/0x6e0 [ 3024.198521] ? rescuer_thread+0x570/0x570 [ 3024.200403] kthread+0x1d1/0x200 [ 3024.202092] ? set_kthread_struct+0x70/0x70 [ 3024.204068] ret_from_fork+0x1f/0x30 [ 3028.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3028.062056] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3028.095594] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3028.102610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3028.106562] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3028.110233] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3028.118104] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3028.121228] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3028.124145] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3028.127145] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3028.130341] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3028.133247] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3028.136148] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3028.139480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3028.142062] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3028.144939] Call Trace: [ 3028.146013] [ 3028.146900] ? show_regs.cold.9+0x22/0x2f [ 3028.148593] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3028.150557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3028.152394] ? watchdog+0x40/0x40 [ 3028.153670] ? hrtimer_interrupt+0x128/0x2c0 [ 3028.156566] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3028.163632] ? apic_timer_interrupt+0xf/0x20 [ 3028.165469] [ 3028.166359] ? on_each_cpu_cond_mask+0x46/0x150 [ 3028.173355] ? __x64_sys_fspick+0x210/0x210 [ 3028.177395] ? smp_call_function_many_cond+0x3db/0x540 [ 3028.183261] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3028.187470] ? __x64_sys_fspick+0x210/0x210 [ 3028.189676] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3028.192249] on_each_cpu_cond_mask+0x46/0x150 [ 3028.194605] on_each_cpu_cond+0x1c/0x30 [ 3028.196490] invalidate_bh_lrus+0x27/0x30 [ 3028.198516] invalidate_bdev+0x5a/0x90 [ 3028.200401] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3028.203306] generic_shutdown_super+0xb1/0x1b0 [ 3028.205400] kill_block_super+0x29/0x80 [ 3028.207503] deactivate_locked_super+0x52/0xd0 [ 3028.210002] deactivate_super+0x83/0x90 [ 3028.212837] cleanup_mnt+0x5f/0xe0 [ 3028.214245] __cleanup_mnt+0x16/0x20 [ 3028.222684] task_work_run+0xc6/0x110 [ 3028.227466] exit_to_usermode_loop+0x1dd/0x1f0 [ 3028.233090] do_syscall_64+0x3d6/0x440 [ 3028.237657] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3028.240676] RIP: 0033:0x7fbb019648fb [ 3028.242412] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3028.258051] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3028.262111] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3028.265935] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3028.269616] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3028.272877] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3028.276627] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3029.983028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 671s! [ 3029.990040] Showing busy workqueues and worker pools: [ 3029.992550] workqueue events: flags=0x0 [ 3029.995043] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3029.999102] in-flight: 56293:free_work [libcfs] [ 3030.003172] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3030.013269] in-flight: 22277:free_work [libcfs] [ 3030.015250] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3030.023214] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3030.030384] pending: nfs4_renew_state [nfsv4] [ 3030.035606] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3030.039576] in-flight: 47596:free_work [libcfs] [ 3030.042443] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3030.046556] workqueue events_unbound: flags=0x2 [ 3030.048449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3030.052597] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3030.058508] workqueue events_power_efficient: flags=0x80 [ 3030.062915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3030.067499] pending: neigh_periodic_work, do_cache_clean [ 3030.072197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3030.076594] pending: check_lifetime [ 3030.079013] workqueue mm_percpu_wq: flags=0x8 [ 3030.080992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3030.086745] pending: vmstat_update [ 3030.088273] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3030.095169] pending: vmstat_update [ 3030.096698] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3030.098781] pending: vmstat_update [ 3030.100997] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3030.108647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3030.117216] pending: cgroup_pidlist_destroy_work_fn [ 3030.120612] workqueue writeback: flags=0x4a [ 3030.123136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3030.125980] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3030.129258] workqueue xprtiod: flags=0xa [ 3030.131631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3030.135167] pending: xprt_autoclose [ 3030.138390] workqueue ipv6_addrconf: flags=0x40008 [ 3030.142121] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3030.147204] pending: addrconf_verify_work [ 3030.148936] workqueue knbd0-recv: flags=0x1a [ 3030.156219] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3030.159992] in-flight: 655:recv_work [ 3030.162178] workqueue ptlrpc_pinger: flags=0x2 [ 3030.168859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3030.174008] pending: ptlrpc_pinger_main [ptlrpc] [ 3030.177152] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=672s workers=5 idle: 53684 27131 54850 46584 [ 3030.183626] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=672s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3030.189620] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=672s workers=6 idle: 53660 37674 46417 53664 46418 [ 3030.196174] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3020s workers=2 idle: 657 [ 3040.059131] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3040.064421] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3040.102609] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3040.111412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3040.115307] Workqueue: events free_work [libcfs] [ 3040.118356] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3040.121260] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3040.130244] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3040.134273] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3040.137957] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3040.141140] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3040.144243] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3040.146957] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3040.152564] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3040.156992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3040.159306] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3040.162406] Call Trace: [ 3040.163611] [ 3040.164397] ? show_regs.cold.9+0x22/0x2f [ 3040.166139] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3040.168178] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3040.169981] ? watchdog+0x40/0x40 [ 3040.171381] ? hrtimer_interrupt+0x128/0x2c0 [ 3040.173145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3040.174764] ? apic_timer_interrupt+0xf/0x20 [ 3040.176164] [ 3040.176922] ? on_each_cpu+0x51/0x120 [ 3040.177965] ? smp_call_function_many_cond+0x3db/0x540 [ 3040.179875] ? smp_call_function_many_cond+0x3a8/0x540 [ 3040.181602] ? load_new_mm_cr3+0x120/0x120 [ 3040.182881] ? load_new_mm_cr3+0x120/0x120 [ 3040.184298] on_each_cpu+0x51/0x120 [ 3040.185407] flush_tlb_kernel_range+0x93/0xb0 [ 3040.186780] free_unmap_vmap_area+0x63/0x70 [ 3040.188090] remove_vm_area+0xe7/0x110 [ 3040.189235] __vunmap+0x9c/0x400 [ 3040.190182] vfree+0x4d/0x100 [ 3040.191212] free_work+0x34/0x50 [libcfs] [ 3040.192585] process_one_work+0x2c8/0x700 [ 3040.193821] worker_thread+0x296/0x6e0 [ 3040.195257] ? rescuer_thread+0x570/0x570 [ 3040.196644] kthread+0x1d1/0x200 [ 3040.197659] ? set_kthread_struct+0x70/0x70 [ 3040.198868] ret_from_fork+0x1f/0x30 [ 3056.055058] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3056.060446] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3056.061999] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3056.093763] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3056.093778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3056.093784] Workqueue: events free_work [libcfs] [ 3056.093829] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3056.093838] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3056.093842] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3056.093846] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3056.093848] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3056.093850] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3056.093852] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3056.093854] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3056.093858] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3056.093861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3056.093863] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3056.094049] Call Trace: [ 3056.094054] [ 3056.094060] ? show_regs.cold.9+0x22/0x2f [ 3056.094069] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3056.094076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3056.094080] ? watchdog+0x40/0x40 [ 3056.094084] ? hrtimer_interrupt+0x128/0x2c0 [ 3056.094088] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3056.094097] ? apic_timer_interrupt+0xf/0x20 [ 3056.101731] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3056.101818] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3056.101825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3056.101828] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3056.101839] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3056.101843] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3056.101848] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3056.101851] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3056.101853] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3056.101855] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3056.101858] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3056.101861] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3056.101864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3056.101867] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3056.101875] Call Trace: [ 3056.101880] [ 3056.101885] ? show_regs.cold.9+0x22/0x2f [ 3056.101895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3056.101900] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3056.101904] ? watchdog+0x40/0x40 [ 3056.101908] ? hrtimer_interrupt+0x128/0x2c0 [ 3056.101912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3056.101920] ? apic_timer_interrupt+0xf/0x20 [ 3056.101924] [ 3056.101926] ? on_each_cpu_cond_mask+0x46/0x150 [ 3056.101929] ? __x64_sys_fspick+0x210/0x210 [ 3056.101936] ? smp_call_function_many_cond+0x3db/0x540 [ 3056.101939] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3056.101943] ? __x64_sys_fspick+0x210/0x210 [ 3056.101947] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3056.101950] on_each_cpu_cond_mask+0x46/0x150 [ 3056.101954] on_each_cpu_cond+0x1c/0x30 [ 3056.101957] invalidate_bh_lrus+0x27/0x30 [ 3056.101962] invalidate_bdev+0x5a/0x90 [ 3056.101967] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3056.102080] generic_shutdown_super+0xb1/0x1b0 [ 3056.102087] kill_block_super+0x29/0x80 [ 3056.102092] deactivate_locked_super+0x52/0xd0 [ 3056.102096] deactivate_super+0x83/0x90 [ 3056.102101] cleanup_mnt+0x5f/0xe0 [ 3056.102106] __cleanup_mnt+0x16/0x20 [ 3056.102110] task_work_run+0xc6/0x110 [ 3056.102117] exit_to_usermode_loop+0x1dd/0x1f0 [ 3056.117507] [ 3056.117519] ? on_each_cpu+0x51/0x120 [ 3056.118015] ? smp_call_function_many_cond+0x3d9/0x540 [ 3056.118021] ? smp_call_function_many_cond+0x3a8/0x540 [ 3056.118025] ? load_new_mm_cr3+0x120/0x120 [ 3056.125869] do_syscall_64+0x3d6/0x440 [ 3056.125889] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3056.125900] RIP: 0033:0x7fbb019648fb [ 3056.125909] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3056.125914] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3056.125921] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3056.125923] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3056.125925] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3056.125927] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3056.125929] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3056.296433] ? load_new_mm_cr3+0x120/0x120 [ 3056.297128] on_each_cpu+0x51/0x120 [ 3056.297794] flush_tlb_kernel_range+0x93/0xb0 [ 3056.298975] free_unmap_vmap_area+0x63/0x70 [ 3056.299912] remove_vm_area+0xe7/0x110 [ 3056.300611] __vunmap+0x9c/0x400 [ 3056.301220] vfree+0x4d/0x100 [ 3056.301723] free_work+0x34/0x50 [libcfs] [ 3056.302602] process_one_work+0x2c8/0x700 [ 3056.303339] worker_thread+0x296/0x6e0 [ 3056.304315] ? rescuer_thread+0x570/0x570 [ 3056.305034] kthread+0x1d1/0x200 [ 3056.305656] ? set_kthread_struct+0x70/0x70 [ 3056.306420] ret_from_fork+0x1f/0x30 [ 3060.703029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 702s! [ 3060.710660] Showing busy workqueues and worker pools: [ 3060.714015] workqueue events: flags=0x0 [ 3060.715911] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3060.719549] in-flight: 56293:free_work [libcfs] [ 3060.721875] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3060.724980] in-flight: 22277:free_work [libcfs] [ 3060.727348] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3060.731150] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3060.734197] pending: nfs4_renew_state [nfsv4] [ 3060.736243] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3060.739528] in-flight: 47596:free_work [libcfs] [ 3060.742576] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3060.746619] workqueue events_unbound: flags=0x2 [ 3060.748666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3060.754719] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3060.761775] workqueue events_power_efficient: flags=0x80 [ 3060.764597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3060.767953] pending: neigh_periodic_work, do_cache_clean [ 3060.771064] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3060.778253] pending: check_lifetime [ 3060.782870] workqueue mm_percpu_wq: flags=0x8 [ 3060.794605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3060.799943] pending: vmstat_update [ 3060.804116] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3060.812829] pending: vmstat_update [ 3060.816803] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3060.823979] pending: vmstat_update [ 3060.825958] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3060.828736] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3060.832854] pending: cgroup_pidlist_destroy_work_fn [ 3060.835940] workqueue writeback: flags=0x4a [ 3060.841286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3060.844620] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3060.849807] workqueue xprtiod: flags=0xa [ 3060.851360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3060.854778] pending: xprt_autoclose [ 3060.856800] workqueue ipv6_addrconf: flags=0x40008 [ 3060.859119] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3060.869865] pending: addrconf_verify_work [ 3060.874095] workqueue knbd0-recv: flags=0x1a [ 3060.876512] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3060.883879] in-flight: 655:recv_work [ 3060.888939] workqueue ptlrpc_pinger: flags=0x2 [ 3060.892878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3060.896698] pending: ptlrpc_pinger_main [ptlrpc] [ 3060.900397] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=703s workers=5 idle: 53684 27131 54850 46584 [ 3060.904708] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=703s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3060.912358] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=703s workers=6 idle: 53660 37674 46417 53664 46418 [ 3060.921609] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3050s workers=2 idle: 657 [ 3068.061847] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3068.068097] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3068.106983] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3068.116017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3068.122728] Workqueue: events free_work [libcfs] [ 3068.128012] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3068.136688] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3068.163590] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3068.177514] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3068.184289] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3068.189408] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3068.194658] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3068.199127] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3068.203057] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3068.209620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3068.214572] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3068.220205] Call Trace: [ 3068.221322] [ 3068.222316] ? show_regs.cold.9+0x22/0x2f [ 3068.224071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3068.226528] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3068.228667] ? watchdog+0x40/0x40 [ 3068.230406] ? hrtimer_interrupt+0x128/0x2c0 [ 3068.232281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3068.234007] ? apic_timer_interrupt+0xf/0x20 [ 3068.237941] [ 3068.238789] ? on_each_cpu+0x51/0x120 [ 3068.240156] ? smp_call_function_many_cond+0x3db/0x540 [ 3068.243631] ? smp_call_function_many_cond+0x3a8/0x540 [ 3068.246556] ? load_new_mm_cr3+0x120/0x120 [ 3068.248653] ? load_new_mm_cr3+0x120/0x120 [ 3068.253916] on_each_cpu+0x51/0x120 [ 3068.255240] flush_tlb_kernel_range+0x93/0xb0 [ 3068.257086] free_unmap_vmap_area+0x63/0x70 [ 3068.261801] remove_vm_area+0xe7/0x110 [ 3068.265782] __vunmap+0x9c/0x400 [ 3068.267460] vfree+0x4d/0x100 [ 3068.269536] free_work+0x34/0x50 [libcfs] [ 3068.272671] process_one_work+0x2c8/0x700 [ 3068.274796] worker_thread+0x296/0x6e0 [ 3068.278828] ? rescuer_thread+0x570/0x570 [ 3068.280695] kthread+0x1d1/0x200 [ 3068.282868] ? set_kthread_struct+0x70/0x70 [ 3068.286418] ret_from_fork+0x1f/0x30 [ 3084.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3084.063744] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3084.102215] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3084.109130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3084.113014] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3084.115483] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3084.122700] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3084.125629] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3084.130933] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3084.134307] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3084.139035] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3084.144690] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3084.148512] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3084.151379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3084.155093] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3084.161215] Call Trace: [ 3084.162692] [ 3084.163728] ? show_regs.cold.9+0x22/0x2f [ 3084.166687] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3084.168690] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3084.170963] ? watchdog+0x40/0x40 [ 3084.173766] ? hrtimer_interrupt+0x128/0x2c0 [ 3084.176322] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3084.180844] ? apic_timer_interrupt+0xf/0x20 [ 3084.183705] [ 3084.185046] ? on_each_cpu_cond_mask+0x46/0x150 [ 3084.186751] ? __x64_sys_fspick+0x210/0x210 [ 3084.191257] ? smp_call_function_many_cond+0x3db/0x540 [ 3084.196903] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3084.201232] ? __x64_sys_fspick+0x210/0x210 [ 3084.202749] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3084.209455] on_each_cpu_cond_mask+0x46/0x150 [ 3084.213341] on_each_cpu_cond+0x1c/0x30 [ 3084.215979] invalidate_bh_lrus+0x27/0x30 [ 3084.218057] invalidate_bdev+0x5a/0x90 [ 3084.222401] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3084.226788] generic_shutdown_super+0xb1/0x1b0 [ 3084.231324] kill_block_super+0x29/0x80 [ 3084.235628] deactivate_locked_super+0x52/0xd0 [ 3084.237835] deactivate_super+0x83/0x90 [ 3084.240279] cleanup_mnt+0x5f/0xe0 [ 3084.243252] __cleanup_mnt+0x16/0x20 [ 3084.246518] task_work_run+0xc6/0x110 [ 3084.248491] exit_to_usermode_loop+0x1dd/0x1f0 [ 3084.255223] do_syscall_64+0x3d6/0x440 [ 3084.258822] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3084.261223] RIP: 0033:0x7fbb019648fb [ 3084.262639] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3084.273353] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3084.278583] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3084.285349] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3084.291213] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3084.295272] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3084.298932] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3088.055916] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3088.062574] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3088.094395] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3088.100575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3088.105785] Workqueue: events free_work [libcfs] [ 3088.110044] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3088.119410] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3088.136892] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3088.145947] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3088.152241] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3088.156764] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3088.166219] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3088.170287] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3088.175296] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3088.178690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3088.182778] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3088.189460] Call Trace: [ 3088.191739] [ 3088.193933] ? show_regs.cold.9+0x22/0x2f [ 3088.196241] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3088.200027] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3088.202925] ? watchdog+0x40/0x40 [ 3088.204248] ? hrtimer_interrupt+0x128/0x2c0 [ 3088.208590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3088.214227] ? apic_timer_interrupt+0xf/0x20 [ 3088.216571] [ 3088.218309] ? on_each_cpu+0x51/0x120 [ 3088.220349] ? smp_call_function_many_cond+0x3db/0x540 [ 3088.223932] ? smp_call_function_many_cond+0x3a8/0x540 [ 3088.226416] ? load_new_mm_cr3+0x120/0x120 [ 3088.228726] ? load_new_mm_cr3+0x120/0x120 [ 3088.230613] on_each_cpu+0x51/0x120 [ 3088.231553] flush_tlb_kernel_range+0x93/0xb0 [ 3088.234307] free_unmap_vmap_area+0x63/0x70 [ 3088.237232] remove_vm_area+0xe7/0x110 [ 3088.239915] __vunmap+0x9c/0x400 [ 3088.241435] vfree+0x4d/0x100 [ 3088.243386] free_work+0x34/0x50 [libcfs] [ 3088.247980] process_one_work+0x2c8/0x700 [ 3088.250661] worker_thread+0x296/0x6e0 [ 3088.254052] ? rescuer_thread+0x570/0x570 [ 3088.256152] kthread+0x1d1/0x200 [ 3088.258990] ? set_kthread_struct+0x70/0x70 [ 3088.261537] ret_from_fork+0x1f/0x30 [ 3091.425191] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 733s! [ 3091.431562] Showing busy workqueues and worker pools: [ 3091.437320] workqueue events: flags=0x0 [ 3091.440792] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3091.448609] in-flight: 56293:free_work [libcfs] [ 3091.453214] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3091.461575] in-flight: 22277:free_work [libcfs] [ 3091.468016] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3091.472204] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3091.479625] pending: nfs4_renew_state [nfsv4] [ 3091.482932] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3091.490056] in-flight: 47596:free_work [libcfs] [ 3091.496907] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3091.503342] workqueue events_unbound: flags=0x2 [ 3091.506197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3091.515471] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3091.528113] workqueue events_power_efficient: flags=0x80 [ 3091.531604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3091.538369] pending: neigh_periodic_work, do_cache_clean [ 3091.542888] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3091.547769] pending: check_lifetime [ 3091.552523] workqueue mm_percpu_wq: flags=0x8 [ 3091.557126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3091.561525] pending: vmstat_update [ 3091.564043] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3091.574971] pending: vmstat_update [ 3091.577932] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3091.581818] pending: vmstat_update [ 3091.585652] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3091.590203] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3091.594550] pending: cgroup_pidlist_destroy_work_fn [ 3091.599781] workqueue writeback: flags=0x4a [ 3091.604829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3091.610194] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3091.620513] workqueue xprtiod: flags=0xa [ 3091.622860] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3091.626980] pending: xprt_autoclose [ 3091.629231] workqueue ipv6_addrconf: flags=0x40008 [ 3091.631793] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3091.635241] pending: addrconf_verify_work [ 3091.638025] workqueue knbd0-recv: flags=0x1a [ 3091.640718] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3091.644690] in-flight: 655:recv_work [ 3091.652945] workqueue ptlrpc_pinger: flags=0x2 [ 3091.659213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3091.663055] pending: ptlrpc_pinger_main [ptlrpc] [ 3091.671962] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=733s workers=5 idle: 53684 27131 54850 46584 [ 3091.678805] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=733s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3091.693620] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=733s workers=6 idle: 53660 37674 46417 53664 46418 [ 3091.709642] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3081s workers=2 idle: 657 [ 3096.059816] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3096.069820] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3096.118177] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3096.124986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3096.133205] Workqueue: events free_work [libcfs] [ 3096.143048] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3096.145662] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3096.163854] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3096.170624] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3096.175048] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3096.180177] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3096.185889] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3096.192759] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3096.200011] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3096.205906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3096.217233] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3096.225168] Call Trace: [ 3096.226747] [ 3096.228768] ? show_regs.cold.9+0x22/0x2f [ 3096.230873] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3096.235627] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3096.239043] ? watchdog+0x40/0x40 [ 3096.241652] ? hrtimer_interrupt+0x128/0x2c0 [ 3096.244310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3096.248220] ? apic_timer_interrupt+0xf/0x20 [ 3096.250828] [ 3096.252158] ? on_each_cpu+0x51/0x120 [ 3096.254411] ? smp_call_function_many_cond+0x3db/0x540 [ 3096.256866] ? smp_call_function_many_cond+0x3a8/0x540 [ 3096.260773] ? load_new_mm_cr3+0x120/0x120 [ 3096.263040] ? load_new_mm_cr3+0x120/0x120 [ 3096.265490] on_each_cpu+0x51/0x120 [ 3096.267174] flush_tlb_kernel_range+0x93/0xb0 [ 3096.268834] free_unmap_vmap_area+0x63/0x70 [ 3096.271546] remove_vm_area+0xe7/0x110 [ 3096.274733] __vunmap+0x9c/0x400 [ 3096.276690] vfree+0x4d/0x100 [ 3096.278359] free_work+0x34/0x50 [libcfs] [ 3096.280594] process_one_work+0x2c8/0x700 [ 3096.282635] worker_thread+0x296/0x6e0 [ 3096.284655] ? rescuer_thread+0x570/0x570 [ 3096.286533] kthread+0x1d1/0x200 [ 3096.287897] ? set_kthread_struct+0x70/0x70 [ 3096.289857] ret_from_fork+0x1f/0x30 [ 3112.058041] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3112.061855] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3112.101094] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3112.106218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3112.111608] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3112.115815] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3112.128891] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3112.135560] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3112.142401] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3112.147870] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3112.155752] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3112.163110] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3112.169619] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3112.177321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3112.180515] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3112.184201] Call Trace: [ 3112.185491] [ 3112.186306] ? show_regs.cold.9+0x22/0x2f [ 3112.188063] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3112.190336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3112.192776] ? watchdog+0x40/0x40 [ 3112.194319] ? hrtimer_interrupt+0x128/0x2c0 [ 3112.196316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3112.198401] ? apic_timer_interrupt+0xf/0x20 [ 3112.200480] [ 3112.201622] ? on_each_cpu_cond_mask+0x46/0x150 [ 3112.203714] ? __x64_sys_fspick+0x210/0x210 [ 3112.205852] ? smp_call_function_many_cond+0x3db/0x540 [ 3112.208416] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3112.210310] ? __x64_sys_fspick+0x210/0x210 [ 3112.212268] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3112.214234] on_each_cpu_cond_mask+0x46/0x150 [ 3112.216427] on_each_cpu_cond+0x1c/0x30 [ 3112.218424] invalidate_bh_lrus+0x27/0x30 [ 3112.220438] invalidate_bdev+0x5a/0x90 [ 3112.222309] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3112.224778] generic_shutdown_super+0xb1/0x1b0 [ 3112.227222] kill_block_super+0x29/0x80 [ 3112.229021] deactivate_locked_super+0x52/0xd0 [ 3112.231380] deactivate_super+0x83/0x90 [ 3112.233327] cleanup_mnt+0x5f/0xe0 [ 3112.235170] __cleanup_mnt+0x16/0x20 [ 3112.237088] task_work_run+0xc6/0x110 [ 3112.238838] exit_to_usermode_loop+0x1dd/0x1f0 [ 3112.240334] do_syscall_64+0x3d6/0x440 [ 3112.242295] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3112.244068] RIP: 0033:0x7fbb019648fb [ 3112.245201] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3112.253442] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3112.257168] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3112.260730] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3112.264060] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3112.267420] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3112.271140] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3116.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3116.060769] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3116.113976] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3116.123412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3116.130395] Workqueue: events free_work [libcfs] [ 3116.133704] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3116.137557] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3116.149688] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3116.154681] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3116.160118] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3116.165531] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3116.172715] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3116.178630] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3116.183613] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3116.190196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3116.195793] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3116.199995] Call Trace: [ 3116.201034] [ 3116.203676] ? show_regs.cold.9+0x22/0x2f [ 3116.208920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3116.217059] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3116.224590] ? watchdog+0x40/0x40 [ 3116.226613] ? hrtimer_interrupt+0x128/0x2c0 [ 3116.229210] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3116.232333] ? apic_timer_interrupt+0xf/0x20 [ 3116.234711] [ 3116.235772] ? on_each_cpu+0x51/0x120 [ 3116.237764] ? smp_call_function_many_cond+0x3d9/0x540 [ 3116.241055] ? smp_call_function_many_cond+0x3a8/0x540 [ 3116.243808] ? load_new_mm_cr3+0x120/0x120 [ 3116.246241] ? load_new_mm_cr3+0x120/0x120 [ 3116.247795] on_each_cpu+0x51/0x120 [ 3116.249657] flush_tlb_kernel_range+0x93/0xb0 [ 3116.252324] free_unmap_vmap_area+0x63/0x70 [ 3116.254647] remove_vm_area+0xe7/0x110 [ 3116.258482] __vunmap+0x9c/0x400 [ 3116.260469] vfree+0x4d/0x100 [ 3116.266751] free_work+0x34/0x50 [libcfs] [ 3116.273240] process_one_work+0x2c8/0x700 [ 3116.274645] worker_thread+0x296/0x6e0 [ 3116.277720] ? rescuer_thread+0x570/0x570 [ 3116.280143] kthread+0x1d1/0x200 [ 3116.281364] ? set_kthread_struct+0x70/0x70 [ 3116.285508] ret_from_fork+0x1f/0x30 [ 3122.143029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 763s! [ 3122.148731] Showing busy workqueues and worker pools: [ 3122.152099] workqueue events: flags=0x0 [ 3122.154378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3122.158499] in-flight: 56293:free_work [libcfs] [ 3122.163750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3122.170648] in-flight: 22277:free_work [libcfs] [ 3122.174070] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3122.179653] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3122.182573] pending: nfs4_renew_state [nfsv4] [ 3122.184929] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3122.189284] in-flight: 47596:free_work [libcfs] [ 3122.191223] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3122.194988] workqueue events_unbound: flags=0x2 [ 3122.197383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3122.200916] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3122.206650] workqueue events_power_efficient: flags=0x80 [ 3122.212735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3122.215815] pending: neigh_periodic_work, do_cache_clean [ 3122.218239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3122.221768] pending: check_lifetime [ 3122.223606] workqueue mm_percpu_wq: flags=0x8 [ 3122.227130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3122.232044] pending: vmstat_update [ 3122.235320] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3122.241292] pending: vmstat_update [ 3122.244471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3122.246211] pending: vmstat_update [ 3122.247415] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3122.249703] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3122.252300] pending: cgroup_pidlist_destroy_work_fn [ 3122.254453] workqueue writeback: flags=0x4a [ 3122.255981] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3122.258595] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3122.261510] workqueue xprtiod: flags=0xa [ 3122.263222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3122.265865] pending: xprt_autoclose [ 3122.267536] workqueue ipv6_addrconf: flags=0x40008 [ 3122.269830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3122.272768] pending: addrconf_verify_work [ 3122.274453] workqueue knbd0-recv: flags=0x1a [ 3122.276576] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3122.280960] in-flight: 655:recv_work [ 3122.282667] workqueue ptlrpc_pinger: flags=0x2 [ 3122.285224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3122.288790] pending: ptlrpc_pinger_main [ptlrpc] [ 3122.291657] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=764s workers=5 idle: 53684 27131 54850 46584 [ 3122.296354] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=764s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3122.301359] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=764s workers=6 idle: 53660 37674 46417 53664 46418 [ 3122.305143] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3112s workers=2 idle: 657 [ 3124.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3124.062280] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3124.082589] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3124.086984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3124.090337] Workqueue: events free_work [libcfs] [ 3124.092181] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3124.094345] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3124.101135] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3124.102784] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3124.104283] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3124.105923] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3124.108229] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3124.111227] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3124.113969] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3124.116607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3124.118446] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3124.120519] Call Trace: [ 3124.121594] [ 3124.122560] ? show_regs.cold.9+0x22/0x2f [ 3124.124225] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3124.125546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3124.127198] ? watchdog+0x40/0x40 [ 3124.128630] ? hrtimer_interrupt+0x128/0x2c0 [ 3124.130293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3124.132342] ? apic_timer_interrupt+0xf/0x20 [ 3124.134258] [ 3124.135117] ? on_each_cpu+0x51/0x120 [ 3124.136777] ? smp_call_function_many_cond+0x3db/0x540 [ 3124.139155] ? smp_call_function_many_cond+0x3a8/0x540 [ 3124.141494] ? load_new_mm_cr3+0x120/0x120 [ 3124.143944] ? load_new_mm_cr3+0x120/0x120 [ 3124.145023] on_each_cpu+0x51/0x120 [ 3124.146415] flush_tlb_kernel_range+0x93/0xb0 [ 3124.148498] free_unmap_vmap_area+0x63/0x70 [ 3124.151495] remove_vm_area+0xe7/0x110 [ 3124.153280] __vunmap+0x9c/0x400 [ 3124.154523] vfree+0x4d/0x100 [ 3124.156361] free_work+0x34/0x50 [libcfs] [ 3124.159640] process_one_work+0x2c8/0x700 [ 3124.161259] worker_thread+0x296/0x6e0 [ 3124.165038] ? rescuer_thread+0x570/0x570 [ 3124.168760] kthread+0x1d1/0x200 [ 3124.171717] ? set_kthread_struct+0x70/0x70 [ 3124.175124] ret_from_fork+0x1f/0x30 [ 3137.753015] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 3137.756545] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 3137.761338] (detected by 2, t=780023 jiffies, g=626849, q=1398) [ 3137.764551] Sending NMI from CPU 2 to CPUs 3: [ 3137.773004] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 3137.773015] rcu: rcu_sched kthread starved for 779665 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3137.793300] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3137.799012] rcu: RCU grace-period kthread stack dump: [ 3137.805845] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3137.819975] Call Trace: [ 3137.821296] __schedule+0x351/0xcb0 [ 3137.826793] schedule+0xc0/0x180 [ 3137.832233] schedule_timeout+0xb4/0x190 [ 3137.835371] ? do_raw_spin_unlock+0x75/0x190 [ 3137.837735] ? __next_timer_interrupt+0x160/0x160 [ 3137.841363] ? prepare_to_swait_event+0xf3/0x140 [ 3137.845104] rcu_gp_kthread+0x8dd/0x15b0 [ 3137.852727] ? force_qs_rnp+0x2c0/0x2c0 [ 3137.856287] kthread+0x1d1/0x200 [ 3137.859879] ? set_kthread_struct+0x70/0x70 [ 3137.862762] ret_from_fork+0x1f/0x30 [ 3137.865094] rcu: Stack dump where RCU GP kthread last ran: [ 3137.870808] NMI backtrace for cpu 2 [ 3137.872189] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3137.883067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3137.892876] Workqueue: events free_work [libcfs] [ 3137.896573] Call Trace: [ 3137.897553] [ 3137.898419] ? dump_stack+0xbb/0x10e [ 3137.901961] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3137.904795] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3137.910127] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3137.913525] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3137.915039] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3137.916519] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3137.923201] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 3137.926953] ? update_process_times+0x6d/0xc0 [ 3137.936229] ? tick_sched_handle.isra.18+0x27/0x90 [ 3137.939558] ? tick_sched_timer+0x53/0xb0 [ 3137.942531] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3137.946459] ? tick_sched_handle.isra.18+0x90/0x90 [ 3137.950041] ? hrtimer_interrupt+0x128/0x2c0 [ 3137.953236] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3137.957998] ? apic_timer_interrupt+0xf/0x20 [ 3137.959656] [ 3137.960476] ? on_each_cpu+0x51/0x120 [ 3137.967328] ? smp_call_function_many_cond+0x3db/0x540 [ 3137.975277] ? smp_call_function_many_cond+0x3a8/0x540 [ 3137.979823] ? load_new_mm_cr3+0x120/0x120 [ 3137.983421] ? load_new_mm_cr3+0x120/0x120 [ 3137.986332] ? on_each_cpu+0x51/0x120 [ 3137.988916] ? flush_tlb_kernel_range+0x93/0xb0 [ 3137.995776] ? free_unmap_vmap_area+0x63/0x70 [ 3137.998840] ? remove_vm_area+0xe7/0x110 [ 3138.002704] ? __vunmap+0x9c/0x400 [ 3138.004990] ? vfree+0x4d/0x100 [ 3138.008791] ? free_work+0x34/0x50 [libcfs] [ 3138.012601] ? process_one_work+0x2c8/0x700 [ 3138.017052] ? worker_thread+0x296/0x6e0 [ 3138.021700] ? rescuer_thread+0x570/0x570 [ 3138.027576] ? kthread+0x1d1/0x200 [ 3138.029715] ? set_kthread_struct+0x70/0x70 [ 3138.031954] ? ret_from_fork+0x1f/0x30 [ 3140.060402] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3140.067188] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3140.110890] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3140.118288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3140.128463] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3140.131824] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3140.162484] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3140.168170] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3140.178681] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3140.183259] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3140.188870] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3140.196439] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3140.199308] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3140.206105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3140.215070] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3140.218779] Call Trace: [ 3140.220220] [ 3140.221338] ? show_regs.cold.9+0x22/0x2f [ 3140.228450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3140.232378] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3140.234163] ? watchdog+0x40/0x40 [ 3140.238976] ? hrtimer_interrupt+0x128/0x2c0 [ 3140.242200] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3140.248939] ? apic_timer_interrupt+0xf/0x20 [ 3140.251506] [ 3140.252706] ? on_each_cpu_cond_mask+0x46/0x150 [ 3140.255243] ? __x64_sys_fspick+0x210/0x210 [ 3140.258515] ? smp_call_function_many_cond+0x3db/0x540 [ 3140.262590] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3140.265664] ? __x64_sys_fspick+0x210/0x210 [ 3140.267556] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3140.269687] on_each_cpu_cond_mask+0x46/0x150 [ 3140.271857] on_each_cpu_cond+0x1c/0x30 [ 3140.275877] invalidate_bh_lrus+0x27/0x30 [ 3140.284094] invalidate_bdev+0x5a/0x90 [ 3140.285490] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3140.290136] generic_shutdown_super+0xb1/0x1b0 [ 3140.293462] kill_block_super+0x29/0x80 [ 3140.299066] deactivate_locked_super+0x52/0xd0 [ 3140.301700] deactivate_super+0x83/0x90 [ 3140.304829] cleanup_mnt+0x5f/0xe0 [ 3140.306831] __cleanup_mnt+0x16/0x20 [ 3140.308670] task_work_run+0xc6/0x110 [ 3140.309901] exit_to_usermode_loop+0x1dd/0x1f0 [ 3140.311233] do_syscall_64+0x3d6/0x440 [ 3140.324982] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3140.334879] RIP: 0033:0x7fbb019648fb [ 3140.339960] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3140.356380] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3140.362648] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3140.366378] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3140.371820] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3140.377193] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3140.384375] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3148.055651] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3148.062884] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3148.119869] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3148.128858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3148.136825] Workqueue: events free_work [libcfs] [ 3148.141608] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3148.144062] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3148.158007] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3148.161923] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3148.169718] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3148.174114] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3148.178717] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3148.188022] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3148.198848] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3148.215899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3148.222960] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3148.233941] Call Trace: [ 3148.236100] [ 3148.242249] ? show_regs.cold.9+0x22/0x2f [ 3148.249333] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3148.256778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3148.261544] ? watchdog+0x40/0x40 [ 3148.262898] ? hrtimer_interrupt+0x128/0x2c0 [ 3148.275471] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3148.277300] ? apic_timer_interrupt+0xf/0x20 [ 3148.290870] [ 3148.291641] ? on_each_cpu+0x51/0x120 [ 3148.295948] ? smp_call_function_many_cond+0x3d9/0x540 [ 3148.302140] ? smp_call_function_many_cond+0x3a8/0x540 [ 3148.306124] ? load_new_mm_cr3+0x120/0x120 [ 3148.311790] ? load_new_mm_cr3+0x120/0x120 [ 3148.317641] on_each_cpu+0x51/0x120 [ 3148.321408] flush_tlb_kernel_range+0x93/0xb0 [ 3148.325947] free_unmap_vmap_area+0x63/0x70 [ 3148.327470] remove_vm_area+0xe7/0x110 [ 3148.334687] __vunmap+0x9c/0x400 [ 3148.337257] vfree+0x4d/0x100 [ 3148.338562] free_work+0x34/0x50 [libcfs] [ 3148.344534] process_one_work+0x2c8/0x700 [ 3148.347445] worker_thread+0x296/0x6e0 [ 3148.348915] ? rescuer_thread+0x570/0x570 [ 3148.354951] kthread+0x1d1/0x200 [ 3148.356975] ? set_kthread_struct+0x70/0x70 [ 3148.358985] ret_from_fork+0x1f/0x30 [ 3152.351028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 794s! [ 3152.356528] Showing busy workqueues and worker pools: [ 3152.359950] workqueue events: flags=0x0 [ 3152.361491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3152.365974] in-flight: 56293:free_work [libcfs] [ 3152.369693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3152.375507] in-flight: 22277:free_work [libcfs] [ 3152.378528] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3152.386825] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3152.392078] pending: nfs4_renew_state [nfsv4] [ 3152.400674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3152.407985] in-flight: 47596:free_work [libcfs] [ 3152.412830] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3152.421403] workqueue events_unbound: flags=0x2 [ 3152.423203] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3152.430568] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3152.442154] workqueue events_power_efficient: flags=0x80 [ 3152.446397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3152.452055] pending: neigh_periodic_work, do_cache_clean [ 3152.459509] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3152.465176] pending: check_lifetime [ 3152.470958] workqueue mm_percpu_wq: flags=0x8 [ 3152.477638] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3152.486464] pending: vmstat_update [ 3152.489625] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3152.498628] pending: vmstat_update [ 3152.502008] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3152.507356] pending: vmstat_update [ 3152.509704] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3152.513022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3152.518436] pending: cgroup_pidlist_destroy_work_fn [ 3152.521923] workqueue writeback: flags=0x4a [ 3152.524817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3152.530491] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3152.539454] workqueue xprtiod: flags=0xa [ 3152.543924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3152.548791] pending: xprt_autoclose [ 3152.550978] workqueue ipv6_addrconf: flags=0x40008 [ 3152.555845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3152.562650] pending: addrconf_verify_work [ 3152.565907] workqueue knbd0-recv: flags=0x1a [ 3152.568660] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3152.571541] in-flight: 655:recv_work [ 3152.576498] workqueue ptlrpc_pinger: flags=0x2 [ 3152.578791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3152.582915] pending: ptlrpc_pinger_main [ptlrpc] [ 3152.588855] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=794s workers=5 idle: 53684 27131 54850 46584 [ 3152.593034] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=794s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3152.609008] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=794s workers=6 idle: 53660 37674 46417 53664 46418 [ 3152.617719] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3142s workers=2 idle: 657 [ 3164.060122] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3164.064496] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3164.106528] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3164.113170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3164.117970] Workqueue: events free_work [libcfs] [ 3164.120839] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3164.127466] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3164.139217] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3164.145043] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3164.154385] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3164.164555] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3164.171196] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3164.178068] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3164.183185] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3164.186898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3164.193196] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3164.199838] Call Trace: [ 3164.201154] [ 3164.202425] ? show_regs.cold.9+0x22/0x2f [ 3164.206246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3164.211337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3164.219425] ? watchdog+0x40/0x40 [ 3164.223642] ? hrtimer_interrupt+0x128/0x2c0 [ 3164.227540] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3164.233186] ? apic_timer_interrupt+0xf/0x20 [ 3164.240389] [ 3164.243282] ? on_each_cpu+0x51/0x120 [ 3164.245582] ? smp_call_function_many_cond+0x3d9/0x540 [ 3164.255052] ? smp_call_function_many_cond+0x3a8/0x540 [ 3164.262942] ? load_new_mm_cr3+0x120/0x120 [ 3164.267756] ? load_new_mm_cr3+0x120/0x120 [ 3164.270599] on_each_cpu+0x51/0x120 [ 3164.273243] flush_tlb_kernel_range+0x93/0xb0 [ 3164.276275] free_unmap_vmap_area+0x63/0x70 [ 3164.278567] remove_vm_area+0xe7/0x110 [ 3164.280636] __vunmap+0x9c/0x400 [ 3164.282307] vfree+0x4d/0x100 [ 3164.284208] free_work+0x34/0x50 [libcfs] [ 3164.285654] process_one_work+0x2c8/0x700 [ 3164.291607] worker_thread+0x296/0x6e0 [ 3164.295409] ? rescuer_thread+0x570/0x570 [ 3164.298784] kthread+0x1d1/0x200 [ 3164.303224] ? set_kthread_struct+0x70/0x70 [ 3164.308264] ret_from_fork+0x1f/0x30 [ 3168.058007] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3168.063427] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3168.099290] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3168.105598] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3168.112614] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3168.117109] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3168.128327] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3168.132245] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3168.136224] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3168.139952] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3168.143799] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3168.149161] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3168.152302] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3168.156262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3168.160008] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3168.164039] Call Trace: [ 3168.165193] [ 3168.166260] ? show_regs.cold.9+0x22/0x2f [ 3168.167713] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3168.170809] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3168.173617] ? watchdog+0x40/0x40 [ 3168.176097] ? hrtimer_interrupt+0x128/0x2c0 [ 3168.179649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3168.182245] ? apic_timer_interrupt+0xf/0x20 [ 3168.184557] [ 3168.185420] ? on_each_cpu_cond_mask+0x46/0x150 [ 3168.188204] ? __x64_sys_fspick+0x210/0x210 [ 3168.192008] ? smp_call_function_many_cond+0x3d9/0x540 [ 3168.194812] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3168.199142] ? __x64_sys_fspick+0x210/0x210 [ 3168.203234] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3168.206605] on_each_cpu_cond_mask+0x46/0x150 [ 3168.211779] on_each_cpu_cond+0x1c/0x30 [ 3168.214109] invalidate_bh_lrus+0x27/0x30 [ 3168.216651] invalidate_bdev+0x5a/0x90 [ 3168.219235] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3168.224040] generic_shutdown_super+0xb1/0x1b0 [ 3168.226166] kill_block_super+0x29/0x80 [ 3168.228866] deactivate_locked_super+0x52/0xd0 [ 3168.230823] deactivate_super+0x83/0x90 [ 3168.233518] cleanup_mnt+0x5f/0xe0 [ 3168.235202] __cleanup_mnt+0x16/0x20 [ 3168.237842] task_work_run+0xc6/0x110 [ 3168.242125] exit_to_usermode_loop+0x1dd/0x1f0 [ 3168.245437] do_syscall_64+0x3d6/0x440 [ 3168.247300] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3168.250335] RIP: 0033:0x7fbb019648fb [ 3168.253060] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3168.268555] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3168.271875] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3168.277156] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3168.285702] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3168.291239] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3168.293831] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3180.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3180.060178] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3180.084851] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3180.090761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3180.094977] Workqueue: events free_work [libcfs] [ 3180.097678] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3180.100572] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3180.109389] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3180.113609] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3180.119180] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3180.123778] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3180.128425] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3180.132995] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3180.144272] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3180.151053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3180.154997] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3180.159775] Call Trace: [ 3180.161148] [ 3180.162732] ? show_regs.cold.9+0x22/0x2f [ 3180.165111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3180.167937] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3180.170921] ? watchdog+0x40/0x40 [ 3180.172982] ? hrtimer_interrupt+0x128/0x2c0 [ 3180.175742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3180.178700] ? apic_timer_interrupt+0xf/0x20 [ 3180.180495] [ 3180.181349] ? on_each_cpu+0x51/0x120 [ 3180.184125] ? smp_call_function_many_cond+0x3db/0x540 [ 3180.187703] ? smp_call_function_many_cond+0x3a8/0x540 [ 3180.190118] ? load_new_mm_cr3+0x120/0x120 [ 3180.192949] ? load_new_mm_cr3+0x120/0x120 [ 3180.195720] on_each_cpu+0x51/0x120 [ 3180.197843] flush_tlb_kernel_range+0x93/0xb0 [ 3180.200701] free_unmap_vmap_area+0x63/0x70 [ 3180.203655] remove_vm_area+0xe7/0x110 [ 3180.205763] __vunmap+0x9c/0x400 [ 3180.207748] vfree+0x4d/0x100 [ 3180.209224] free_work+0x34/0x50 [libcfs] [ 3180.211850] process_one_work+0x2c8/0x700 [ 3180.213814] worker_thread+0x296/0x6e0 [ 3180.215880] ? rescuer_thread+0x570/0x570 [ 3180.217957] kthread+0x1d1/0x200 [ 3180.220054] ? set_kthread_struct+0x70/0x70 [ 3180.221741] ret_from_fork+0x1f/0x30 [ 3183.071273] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 824s! [ 3183.077649] Showing busy workqueues and worker pools: [ 3183.080186] workqueue events: flags=0x0 [ 3183.082093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3183.086201] in-flight: 56293:free_work [libcfs] [ 3183.089411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3183.094703] in-flight: 22277:free_work [libcfs] [ 3183.097354] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3183.103873] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3183.108596] pending: nfs4_renew_state [nfsv4] [ 3183.111064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3183.117126] in-flight: 47596:free_work [libcfs] [ 3183.121193] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3183.128609] workqueue events_unbound: flags=0x2 [ 3183.130354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3183.136054] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3183.141594] workqueue events_power_efficient: flags=0x80 [ 3183.144544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3183.147716] pending: neigh_periodic_work, do_cache_clean [ 3183.150388] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3183.154774] pending: check_lifetime [ 3183.157025] workqueue mm_percpu_wq: flags=0x8 [ 3183.159334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3183.163369] pending: vmstat_update [ 3183.165491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3183.168916] pending: vmstat_update [ 3183.170763] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3183.174421] pending: vmstat_update [ 3183.176933] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3183.180195] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3183.184041] pending: cgroup_pidlist_destroy_work_fn [ 3183.187089] workqueue writeback: flags=0x4a [ 3183.191100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3183.194798] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3183.200066] workqueue xprtiod: flags=0xa [ 3183.202278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3183.209993] pending: xprt_autoclose [ 3183.212423] workqueue ipv6_addrconf: flags=0x40008 [ 3183.215269] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3183.218982] pending: addrconf_verify_work [ 3183.221767] workqueue knbd0-recv: flags=0x1a [ 3183.223760] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3183.227932] in-flight: 655:recv_work [ 3183.231414] workqueue ptlrpc_pinger: flags=0x2 [ 3183.234990] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3183.238253] pending: ptlrpc_pinger_main [ptlrpc] [ 3183.240642] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=825s workers=5 idle: 53684 27131 54850 46584 [ 3183.245942] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=825s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3183.252725] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=825s workers=6 idle: 53660 37674 46417 53664 46418 [ 3183.256948] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3173s workers=2 idle: 657 [ 3192.059047] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3192.063437] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3192.115174] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3192.129295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3192.137713] Workqueue: events free_work [libcfs] [ 3192.140816] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3192.144819] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3192.158153] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3192.163784] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3192.167905] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3192.174775] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3192.180211] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3192.185194] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3192.191389] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3192.198267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3192.203581] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3192.208619] Call Trace: [ 3192.209817] [ 3192.210658] ? show_regs.cold.9+0x22/0x2f [ 3192.212303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3192.217316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3192.219856] ? watchdog+0x40/0x40 [ 3192.221996] ? hrtimer_interrupt+0x128/0x2c0 [ 3192.225994] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3192.230231] ? apic_timer_interrupt+0xf/0x20 [ 3192.233974] [ 3192.234567] ? on_each_cpu+0x51/0x120 [ 3192.237049] ? smp_call_function_many_cond+0x3db/0x540 [ 3192.240113] ? smp_call_function_many_cond+0x3a8/0x540 [ 3192.243714] ? load_new_mm_cr3+0x120/0x120 [ 3192.246772] ? load_new_mm_cr3+0x120/0x120 [ 3192.249105] on_each_cpu+0x51/0x120 [ 3192.250620] flush_tlb_kernel_range+0x93/0xb0 [ 3192.253140] free_unmap_vmap_area+0x63/0x70 [ 3192.255194] remove_vm_area+0xe7/0x110 [ 3192.257595] __vunmap+0x9c/0x400 [ 3192.259521] vfree+0x4d/0x100 [ 3192.261464] free_work+0x34/0x50 [libcfs] [ 3192.263769] process_one_work+0x2c8/0x700 [ 3192.267150] worker_thread+0x296/0x6e0 [ 3192.269106] ? rescuer_thread+0x570/0x570 [ 3192.271170] kthread+0x1d1/0x200 [ 3192.272633] ? set_kthread_struct+0x70/0x70 [ 3192.275528] ret_from_fork+0x1f/0x30 [ 3196.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3196.063708] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3196.094681] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3196.100588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3196.105438] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3196.108294] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3196.116725] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3196.120510] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3196.124731] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3196.128402] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3196.131911] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3196.135307] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3196.138619] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3196.142084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3196.145089] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3196.148386] Call Trace: [ 3196.149573] [ 3196.150537] ? show_regs.cold.9+0x22/0x2f [ 3196.152138] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3196.154461] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3196.156106] ? watchdog+0x40/0x40 [ 3196.157534] ? hrtimer_interrupt+0x128/0x2c0 [ 3196.159602] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3196.161623] ? apic_timer_interrupt+0xf/0x20 [ 3196.163408] [ 3196.164318] ? on_each_cpu_cond_mask+0x46/0x150 [ 3196.166265] ? __x64_sys_fspick+0x210/0x210 [ 3196.168278] ? smp_call_function_many_cond+0x3db/0x540 [ 3196.169928] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3196.171982] ? __x64_sys_fspick+0x210/0x210 [ 3196.173692] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3196.175540] on_each_cpu_cond_mask+0x46/0x150 [ 3196.177416] on_each_cpu_cond+0x1c/0x30 [ 3196.178928] invalidate_bh_lrus+0x27/0x30 [ 3196.180587] invalidate_bdev+0x5a/0x90 [ 3196.182499] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3196.184619] generic_shutdown_super+0xb1/0x1b0 [ 3196.186573] kill_block_super+0x29/0x80 [ 3196.188218] deactivate_locked_super+0x52/0xd0 [ 3196.190823] deactivate_super+0x83/0x90 [ 3196.192692] cleanup_mnt+0x5f/0xe0 [ 3196.194243] __cleanup_mnt+0x16/0x20 [ 3196.195908] task_work_run+0xc6/0x110 [ 3196.197809] exit_to_usermode_loop+0x1dd/0x1f0 [ 3196.199660] do_syscall_64+0x3d6/0x440 [ 3196.201273] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3196.203408] RIP: 0033:0x7fbb019648fb [ 3196.204842] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3196.212286] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3196.215288] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3196.218530] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3196.221585] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3196.224640] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3196.227481] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3208.056050] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3208.061148] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3208.111184] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3208.119775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3208.125596] Workqueue: events free_work [libcfs] [ 3208.128018] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3208.130574] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3208.140104] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3208.143442] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3208.145732] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3208.151558] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3208.156147] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3208.159474] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3208.162809] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3208.167806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3208.171128] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3208.175646] Call Trace: [ 3208.177251] [ 3208.179017] ? show_regs.cold.9+0x22/0x2f [ 3208.181159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3208.183619] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3208.185169] ? watchdog+0x40/0x40 [ 3208.186331] ? hrtimer_interrupt+0x128/0x2c0 [ 3208.187964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3208.189597] ? apic_timer_interrupt+0xf/0x20 [ 3208.190850] [ 3208.191692] ? on_each_cpu+0x51/0x120 [ 3208.192786] ? smp_call_function_many_cond+0x3db/0x540 [ 3208.194511] ? smp_call_function_many_cond+0x3a8/0x540 [ 3208.196445] ? load_new_mm_cr3+0x120/0x120 [ 3208.198818] ? load_new_mm_cr3+0x120/0x120 [ 3208.200947] on_each_cpu+0x51/0x120 [ 3208.202652] flush_tlb_kernel_range+0x93/0xb0 [ 3208.204680] free_unmap_vmap_area+0x63/0x70 [ 3208.206604] remove_vm_area+0xe7/0x110 [ 3208.208450] __vunmap+0x9c/0x400 [ 3208.209646] vfree+0x4d/0x100 [ 3208.210764] free_work+0x34/0x50 [libcfs] [ 3208.213220] process_one_work+0x2c8/0x700 [ 3208.215603] worker_thread+0x296/0x6e0 [ 3208.217587] ? rescuer_thread+0x570/0x570 [ 3208.219717] kthread+0x1d1/0x200 [ 3208.221069] ? set_kthread_struct+0x70/0x70 [ 3208.223572] ret_from_fork+0x1f/0x30 [ 3213.280291] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 855s! [ 3213.286829] Showing busy workqueues and worker pools: [ 3213.292016] workqueue events: flags=0x0 [ 3213.294026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3213.297427] in-flight: 56293:free_work [libcfs] [ 3213.303881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3213.306678] in-flight: 22277:free_work [libcfs] [ 3213.310978] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3213.316082] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3213.325129] pending: nfs4_renew_state [nfsv4] [ 3213.328843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3213.336177] in-flight: 47596:free_work [libcfs] [ 3213.342365] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3213.353572] workqueue events_unbound: flags=0x2 [ 3213.358110] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3213.366395] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3213.374294] workqueue events_power_efficient: flags=0x80 [ 3213.378579] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3213.381656] pending: neigh_periodic_work, do_cache_clean [ 3213.387282] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3213.395974] pending: check_lifetime [ 3213.397517] workqueue mm_percpu_wq: flags=0x8 [ 3213.400113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3213.403019] pending: vmstat_update [ 3213.406634] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3213.410211] pending: vmstat_update [ 3213.413205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3213.420164] pending: vmstat_update [ 3213.422115] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3213.424375] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3213.427327] pending: cgroup_pidlist_destroy_work_fn [ 3213.430049] workqueue writeback: flags=0x4a [ 3213.431959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3213.435127] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3213.438218] workqueue xprtiod: flags=0xa [ 3213.440199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3213.443270] pending: xprt_autoclose [ 3213.444961] workqueue ipv6_addrconf: flags=0x40008 [ 3213.447225] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3213.451439] pending: addrconf_verify_work [ 3213.457788] workqueue knbd0-recv: flags=0x1a [ 3213.462838] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3213.465722] in-flight: 655:recv_work [ 3213.467468] workqueue ptlrpc_pinger: flags=0x2 [ 3213.469515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3213.472960] pending: ptlrpc_pinger_main [ptlrpc] [ 3213.475919] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=855s workers=5 idle: 53684 27131 54850 46584 [ 3213.484255] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=855s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3213.489921] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=855s workers=6 idle: 53660 37674 46417 53664 46418 [ 3213.495514] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3203s workers=2 idle: 657 [ 3220.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3220.064846] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3220.124989] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3220.133462] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3220.145684] Workqueue: events free_work [libcfs] [ 3220.151055] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3220.157971] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3220.183839] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3220.188918] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3220.198425] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3220.206267] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3220.218755] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3220.235912] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3220.258517] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3220.284886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3220.298591] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3220.306494] Call Trace: [ 3220.308881] [ 3220.311996] ? show_regs.cold.9+0x22/0x2f [ 3220.324898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3220.329228] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3220.335485] ? watchdog+0x40/0x40 [ 3220.336555] ? hrtimer_interrupt+0x128/0x2c0 [ 3220.347026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3220.351157] ? apic_timer_interrupt+0xf/0x20 [ 3220.355019] [ 3220.356533] ? on_each_cpu+0x51/0x120 [ 3220.362499] ? smp_call_function_many_cond+0x3db/0x540 [ 3220.365115] ? smp_call_function_many_cond+0x3a8/0x540 [ 3220.371537] ? load_new_mm_cr3+0x120/0x120 [ 3220.378584] ? load_new_mm_cr3+0x120/0x120 [ 3220.381955] on_each_cpu+0x51/0x120 [ 3220.389229] flush_tlb_kernel_range+0x93/0xb0 [ 3220.394093] free_unmap_vmap_area+0x63/0x70 [ 3220.405367] remove_vm_area+0xe7/0x110 [ 3220.409667] __vunmap+0x9c/0x400 [ 3220.413441] vfree+0x4d/0x100 [ 3220.418227] free_work+0x34/0x50 [libcfs] [ 3220.425356] process_one_work+0x2c8/0x700 [ 3220.427490] worker_thread+0x296/0x6e0 [ 3220.430271] ? rescuer_thread+0x570/0x570 [ 3220.434054] kthread+0x1d1/0x200 [ 3220.437954] ? set_kthread_struct+0x70/0x70 [ 3220.441230] ret_from_fork+0x1f/0x30 [ 3224.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3224.061629] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3224.105709] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3224.130090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3224.148428] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3224.154664] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3224.163909] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3224.175857] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3224.187151] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3224.201718] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3224.215342] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3224.226937] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3224.236379] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3224.238655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3224.251200] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3224.262792] Call Trace: [ 3224.267140] [ 3224.270248] ? show_regs.cold.9+0x22/0x2f [ 3224.274182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3224.286509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3224.293493] ? watchdog+0x40/0x40 [ 3224.294566] ? hrtimer_interrupt+0x128/0x2c0 [ 3224.306007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3224.311943] ? apic_timer_interrupt+0xf/0x20 [ 3224.313094] [ 3224.313903] ? on_each_cpu_cond_mask+0x46/0x150 [ 3224.325501] ? __x64_sys_fspick+0x210/0x210 [ 3224.327151] ? smp_call_function_many_cond+0x3db/0x540 [ 3224.333971] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3224.335601] ? __x64_sys_fspick+0x210/0x210 [ 3224.342638] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3224.345185] on_each_cpu_cond_mask+0x46/0x150 [ 3224.353278] on_each_cpu_cond+0x1c/0x30 [ 3224.360341] invalidate_bh_lrus+0x27/0x30 [ 3224.362044] invalidate_bdev+0x5a/0x90 [ 3224.372932] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3224.376325] generic_shutdown_super+0xb1/0x1b0 [ 3224.385949] kill_block_super+0x29/0x80 [ 3224.387449] deactivate_locked_super+0x52/0xd0 [ 3224.394990] deactivate_super+0x83/0x90 [ 3224.399141] cleanup_mnt+0x5f/0xe0 [ 3224.404121] __cleanup_mnt+0x16/0x20 [ 3224.409139] task_work_run+0xc6/0x110 [ 3224.412166] exit_to_usermode_loop+0x1dd/0x1f0 [ 3224.416074] do_syscall_64+0x3d6/0x440 [ 3224.417474] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3224.419967] RIP: 0033:0x7fbb019648fb [ 3224.421359] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3224.429344] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3224.433837] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3224.437786] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3224.441473] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3224.444642] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3224.452439] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3240.057016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3240.064762] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3240.093045] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3240.098788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3240.102728] Workqueue: events free_work [libcfs] [ 3240.105670] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3240.109686] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3240.119022] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3240.122625] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3240.125824] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3240.131405] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3240.134879] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3240.138575] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3240.142342] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3240.146478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3240.151555] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3240.160776] Call Trace: [ 3240.162848] [ 3240.164508] ? show_regs.cold.9+0x22/0x2f [ 3240.167705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3240.171663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3240.174443] ? watchdog+0x40/0x40 [ 3240.176534] ? hrtimer_interrupt+0x128/0x2c0 [ 3240.181708] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3240.185036] ? apic_timer_interrupt+0xf/0x20 [ 3240.188514] [ 3240.189308] ? on_each_cpu+0x51/0x120 [ 3240.192702] ? smp_call_function_many_cond+0x3db/0x540 [ 3240.195815] ? smp_call_function_many_cond+0x3a8/0x540 [ 3240.202610] ? load_new_mm_cr3+0x120/0x120 [ 3240.204174] ? load_new_mm_cr3+0x120/0x120 [ 3240.210338] on_each_cpu+0x51/0x120 [ 3240.215717] flush_tlb_kernel_range+0x93/0xb0 [ 3240.219613] free_unmap_vmap_area+0x63/0x70 [ 3240.223512] remove_vm_area+0xe7/0x110 [ 3240.225464] __vunmap+0x9c/0x400 [ 3240.228919] vfree+0x4d/0x100 [ 3240.234632] free_work+0x34/0x50 [libcfs] [ 3240.238812] process_one_work+0x2c8/0x700 [ 3240.241443] worker_thread+0x296/0x6e0 [ 3240.243035] ? rescuer_thread+0x570/0x570 [ 3240.248067] kthread+0x1d1/0x200 [ 3240.253019] ? set_kthread_struct+0x70/0x70 [ 3240.262699] ret_from_fork+0x1f/0x30 [ 3244.000645] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 885s! [ 3244.010023] Showing busy workqueues and worker pools: [ 3244.014644] workqueue events: flags=0x0 [ 3244.017341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3244.026007] in-flight: 56293:free_work [libcfs] [ 3244.028692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3244.033200] in-flight: 22277:free_work [libcfs] [ 3244.037742] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3244.047791] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3244.054366] pending: nfs4_renew_state [nfsv4] [ 3244.057746] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3244.060550] in-flight: 47596:free_work [libcfs] [ 3244.066471] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3244.074010] workqueue events_unbound: flags=0x2 [ 3244.076392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3244.087786] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3244.093381] workqueue events_power_efficient: flags=0x80 [ 3244.096635] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3244.102746] pending: neigh_periodic_work, do_cache_clean [ 3244.105747] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3244.109520] pending: check_lifetime [ 3244.111429] workqueue mm_percpu_wq: flags=0x8 [ 3244.113370] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3244.118611] pending: vmstat_update [ 3244.119856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3244.124663] pending: vmstat_update [ 3244.126951] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3244.135220] pending: vmstat_update [ 3244.137463] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3244.142690] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3244.150583] pending: cgroup_pidlist_destroy_work_fn [ 3244.155790] workqueue writeback: flags=0x4a [ 3244.157667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3244.160758] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3244.168454] workqueue xprtiod: flags=0xa [ 3244.176000] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3244.183045] pending: xprt_autoclose [ 3244.185381] workqueue ipv6_addrconf: flags=0x40008 [ 3244.190487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3244.196532] pending: addrconf_verify_work [ 3244.201486] workqueue knbd0-recv: flags=0x1a [ 3244.202965] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3244.211792] in-flight: 655:recv_work [ 3244.215217] workqueue ptlrpc_pinger: flags=0x2 [ 3244.219606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3244.223940] pending: ptlrpc_pinger_main [ptlrpc] [ 3244.226727] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=886s workers=5 idle: 53684 27131 54850 46584 [ 3244.230750] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=886s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3244.241866] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=886s workers=6 idle: 53660 37674 46417 53664 46418 [ 3244.247476] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3234s workers=2 idle: 657 [ 3248.060445] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3248.068767] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3248.129496] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3248.155500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3248.178158] Workqueue: events free_work [libcfs] [ 3248.189983] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3248.202609] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3248.231099] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3248.251968] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3248.274924] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3248.277754] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3248.297299] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3248.317742] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3248.326679] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3248.347515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3248.360989] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3248.370201] Call Trace: [ 3248.373580] [ 3248.374133] ? show_regs.cold.9+0x22/0x2f [ 3248.377945] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3248.381115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3248.385760] ? watchdog+0x40/0x40 [ 3248.388654] ? hrtimer_interrupt+0x128/0x2c0 [ 3248.391892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3248.395664] ? apic_timer_interrupt+0xf/0x20 [ 3248.405104] [ 3248.406081] ? on_each_cpu+0x51/0x120 [ 3248.408545] ? smp_call_function_many_cond+0x3db/0x540 [ 3248.413756] ? smp_call_function_many_cond+0x3a8/0x540 [ 3248.426958] ? load_new_mm_cr3+0x120/0x120 [ 3248.431658] ? load_new_mm_cr3+0x120/0x120 [ 3248.439140] on_each_cpu+0x51/0x120 [ 3248.446206] flush_tlb_kernel_range+0x93/0xb0 [ 3248.453870] free_unmap_vmap_area+0x63/0x70 [ 3248.467913] remove_vm_area+0xe7/0x110 [ 3248.469400] __vunmap+0x9c/0x400 [ 3248.470659] vfree+0x4d/0x100 [ 3248.480020] free_work+0x34/0x50 [libcfs] [ 3248.483063] process_one_work+0x2c8/0x700 [ 3248.486813] worker_thread+0x296/0x6e0 [ 3248.488828] ? rescuer_thread+0x570/0x570 [ 3248.491670] kthread+0x1d1/0x200 [ 3248.492698] ? set_kthread_struct+0x70/0x70 [ 3248.510420] ret_from_fork+0x1f/0x30 [ 3252.058226] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3252.063909] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3252.088773] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3252.098823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3252.109558] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3252.112783] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3252.131516] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3252.143666] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3252.148820] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3252.154606] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3252.165885] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3252.175491] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3252.185182] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3252.194429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3252.196663] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3252.200175] Call Trace: [ 3252.201281] [ 3252.202077] ? show_regs.cold.9+0x22/0x2f [ 3252.205451] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3252.207318] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3252.215679] ? watchdog+0x40/0x40 [ 3252.217063] ? hrtimer_interrupt+0x128/0x2c0 [ 3252.221350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3252.224073] ? apic_timer_interrupt+0xf/0x20 [ 3252.229888] [ 3252.230697] ? on_each_cpu_cond_mask+0x46/0x150 [ 3252.232504] ? __x64_sys_fspick+0x210/0x210 [ 3252.234490] ? smp_call_function_many_cond+0x3db/0x540 [ 3252.236821] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3252.238725] ? __x64_sys_fspick+0x210/0x210 [ 3252.240550] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3252.242546] on_each_cpu_cond_mask+0x46/0x150 [ 3252.247044] on_each_cpu_cond+0x1c/0x30 [ 3252.248454] invalidate_bh_lrus+0x27/0x30 [ 3252.251837] invalidate_bdev+0x5a/0x90 [ 3252.255051] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3252.257506] generic_shutdown_super+0xb1/0x1b0 [ 3252.259595] kill_block_super+0x29/0x80 [ 3252.261402] deactivate_locked_super+0x52/0xd0 [ 3252.263547] deactivate_super+0x83/0x90 [ 3252.265324] cleanup_mnt+0x5f/0xe0 [ 3252.266817] __cleanup_mnt+0x16/0x20 [ 3252.268432] task_work_run+0xc6/0x110 [ 3252.270227] exit_to_usermode_loop+0x1dd/0x1f0 [ 3252.272337] do_syscall_64+0x3d6/0x440 [ 3252.273987] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3252.275742] RIP: 0033:0x7fbb019648fb [ 3252.277254] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3252.285446] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3252.288838] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3252.291203] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3252.293824] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3252.297165] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3252.301101] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3272.056980] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3272.064803] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3272.101629] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3272.109629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3272.115082] Workqueue: events free_work [libcfs] [ 3272.116899] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3272.121015] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3272.131091] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3272.138043] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3272.144004] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3272.150416] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3272.155390] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3272.162812] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3272.171900] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3272.179591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3272.185763] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3272.193118] Call Trace: [ 3272.196688] [ 3272.197519] ? show_regs.cold.9+0x22/0x2f [ 3272.199315] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3272.202180] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3272.209070] ? watchdog+0x40/0x40 [ 3272.214150] ? hrtimer_interrupt+0x128/0x2c0 [ 3272.220128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3272.224873] ? apic_timer_interrupt+0xf/0x20 [ 3272.228824] [ 3272.231022] ? on_each_cpu+0x51/0x120 [ 3272.237770] ? smp_call_function_many_cond+0x3db/0x540 [ 3272.241053] ? smp_call_function_many_cond+0x3a8/0x540 [ 3272.244633] ? load_new_mm_cr3+0x120/0x120 [ 3272.248590] ? load_new_mm_cr3+0x120/0x120 [ 3272.250107] on_each_cpu+0x51/0x120 [ 3272.251960] flush_tlb_kernel_range+0x93/0xb0 [ 3272.253802] free_unmap_vmap_area+0x63/0x70 [ 3272.255678] remove_vm_area+0xe7/0x110 [ 3272.257167] __vunmap+0x9c/0x400 [ 3272.258606] vfree+0x4d/0x100 [ 3272.260000] free_work+0x34/0x50 [libcfs] [ 3272.261706] process_one_work+0x2c8/0x700 [ 3272.263578] worker_thread+0x296/0x6e0 [ 3272.266386] ? rescuer_thread+0x570/0x570 [ 3272.268309] kthread+0x1d1/0x200 [ 3272.270358] ? set_kthread_struct+0x70/0x70 [ 3272.274091] ret_from_fork+0x1f/0x30 [ 3274.720343] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 916s! [ 3274.724114] Showing busy workqueues and worker pools: [ 3274.727128] workqueue events: flags=0x0 [ 3274.728575] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3274.734102] in-flight: 56293:free_work [libcfs] [ 3274.736669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3274.740395] in-flight: 22277:free_work [libcfs] [ 3274.742743] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3274.751347] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3274.755068] pending: nfs4_renew_state [nfsv4] [ 3274.757719] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3274.762733] in-flight: 47596:free_work [libcfs] [ 3274.764458] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3274.777552] workqueue events_unbound: flags=0x2 [ 3274.779894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3274.785548] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3274.788960] workqueue events_power_efficient: flags=0x80 [ 3274.791099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3274.797048] pending: neigh_periodic_work, do_cache_clean [ 3274.806029] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3274.810734] pending: check_lifetime [ 3274.814230] workqueue mm_percpu_wq: flags=0x8 [ 3274.816204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3274.822793] pending: vmstat_update [ 3274.825508] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3274.832505] pending: vmstat_update [ 3274.834012] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3274.840355] pending: vmstat_update [ 3274.843318] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3274.849657] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3274.857060] pending: cgroup_pidlist_destroy_work_fn [ 3274.860227] workqueue writeback: flags=0x4a [ 3274.861233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3274.868901] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3274.872398] workqueue xprtiod: flags=0xa [ 3274.874965] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3274.881415] pending: xprt_autoclose [ 3274.883608] workqueue ipv6_addrconf: flags=0x40008 [ 3274.887487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3274.890857] pending: addrconf_verify_work [ 3274.892861] workqueue knbd0-recv: flags=0x1a [ 3274.894638] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3274.904745] in-flight: 655:recv_work [ 3274.906303] workqueue ptlrpc_pinger: flags=0x2 [ 3274.913222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3274.915477] pending: ptlrpc_pinger_main [ptlrpc] [ 3274.918382] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=917s workers=5 idle: 53684 27131 54850 46584 [ 3274.924029] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=917s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3274.934161] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=917s workers=6 idle: 53660 37674 46417 53664 46418 [ 3274.943920] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3264s workers=2 idle: 657 [ 3276.062302] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3276.067762] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3276.109825] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3276.115354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3276.119828] Workqueue: events free_work [libcfs] [ 3276.122017] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3276.124902] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3276.134876] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3276.137729] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3276.141247] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3276.146352] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3276.150334] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3276.154267] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3276.158546] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3276.162127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3276.164778] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3276.168003] Call Trace: [ 3276.169180] [ 3276.169993] ? show_regs.cold.9+0x22/0x2f [ 3276.171720] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3276.173725] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3276.175635] ? watchdog+0x40/0x40 [ 3276.177202] ? hrtimer_interrupt+0x128/0x2c0 [ 3276.179185] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3276.181121] ? apic_timer_interrupt+0xf/0x20 [ 3276.183042] [ 3276.183895] ? on_each_cpu+0x51/0x120 [ 3276.185352] ? smp_call_function_many_cond+0x3db/0x540 [ 3276.190352] ? smp_call_function_many_cond+0x3a8/0x540 [ 3276.193028] ? load_new_mm_cr3+0x120/0x120 [ 3276.194800] ? load_new_mm_cr3+0x120/0x120 [ 3276.196127] on_each_cpu+0x51/0x120 [ 3276.200143] flush_tlb_kernel_range+0x93/0xb0 [ 3276.203897] free_unmap_vmap_area+0x63/0x70 [ 3276.205971] remove_vm_area+0xe7/0x110 [ 3276.208181] __vunmap+0x9c/0x400 [ 3276.210006] vfree+0x4d/0x100 [ 3276.211247] free_work+0x34/0x50 [libcfs] [ 3276.213153] process_one_work+0x2c8/0x700 [ 3276.215046] worker_thread+0x296/0x6e0 [ 3276.216873] ? rescuer_thread+0x570/0x570 [ 3276.218837] kthread+0x1d1/0x200 [ 3276.220360] ? set_kthread_struct+0x70/0x70 [ 3276.222670] ret_from_fork+0x1f/0x30 [ 3280.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3280.062191] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3280.084252] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3280.090210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3280.096704] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3280.099967] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3280.110254] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3280.114579] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3280.118058] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3280.121248] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3280.126143] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3280.129177] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3280.132713] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3280.136426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3280.138982] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3280.142077] Call Trace: [ 3280.143126] [ 3280.144071] ? show_regs.cold.9+0x22/0x2f [ 3280.146195] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3280.148277] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3280.150195] ? watchdog+0x40/0x40 [ 3280.151715] ? hrtimer_interrupt+0x128/0x2c0 [ 3280.153543] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3280.156120] ? apic_timer_interrupt+0xf/0x20 [ 3280.157734] [ 3280.158462] ? on_each_cpu_cond_mask+0x46/0x150 [ 3280.160675] ? __x64_sys_fspick+0x210/0x210 [ 3280.162767] ? smp_call_function_many_cond+0x3db/0x540 [ 3280.164954] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3280.166725] ? __x64_sys_fspick+0x210/0x210 [ 3280.168574] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3280.170397] on_each_cpu_cond_mask+0x46/0x150 [ 3280.172166] on_each_cpu_cond+0x1c/0x30 [ 3280.173746] invalidate_bh_lrus+0x27/0x30 [ 3280.175453] invalidate_bdev+0x5a/0x90 [ 3280.176831] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3280.178730] generic_shutdown_super+0xb1/0x1b0 [ 3280.180564] kill_block_super+0x29/0x80 [ 3280.182014] deactivate_locked_super+0x52/0xd0 [ 3280.183633] deactivate_super+0x83/0x90 [ 3280.185082] cleanup_mnt+0x5f/0xe0 [ 3280.186561] __cleanup_mnt+0x16/0x20 [ 3280.187772] task_work_run+0xc6/0x110 [ 3280.189152] exit_to_usermode_loop+0x1dd/0x1f0 [ 3280.191099] do_syscall_64+0x3d6/0x440 [ 3280.192501] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3280.194617] RIP: 0033:0x7fbb019648fb [ 3280.196236] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3280.203852] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3280.207607] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3280.210695] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3280.214127] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3280.216974] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3280.219620] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3300.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3300.058685] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3300.082134] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3300.089245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3300.093451] Workqueue: events free_work [libcfs] [ 3300.097113] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3300.100011] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3300.112735] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3300.117488] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3300.124641] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3300.130033] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3300.137626] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3300.145135] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3300.151933] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3300.159073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3300.167699] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3300.175761] Call Trace: [ 3300.177361] [ 3300.178404] ? show_regs.cold.9+0x22/0x2f [ 3300.179934] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3300.183859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3300.188502] ? watchdog+0x40/0x40 [ 3300.190708] ? hrtimer_interrupt+0x128/0x2c0 [ 3300.194702] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3300.207207] ? apic_timer_interrupt+0xf/0x20 [ 3300.212018] [ 3300.213189] ? on_each_cpu+0x51/0x120 [ 3300.215813] ? smp_call_function_many_cond+0x3d9/0x540 [ 3300.220125] ? smp_call_function_many_cond+0x3a8/0x540 [ 3300.226140] ? load_new_mm_cr3+0x120/0x120 [ 3300.229578] ? load_new_mm_cr3+0x120/0x120 [ 3300.234273] on_each_cpu+0x51/0x120 [ 3300.235532] flush_tlb_kernel_range+0x93/0xb0 [ 3300.236666] free_unmap_vmap_area+0x63/0x70 [ 3300.241989] remove_vm_area+0xe7/0x110 [ 3300.243284] __vunmap+0x9c/0x400 [ 3300.247436] vfree+0x4d/0x100 [ 3300.250203] free_work+0x34/0x50 [libcfs] [ 3300.252422] process_one_work+0x2c8/0x700 [ 3300.255537] worker_thread+0x296/0x6e0 [ 3300.258569] ? rescuer_thread+0x570/0x570 [ 3300.262052] kthread+0x1d1/0x200 [ 3300.263626] ? set_kthread_struct+0x70/0x70 [ 3300.265203] ret_from_fork+0x1f/0x30 [ 3304.060326] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3304.067387] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3304.124333] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3304.130233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3304.134121] Workqueue: events free_work [libcfs] [ 3304.136046] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3304.138900] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3304.147582] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3304.151080] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3304.154212] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3304.157300] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3304.160661] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3304.163761] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3304.169156] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3304.177320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3304.187325] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3304.191193] Call Trace: [ 3304.192116] [ 3304.193292] ? show_regs.cold.9+0x22/0x2f [ 3304.195182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3304.198107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3304.200343] ? watchdog+0x40/0x40 [ 3304.202004] ? hrtimer_interrupt+0x128/0x2c0 [ 3304.204328] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3304.207306] ? apic_timer_interrupt+0xf/0x20 [ 3304.211102] [ 3304.211901] ? on_each_cpu+0x51/0x120 [ 3304.217202] ? smp_call_function_many_cond+0x3db/0x540 [ 3304.221629] ? smp_call_function_many_cond+0x3a8/0x540 [ 3304.228200] ? load_new_mm_cr3+0x120/0x120 [ 3304.229701] ? load_new_mm_cr3+0x120/0x120 [ 3304.232246] on_each_cpu+0x51/0x120 [ 3304.233969] flush_tlb_kernel_range+0x93/0xb0 [ 3304.235723] free_unmap_vmap_area+0x63/0x70 [ 3304.238298] remove_vm_area+0xe7/0x110 [ 3304.240390] __vunmap+0x9c/0x400 [ 3304.242334] vfree+0x4d/0x100 [ 3304.244197] free_work+0x34/0x50 [libcfs] [ 3304.246519] process_one_work+0x2c8/0x700 [ 3304.248668] worker_thread+0x296/0x6e0 [ 3304.250958] ? rescuer_thread+0x570/0x570 [ 3304.253231] kthread+0x1d1/0x200 [ 3304.255535] ? set_kthread_struct+0x70/0x70 [ 3304.257941] ret_from_fork+0x1f/0x30 [ 3305.439028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 947s! [ 3305.442516] Showing busy workqueues and worker pools: [ 3305.449129] workqueue events: flags=0x0 [ 3305.453665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3305.458494] in-flight: 56293:free_work [libcfs] [ 3305.461021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3305.464257] in-flight: 22277:free_work [libcfs] [ 3305.466464] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3305.474075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3305.476351] pending: nfs4_renew_state [nfsv4] [ 3305.477770] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3305.487190] in-flight: 47596:free_work [libcfs] [ 3305.489550] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3305.493521] workqueue events_unbound: flags=0x2 [ 3305.495424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3305.498065] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3305.501609] workqueue events_power_efficient: flags=0x80 [ 3305.503751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3305.507270] pending: neigh_periodic_work, do_cache_clean [ 3305.511871] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3305.513724] pending: check_lifetime [ 3305.518915] workqueue mm_percpu_wq: flags=0x8 [ 3305.520364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3305.522572] pending: vmstat_update [ 3305.523662] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3305.526231] pending: vmstat_update [ 3305.527347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3305.530179] pending: vmstat_update [ 3305.531377] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3305.533543] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3305.537926] pending: cgroup_pidlist_destroy_work_fn [ 3305.542707] workqueue writeback: flags=0x4a [ 3305.546404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3305.549922] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3305.554094] workqueue xprtiod: flags=0xa [ 3305.556240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3305.562933] pending: xprt_autoclose [ 3305.565314] workqueue ipv6_addrconf: flags=0x40008 [ 3305.567234] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3305.570491] pending: addrconf_verify_work [ 3305.571937] workqueue knbd0-recv: flags=0x1a [ 3305.573240] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3305.576755] in-flight: 655:recv_work [ 3305.578297] workqueue ptlrpc_pinger: flags=0x2 [ 3305.583329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3305.589702] pending: ptlrpc_pinger_main [ptlrpc] [ 3305.593052] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=947s workers=5 idle: 53684 27131 54850 46584 [ 3305.596800] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=947s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3305.612734] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=947s workers=6 idle: 53660 37674 46417 53664 46418 [ 3305.621732] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3295s workers=2 idle: 657 [ 3308.058028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3308.063640] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3308.104185] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3308.118108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3308.127670] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3308.135416] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3308.152707] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3308.157028] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3308.165342] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3308.171126] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3308.175833] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3308.183381] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3308.189609] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3308.194028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3308.196145] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3308.202230] Call Trace: [ 3308.205490] [ 3308.206365] ? show_regs.cold.9+0x22/0x2f [ 3308.207871] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3308.212588] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3308.216354] ? watchdog+0x40/0x40 [ 3308.221952] ? hrtimer_interrupt+0x128/0x2c0 [ 3308.223613] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3308.226620] ? apic_timer_interrupt+0xf/0x20 [ 3308.229333] [ 3308.230092] ? on_each_cpu_cond_mask+0x46/0x150 [ 3308.232977] ? __x64_sys_fspick+0x210/0x210 [ 3308.236167] ? smp_call_function_many_cond+0x3d9/0x540 [ 3308.245686] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3308.248075] ? __x64_sys_fspick+0x210/0x210 [ 3308.251626] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3308.253917] on_each_cpu_cond_mask+0x46/0x150 [ 3308.257301] on_each_cpu_cond+0x1c/0x30 [ 3308.259444] invalidate_bh_lrus+0x27/0x30 [ 3308.261377] invalidate_bdev+0x5a/0x90 [ 3308.264023] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3308.266781] generic_shutdown_super+0xb1/0x1b0 [ 3308.268860] kill_block_super+0x29/0x80 [ 3308.272986] deactivate_locked_super+0x52/0xd0 [ 3308.275141] deactivate_super+0x83/0x90 [ 3308.276885] cleanup_mnt+0x5f/0xe0 [ 3308.278605] __cleanup_mnt+0x16/0x20 [ 3308.280794] task_work_run+0xc6/0x110 [ 3308.282660] exit_to_usermode_loop+0x1dd/0x1f0 [ 3308.289675] do_syscall_64+0x3d6/0x440 [ 3308.294423] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3308.299044] RIP: 0033:0x7fbb019648fb [ 3308.303964] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3308.314293] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3308.317310] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3308.319829] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3308.326452] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3308.330410] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3308.336274] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3317.758376] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 3317.761850] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 3317.764955] (detected by 2, t=960028 jiffies, g=626849, q=1398) [ 3317.766938] Sending NMI from CPU 2 to CPUs 3: [ 3317.769412] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 3317.769420] rcu: rcu_sched kthread starved for 959670 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3317.775442] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3317.778990] rcu: RCU grace-period kthread stack dump: [ 3317.780812] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3317.784177] Call Trace: [ 3317.785047] __schedule+0x351/0xcb0 [ 3317.787760] schedule+0xc0/0x180 [ 3317.788855] schedule_timeout+0xb4/0x190 [ 3317.790017] ? do_raw_spin_unlock+0x75/0x190 [ 3317.791386] ? __next_timer_interrupt+0x160/0x160 [ 3317.793691] ? prepare_to_swait_event+0xf3/0x140 [ 3317.797604] rcu_gp_kthread+0x8dd/0x15b0 [ 3317.800345] ? force_qs_rnp+0x2c0/0x2c0 [ 3317.802176] kthread+0x1d1/0x200 [ 3317.803166] ? set_kthread_struct+0x70/0x70 [ 3317.806273] ret_from_fork+0x1f/0x30 [ 3317.809039] rcu: Stack dump where RCU GP kthread last ran: [ 3317.812343] NMI backtrace for cpu 2 [ 3317.817195] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3317.822915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3317.828039] Workqueue: events free_work [libcfs] [ 3317.836421] Call Trace: [ 3317.838186] [ 3317.838985] ? dump_stack+0xbb/0x10e [ 3317.841273] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3317.847615] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3317.850363] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3317.853280] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3317.855699] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3317.858274] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3317.867199] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 3317.872391] ? update_process_times+0x6d/0xc0 [ 3317.876894] ? tick_sched_handle.isra.18+0x27/0x90 [ 3317.879029] ? tick_sched_timer+0x53/0xb0 [ 3317.880291] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3317.882230] ? tick_sched_handle.isra.18+0x90/0x90 [ 3317.886416] ? hrtimer_interrupt+0x128/0x2c0 [ 3317.888370] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3317.891144] ? apic_timer_interrupt+0xf/0x20 [ 3317.893311] [ 3317.894376] ? on_each_cpu+0x51/0x120 [ 3317.901236] ? smp_call_function_many_cond+0x3d9/0x540 [ 3317.907401] ? smp_call_function_many_cond+0x3a8/0x540 [ 3317.909211] ? load_new_mm_cr3+0x120/0x120 [ 3317.913360] ? load_new_mm_cr3+0x120/0x120 [ 3317.917513] ? on_each_cpu+0x51/0x120 [ 3317.919080] ? flush_tlb_kernel_range+0x93/0xb0 [ 3317.929428] ? free_unmap_vmap_area+0x63/0x70 [ 3317.931524] ? remove_vm_area+0xe7/0x110 [ 3317.933293] ? __vunmap+0x9c/0x400 [ 3317.936211] ? vfree+0x4d/0x100 [ 3317.938162] ? free_work+0x34/0x50 [libcfs] [ 3317.940423] ? process_one_work+0x2c8/0x700 [ 3317.943092] ? worker_thread+0x296/0x6e0 [ 3317.945360] ? rescuer_thread+0x570/0x570 [ 3317.950452] ? kthread+0x1d1/0x200 [ 3317.952096] ? set_kthread_struct+0x70/0x70 [ 3317.957960] ? ret_from_fork+0x1f/0x30 [ 3332.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3332.064060] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3332.109017] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3332.119570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3332.126494] Workqueue: events free_work [libcfs] [ 3332.129882] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3332.133615] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3332.148033] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3332.153660] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3332.167836] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3332.171561] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3332.181049] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3332.187110] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3332.190768] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3332.197548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3332.202014] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3332.206344] Call Trace: [ 3332.207740] [ 3332.208572] ? show_regs.cold.9+0x22/0x2f [ 3332.211079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3332.212371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3332.216672] ? watchdog+0x40/0x40 [ 3332.218388] ? hrtimer_interrupt+0x128/0x2c0 [ 3332.219791] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3332.221978] ? apic_timer_interrupt+0xf/0x20 [ 3332.223781] [ 3332.224819] ? on_each_cpu+0x51/0x120 [ 3332.226075] ? smp_call_function_many_cond+0x3db/0x540 [ 3332.228461] ? smp_call_function_many_cond+0x3a8/0x540 [ 3332.230511] ? load_new_mm_cr3+0x120/0x120 [ 3332.232386] ? load_new_mm_cr3+0x120/0x120 [ 3332.234516] on_each_cpu+0x51/0x120 [ 3332.236637] flush_tlb_kernel_range+0x93/0xb0 [ 3332.239022] free_unmap_vmap_area+0x63/0x70 [ 3332.241280] remove_vm_area+0xe7/0x110 [ 3332.243872] __vunmap+0x9c/0x400 [ 3332.245684] vfree+0x4d/0x100 [ 3332.247439] free_work+0x34/0x50 [libcfs] [ 3332.248743] process_one_work+0x2c8/0x700 [ 3332.252695] worker_thread+0x296/0x6e0 [ 3332.255775] ? rescuer_thread+0x570/0x570 [ 3332.259056] kthread+0x1d1/0x200 [ 3332.260731] ? set_kthread_struct+0x70/0x70 [ 3332.263144] ret_from_fork+0x1f/0x30 [ 3335.648113] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 977s! [ 3335.652137] Showing busy workqueues and worker pools: [ 3335.654047] workqueue events: flags=0x0 [ 3335.656583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3335.664438] in-flight: 56293:free_work [libcfs] [ 3335.670301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3335.677130] in-flight: 22277:free_work [libcfs] [ 3335.681274] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3335.687775] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3335.692026] pending: nfs4_renew_state [nfsv4] [ 3335.694734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3335.697338] in-flight: 47596:free_work [libcfs] [ 3335.699459] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3335.703702] workqueue events_unbound: flags=0x2 [ 3335.706620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3335.709770] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3335.714045] workqueue events_power_efficient: flags=0x80 [ 3335.716674] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3335.719696] pending: neigh_periodic_work, do_cache_clean [ 3335.722439] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3335.725480] pending: check_lifetime [ 3335.727366] workqueue mm_percpu_wq: flags=0x8 [ 3335.729405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3335.733097] pending: vmstat_update [ 3335.734973] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3335.738072] pending: vmstat_update [ 3335.739792] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3335.743010] pending: vmstat_update [ 3335.744713] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3335.749546] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3335.754010] pending: cgroup_pidlist_destroy_work_fn [ 3335.761014] workqueue writeback: flags=0x4a [ 3335.767289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3335.773378] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3335.779776] workqueue xprtiod: flags=0xa [ 3335.784800] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3335.790471] pending: xprt_autoclose [ 3335.795802] workqueue ipv6_addrconf: flags=0x40008 [ 3335.801996] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3335.808084] pending: addrconf_verify_work [ 3335.811502] workqueue knbd0-recv: flags=0x1a [ 3335.814226] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3335.817834] in-flight: 655:recv_work [ 3335.819752] workqueue ptlrpc_pinger: flags=0x2 [ 3335.821845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3335.824368] pending: ptlrpc_pinger_main [ptlrpc] [ 3335.827253] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=978s workers=5 idle: 53684 27131 54850 46584 [ 3335.831093] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=978s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3335.836336] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=978s workers=6 idle: 53660 37674 46417 53664 46418 [ 3335.841256] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3325s workers=2 idle: 657 [ 3336.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3336.061825] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3336.096315] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3336.106470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3336.112753] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3336.123413] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3336.135735] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3336.139300] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3336.149678] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3336.155918] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3336.159235] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3336.162572] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3336.165232] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3336.172792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3336.174869] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3336.183077] Call Trace: [ 3336.187583] [ 3336.188743] ? show_regs.cold.9+0x22/0x2f [ 3336.190068] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3336.192396] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3336.194186] ? watchdog+0x40/0x40 [ 3336.195787] ? hrtimer_interrupt+0x128/0x2c0 [ 3336.198748] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3336.200816] ? apic_timer_interrupt+0xf/0x20 [ 3336.203922] [ 3336.206009] ? on_each_cpu_cond_mask+0x46/0x150 [ 3336.210682] ? __x64_sys_fspick+0x210/0x210 [ 3336.215418] ? smp_call_function_many_cond+0x3db/0x540 [ 3336.224213] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3336.226205] ? __x64_sys_fspick+0x210/0x210 [ 3336.228335] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3336.231108] on_each_cpu_cond_mask+0x46/0x150 [ 3336.236432] on_each_cpu_cond+0x1c/0x30 [ 3336.238788] invalidate_bh_lrus+0x27/0x30 [ 3336.242266] invalidate_bdev+0x5a/0x90 [ 3336.247235] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3336.249603] generic_shutdown_super+0xb1/0x1b0 [ 3336.257181] kill_block_super+0x29/0x80 [ 3336.260771] deactivate_locked_super+0x52/0xd0 [ 3336.262694] deactivate_super+0x83/0x90 [ 3336.267979] cleanup_mnt+0x5f/0xe0 [ 3336.270865] __cleanup_mnt+0x16/0x20 [ 3336.276653] task_work_run+0xc6/0x110 [ 3336.281111] exit_to_usermode_loop+0x1dd/0x1f0 [ 3336.291078] do_syscall_64+0x3d6/0x440 [ 3336.301541] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3336.306460] RIP: 0033:0x7fbb019648fb [ 3336.317971] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3336.337520] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3336.348059] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3336.355861] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3336.360941] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3336.365598] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3336.369466] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3344.059014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3344.072434] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3344.108159] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3344.116422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3344.120763] Workqueue: events free_work [libcfs] [ 3344.123015] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3344.125745] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3344.138057] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3344.142555] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3344.146910] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3344.151428] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3344.155833] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3344.160602] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3344.165105] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3344.173625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3344.178623] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3344.184243] Call Trace: [ 3344.185869] [ 3344.187212] ? show_regs.cold.9+0x22/0x2f [ 3344.189896] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3344.193394] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3344.198560] ? watchdog+0x40/0x40 [ 3344.202398] ? hrtimer_interrupt+0x128/0x2c0 [ 3344.204461] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3344.207518] ? apic_timer_interrupt+0xf/0x20 [ 3344.210900] [ 3344.212175] ? on_each_cpu+0x51/0x120 [ 3344.213905] ? smp_call_function_many_cond+0x3db/0x540 [ 3344.217007] ? smp_call_function_many_cond+0x3a8/0x540 [ 3344.220227] ? load_new_mm_cr3+0x120/0x120 [ 3344.222598] ? load_new_mm_cr3+0x120/0x120 [ 3344.225949] on_each_cpu+0x51/0x120 [ 3344.228445] flush_tlb_kernel_range+0x93/0xb0 [ 3344.232997] free_unmap_vmap_area+0x63/0x70 [ 3344.234706] remove_vm_area+0xe7/0x110 [ 3344.244219] __vunmap+0x9c/0x400 [ 3344.246527] vfree+0x4d/0x100 [ 3344.249624] free_work+0x34/0x50 [libcfs] [ 3344.254121] process_one_work+0x2c8/0x700 [ 3344.256664] worker_thread+0x296/0x6e0 [ 3344.259069] ? rescuer_thread+0x570/0x570 [ 3344.263618] kthread+0x1d1/0x200 [ 3344.268150] ? set_kthread_struct+0x70/0x70 [ 3344.271329] ret_from_fork+0x1f/0x30 [ 3360.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3360.060342] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3360.086497] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3360.093117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3360.097502] Workqueue: events free_work [libcfs] [ 3360.100997] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3360.104146] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3360.114521] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3360.119050] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3360.122657] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3360.125930] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3360.128723] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3360.131234] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3360.133524] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3360.136152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3360.137876] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3360.139897] Call Trace: [ 3360.140684] [ 3360.141303] ? show_regs.cold.9+0x22/0x2f [ 3360.142502] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3360.144300] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3360.145545] ? watchdog+0x40/0x40 [ 3360.146547] ? hrtimer_interrupt+0x128/0x2c0 [ 3360.147945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3360.149234] ? apic_timer_interrupt+0xf/0x20 [ 3360.150393] [ 3360.151270] ? on_each_cpu+0x51/0x120 [ 3360.152387] ? smp_call_function_many_cond+0x3db/0x540 [ 3360.153867] ? smp_call_function_many_cond+0x3a8/0x540 [ 3360.155225] ? load_new_mm_cr3+0x120/0x120 [ 3360.156712] ? load_new_mm_cr3+0x120/0x120 [ 3360.157915] on_each_cpu+0x51/0x120 [ 3360.158771] flush_tlb_kernel_range+0x93/0xb0 [ 3360.161624] free_unmap_vmap_area+0x63/0x70 [ 3360.163831] remove_vm_area+0xe7/0x110 [ 3360.164881] __vunmap+0x9c/0x400 [ 3360.170076] vfree+0x4d/0x100 [ 3360.172926] free_work+0x34/0x50 [libcfs] [ 3360.176372] process_one_work+0x2c8/0x700 [ 3360.178124] worker_thread+0x296/0x6e0 [ 3360.181246] ? rescuer_thread+0x570/0x570 [ 3360.186507] kthread+0x1d1/0x200 [ 3360.188461] ? set_kthread_struct+0x70/0x70 [ 3360.192875] ret_from_fork+0x1f/0x30 [ 3364.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3364.061992] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3364.103344] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3364.109135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3364.116869] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3364.122891] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3364.132464] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3364.134555] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3364.137143] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3364.142234] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3364.149605] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3364.154170] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3364.157347] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3364.161012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3364.163575] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3364.169348] Call Trace: [ 3364.171331] [ 3364.173201] ? show_regs.cold.9+0x22/0x2f [ 3364.176237] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3364.180281] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3364.183771] ? watchdog+0x40/0x40 [ 3364.185391] ? hrtimer_interrupt+0x128/0x2c0 [ 3364.189965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3364.192776] ? apic_timer_interrupt+0xf/0x20 [ 3364.195381] [ 3364.196594] ? on_each_cpu_cond_mask+0x46/0x150 [ 3364.199097] ? __x64_sys_fspick+0x210/0x210 [ 3364.200629] ? smp_call_function_many_cond+0x3d9/0x540 [ 3364.203172] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3364.205752] ? __x64_sys_fspick+0x210/0x210 [ 3364.208338] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3364.211028] on_each_cpu_cond_mask+0x46/0x150 [ 3364.213417] on_each_cpu_cond+0x1c/0x30 [ 3364.215935] invalidate_bh_lrus+0x27/0x30 [ 3364.217991] invalidate_bdev+0x5a/0x90 [ 3364.219806] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3364.222484] generic_shutdown_super+0xb1/0x1b0 [ 3364.224619] kill_block_super+0x29/0x80 [ 3364.226616] deactivate_locked_super+0x52/0xd0 [ 3364.229059] deactivate_super+0x83/0x90 [ 3364.232867] cleanup_mnt+0x5f/0xe0 [ 3364.234980] __cleanup_mnt+0x16/0x20 [ 3364.237310] task_work_run+0xc6/0x110 [ 3364.239229] exit_to_usermode_loop+0x1dd/0x1f0 [ 3364.241567] do_syscall_64+0x3d6/0x440 [ 3364.243846] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3364.246257] RIP: 0033:0x7fbb019648fb [ 3364.248861] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3364.257882] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3364.261400] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3364.267262] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3364.271081] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3364.276032] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3364.282490] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3365.856027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1007s! [ 3365.861264] Showing busy workqueues and worker pools: [ 3365.863969] workqueue events: flags=0x0 [ 3365.865653] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3365.869614] in-flight: 56293:free_work [libcfs] [ 3365.872080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3365.875420] in-flight: 22277:free_work [libcfs] [ 3365.878048] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3365.882942] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3365.885722] pending: nfs4_renew_state [nfsv4] [ 3365.888350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3365.893392] in-flight: 47596:free_work [libcfs] [ 3365.898193] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3365.903837] workqueue events_unbound: flags=0x2 [ 3365.906956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3365.921514] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3365.926300] workqueue events_power_efficient: flags=0x80 [ 3365.930119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3365.935654] pending: neigh_periodic_work, do_cache_clean [ 3365.938684] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3365.942293] pending: check_lifetime [ 3365.943826] workqueue mm_percpu_wq: flags=0x8 [ 3365.945670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3365.949475] pending: vmstat_update [ 3365.951117] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3365.953510] pending: vmstat_update [ 3365.956610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3365.964062] pending: vmstat_update [ 3365.968739] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3365.971321] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3365.974775] pending: cgroup_pidlist_destroy_work_fn [ 3365.976953] workqueue writeback: flags=0x4a [ 3365.979018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3365.983067] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3365.987374] workqueue xprtiod: flags=0xa [ 3365.988844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3365.994643] pending: xprt_autoclose [ 3365.998804] workqueue ipv6_addrconf: flags=0x40008 [ 3366.004628] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3366.008579] pending: addrconf_verify_work [ 3366.010063] workqueue knbd0-recv: flags=0x1a [ 3366.012019] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3366.017142] in-flight: 655:recv_work [ 3366.021412] workqueue ptlrpc_pinger: flags=0x2 [ 3366.023938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3366.028370] pending: ptlrpc_pinger_main [ptlrpc] [ 3366.034437] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1008s workers=5 idle: 53684 27131 54850 46584 [ 3366.041905] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1008s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3366.046957] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1008s workers=6 idle: 53660 37674 46417 53664 46418 [ 3366.053864] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3355s workers=2 idle: 657 [ 3372.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3372.063396] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3372.103794] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3372.111463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3372.117176] Workqueue: events free_work [libcfs] [ 3372.120653] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3372.129951] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3372.138059] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3372.142442] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3372.146522] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3372.154463] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3372.159447] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3372.163689] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3372.169965] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3372.175518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3372.179397] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3372.184138] Call Trace: [ 3372.184879] [ 3372.185519] ? show_regs.cold.9+0x22/0x2f [ 3372.189188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3372.191766] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3372.196560] ? watchdog+0x40/0x40 [ 3372.198718] ? hrtimer_interrupt+0x128/0x2c0 [ 3372.203533] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3372.205450] ? apic_timer_interrupt+0xf/0x20 [ 3372.208857] [ 3372.209937] ? on_each_cpu+0x51/0x120 [ 3372.211877] ? smp_call_function_many_cond+0x3db/0x540 [ 3372.214714] ? smp_call_function_many_cond+0x3a8/0x540 [ 3372.217799] ? load_new_mm_cr3+0x120/0x120 [ 3372.220248] ? load_new_mm_cr3+0x120/0x120 [ 3372.222338] on_each_cpu+0x51/0x120 [ 3372.224603] flush_tlb_kernel_range+0x93/0xb0 [ 3372.226890] free_unmap_vmap_area+0x63/0x70 [ 3372.229300] remove_vm_area+0xe7/0x110 [ 3372.231306] __vunmap+0x9c/0x400 [ 3372.233091] vfree+0x4d/0x100 [ 3372.235580] free_work+0x34/0x50 [libcfs] [ 3372.238694] process_one_work+0x2c8/0x700 [ 3372.241295] worker_thread+0x296/0x6e0 [ 3372.247841] ? rescuer_thread+0x570/0x570 [ 3372.252008] kthread+0x1d1/0x200 [ 3372.254313] ? set_kthread_struct+0x70/0x70 [ 3372.260001] ret_from_fork+0x1f/0x30 [ 3392.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3392.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3392.067715] Modules linked in: zfs(O) [ 3392.072413] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3392.072512] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3392.072521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3392.072524] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3392.072536] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3392.072540] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3392.072545] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3392.072547] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3392.072549] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3392.072551] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3392.072553] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3392.072556] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3392.072559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3392.072561] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3392.072569] Call Trace: [ 3392.072574] [ 3392.072580] ? show_regs.cold.9+0x22/0x2f [ 3392.072600] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3392.072606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3392.072609] ? watchdog+0x40/0x40 [ 3392.072614] ? hrtimer_interrupt+0x128/0x2c0 [ 3392.072618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3392.072626] ? apic_timer_interrupt+0xf/0x20 [ 3392.072630] [ 3392.072632] ? on_each_cpu_cond_mask+0x46/0x150 [ 3392.072635] ? __x64_sys_fspick+0x210/0x210 [ 3392.072644] ? smp_call_function_many_cond+0x3db/0x540 [ 3392.072647] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3392.072651] ? __x64_sys_fspick+0x210/0x210 [ 3392.072655] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3392.072658] on_each_cpu_cond_mask+0x46/0x150 [ 3392.072662] on_each_cpu_cond+0x1c/0x30 [ 3392.072666] invalidate_bh_lrus+0x27/0x30 [ 3392.072669] invalidate_bdev+0x5a/0x90 [ 3392.072676] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3392.072797] generic_shutdown_super+0xb1/0x1b0 [ 3392.072805] kill_block_super+0x29/0x80 [ 3392.072811] deactivate_locked_super+0x52/0xd0 [ 3392.072815] deactivate_super+0x83/0x90 [ 3392.072821] cleanup_mnt+0x5f/0xe0 [ 3392.072826] __cleanup_mnt+0x16/0x20 [ 3392.072831] task_work_run+0xc6/0x110 [ 3392.072839] exit_to_usermode_loop+0x1dd/0x1f0 [ 3392.072845] do_syscall_64+0x3d6/0x440 [ 3392.072850] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3392.072855] RIP: 0033:0x7fbb019648fb [ 3392.072860] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3392.072863] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3392.072867] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3392.072869] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3392.072871] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3392.072873] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3392.072875] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3392.396749] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3392.439323] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3392.450818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3392.464415] Workqueue: events free_work [libcfs] [ 3392.467469] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3392.473005] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3392.484650] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3392.490161] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3392.495164] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3392.500072] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3392.503940] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3392.506617] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3392.518075] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3392.530865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3392.538693] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3392.543652] Call Trace: [ 3392.547067] [ 3392.548485] ? show_regs.cold.9+0x22/0x2f [ 3392.550765] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3392.555570] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3392.557255] ? watchdog+0x40/0x40 [ 3392.558611] ? hrtimer_interrupt+0x128/0x2c0 [ 3392.560540] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3392.562704] ? apic_timer_interrupt+0xf/0x20 [ 3392.564647] [ 3392.565596] ? on_each_cpu+0x51/0x120 [ 3392.567350] ? smp_call_function_many_cond+0x3db/0x540 [ 3392.569479] ? smp_call_function_many_cond+0x3a8/0x540 [ 3392.571772] ? load_new_mm_cr3+0x120/0x120 [ 3392.573287] ? load_new_mm_cr3+0x120/0x120 [ 3392.574441] on_each_cpu+0x51/0x120 [ 3392.575681] flush_tlb_kernel_range+0x93/0xb0 [ 3392.577454] free_unmap_vmap_area+0x63/0x70 [ 3392.579403] remove_vm_area+0xe7/0x110 [ 3392.581037] __vunmap+0x9c/0x400 [ 3392.582561] vfree+0x4d/0x100 [ 3392.583867] free_work+0x34/0x50 [libcfs] [ 3392.585789] process_one_work+0x2c8/0x700 [ 3392.587693] worker_thread+0x296/0x6e0 [ 3392.589412] ? rescuer_thread+0x570/0x570 [ 3392.591244] kthread+0x1d1/0x200 [ 3392.592763] ? set_kthread_struct+0x70/0x70 [ 3392.594562] ret_from_fork+0x1f/0x30 [ 3396.063456] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1037s! [ 3396.069516] Showing busy workqueues and worker pools: [ 3396.072199] workqueue events: flags=0x0 [ 3396.075210] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3396.080530] in-flight: 56293:free_work [libcfs] [ 3396.084041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3396.088035] in-flight: 22277:free_work [libcfs] [ 3396.091137] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3396.095958] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3396.102944] pending: nfs4_renew_state [nfsv4] [ 3396.109762] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3396.114502] in-flight: 47596:free_work [libcfs] [ 3396.118191] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3396.129629] workqueue events_unbound: flags=0x2 [ 3396.132418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3396.135814] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3396.143536] workqueue events_power_efficient: flags=0x80 [ 3396.156126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3396.158942] pending: neigh_periodic_work, do_cache_clean [ 3396.167356] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3396.172645] pending: check_lifetime [ 3396.174944] workqueue mm_percpu_wq: flags=0x8 [ 3396.182718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3396.189297] pending: vmstat_update [ 3396.194249] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3396.202648] pending: vmstat_update [ 3396.205033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3396.212721] pending: vmstat_update [ 3396.214409] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3396.218289] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3396.222685] pending: cgroup_pidlist_destroy_work_fn [ 3396.225973] workqueue writeback: flags=0x4a [ 3396.228742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3396.231614] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3396.241261] workqueue xprtiod: flags=0xa [ 3396.244319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3396.247476] pending: xprt_autoclose [ 3396.254200] workqueue ipv6_addrconf: flags=0x40008 [ 3396.257230] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3396.260345] pending: addrconf_verify_work [ 3396.262887] workqueue knbd0-recv: flags=0x1a [ 3396.265049] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3396.267488] in-flight: 655:recv_work [ 3396.275640] workqueue ptlrpc_pinger: flags=0x2 [ 3396.281774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3396.287193] pending: ptlrpc_pinger_main [ptlrpc] [ 3396.295621] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1038s workers=5 idle: 53684 27131 54850 46584 [ 3396.307244] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1038s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3396.317766] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1038s workers=6 idle: 53660 37674 46417 53664 46418 [ 3396.326279] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3386s workers=2 idle: 657 [ 3400.060171] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3400.070612] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3400.112069] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3400.122354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3400.141421] Workqueue: events free_work [libcfs] [ 3400.146145] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3400.155353] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3400.166310] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3400.173325] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3400.178252] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3400.181679] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3400.187275] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3400.192092] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3400.199266] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3400.207347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3400.211406] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3400.217466] Call Trace: [ 3400.218642] [ 3400.219649] ? show_regs.cold.9+0x22/0x2f [ 3400.221567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3400.224129] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3400.228515] ? watchdog+0x40/0x40 [ 3400.229757] ? hrtimer_interrupt+0x128/0x2c0 [ 3400.243011] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3400.245704] ? apic_timer_interrupt+0xf/0x20 [ 3400.249366] [ 3400.250562] ? on_each_cpu+0x51/0x120 [ 3400.256143] ? smp_call_function_many_cond+0x3db/0x540 [ 3400.259160] ? smp_call_function_many_cond+0x3a8/0x540 [ 3400.263451] ? load_new_mm_cr3+0x120/0x120 [ 3400.268069] ? load_new_mm_cr3+0x120/0x120 [ 3400.270673] on_each_cpu+0x51/0x120 [ 3400.275775] flush_tlb_kernel_range+0x93/0xb0 [ 3400.278191] free_unmap_vmap_area+0x63/0x70 [ 3400.282366] remove_vm_area+0xe7/0x110 [ 3400.284624] __vunmap+0x9c/0x400 [ 3400.288012] vfree+0x4d/0x100 [ 3400.289921] free_work+0x34/0x50 [libcfs] [ 3400.297683] process_one_work+0x2c8/0x700 [ 3400.301572] worker_thread+0x296/0x6e0 [ 3400.304233] ? rescuer_thread+0x570/0x570 [ 3400.307908] kthread+0x1d1/0x200 [ 3400.309239] ? set_kthread_struct+0x70/0x70 [ 3400.310907] ret_from_fork+0x1f/0x30 [ 3420.058291] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3420.068756] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3420.098713] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3420.105923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3420.111857] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3420.114735] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3420.125817] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3420.132570] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3420.138566] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3420.141891] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3420.148012] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3420.154962] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3420.160408] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3420.168444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3420.176228] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3420.181108] Call Trace: [ 3420.185802] [ 3420.188660] ? show_regs.cold.9+0x22/0x2f [ 3420.192147] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3420.196782] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3420.205019] ? watchdog+0x40/0x40 [ 3420.206226] ? hrtimer_interrupt+0x128/0x2c0 [ 3420.214478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3420.216290] ? apic_timer_interrupt+0xf/0x20 [ 3420.227930] [ 3420.231447] ? on_each_cpu_cond_mask+0x46/0x150 [ 3420.234711] ? __x64_sys_fspick+0x210/0x210 [ 3420.236314] ? smp_call_function_many_cond+0x3db/0x540 [ 3420.240143] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3420.245692] ? __x64_sys_fspick+0x210/0x210 [ 3420.247976] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3420.250049] on_each_cpu_cond_mask+0x46/0x150 [ 3420.253516] on_each_cpu_cond+0x1c/0x30 [ 3420.255801] invalidate_bh_lrus+0x27/0x30 [ 3420.258574] invalidate_bdev+0x5a/0x90 [ 3420.260809] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3420.265858] generic_shutdown_super+0xb1/0x1b0 [ 3420.274291] kill_block_super+0x29/0x80 [ 3420.277554] deactivate_locked_super+0x52/0xd0 [ 3420.282815] deactivate_super+0x83/0x90 [ 3420.286552] cleanup_mnt+0x5f/0xe0 [ 3420.289978] __cleanup_mnt+0x16/0x20 [ 3420.291237] task_work_run+0xc6/0x110 [ 3420.293029] exit_to_usermode_loop+0x1dd/0x1f0 [ 3420.297860] do_syscall_64+0x3d6/0x440 [ 3420.300974] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3420.304894] RIP: 0033:0x7fbb019648fb [ 3420.306597] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3420.320864] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3420.325765] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3420.329715] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3420.334627] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3420.339655] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3420.344791] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3424.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3424.061659] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3424.128223] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3424.139441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3424.151582] Workqueue: events free_work [libcfs] [ 3424.156256] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3424.158500] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3424.176491] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3424.180112] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3424.191596] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3424.193517] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3424.201721] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3424.206946] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3424.216823] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3424.222760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3424.227775] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3424.237647] Call Trace: [ 3424.238833] [ 3424.240697] ? show_regs.cold.9+0x22/0x2f [ 3424.246050] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3424.250341] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3424.253758] ? watchdog+0x40/0x40 [ 3424.256671] ? hrtimer_interrupt+0x128/0x2c0 [ 3424.258944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3424.262619] ? apic_timer_interrupt+0xf/0x20 [ 3424.265853] [ 3424.267696] ? on_each_cpu+0x51/0x120 [ 3424.275783] ? smp_call_function_many_cond+0x3db/0x540 [ 3424.279676] ? smp_call_function_many_cond+0x3a8/0x540 [ 3424.285155] ? load_new_mm_cr3+0x120/0x120 [ 3424.295196] ? load_new_mm_cr3+0x120/0x120 [ 3424.301884] on_each_cpu+0x51/0x120 [ 3424.312086] flush_tlb_kernel_range+0x93/0xb0 [ 3424.316098] free_unmap_vmap_area+0x63/0x70 [ 3424.335627] remove_vm_area+0xe7/0x110 [ 3424.337019] __vunmap+0x9c/0x400 [ 3424.338299] vfree+0x4d/0x100 [ 3424.346117] free_work+0x34/0x50 [libcfs] [ 3424.349004] process_one_work+0x2c8/0x700 [ 3424.351720] worker_thread+0x296/0x6e0 [ 3424.353932] ? rescuer_thread+0x570/0x570 [ 3424.356130] kthread+0x1d1/0x200 [ 3424.358156] ? set_kthread_struct+0x70/0x70 [ 3424.365568] ret_from_fork+0x1f/0x30 [ 3426.784706] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1068s! [ 3426.796121] Showing busy workqueues and worker pools: [ 3426.804902] workqueue events: flags=0x0 [ 3426.807852] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3426.814566] in-flight: 56293:free_work [libcfs] [ 3426.821352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3426.831040] in-flight: 22277:free_work [libcfs] [ 3426.846473] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3426.856900] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3426.862170] pending: nfs4_renew_state [nfsv4] [ 3426.869994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3426.878611] in-flight: 47596:free_work [libcfs] [ 3426.886630] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3426.900416] workqueue events_unbound: flags=0x2 [ 3426.903034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3426.909217] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3426.916493] workqueue events_power_efficient: flags=0x80 [ 3426.920636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3426.928573] pending: neigh_periodic_work, do_cache_clean [ 3426.937429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3426.949374] pending: check_lifetime [ 3426.955026] workqueue mm_percpu_wq: flags=0x8 [ 3426.963339] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3426.968378] pending: vmstat_update [ 3426.972901] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3426.979409] pending: vmstat_update [ 3426.982449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3426.988787] pending: vmstat_update [ 3426.990038] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3426.996050] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3427.001864] pending: cgroup_pidlist_destroy_work_fn [ 3427.003202] workqueue writeback: flags=0x4a [ 3427.005502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3427.009573] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3427.012898] workqueue xprtiod: flags=0xa [ 3427.014663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3427.017606] pending: xprt_autoclose [ 3427.019720] workqueue ipv6_addrconf: flags=0x40008 [ 3427.025847] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3427.029592] pending: addrconf_verify_work [ 3427.032735] workqueue knbd0-recv: flags=0x1a [ 3427.036225] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3427.045353] in-flight: 655:recv_work [ 3427.047014] workqueue ptlrpc_pinger: flags=0x2 [ 3427.052025] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3427.072401] pending: ptlrpc_pinger_main [ptlrpc] [ 3427.086908] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1069s workers=5 idle: 53684 27131 54850 46584 [ 3427.093274] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1069s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3427.111961] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1069s workers=6 idle: 53660 37674 46417 53664 46418 [ 3427.127556] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3416s workers=2 idle: 657 [ 3428.064053] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3428.078196] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3428.153993] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3428.176742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3428.192760] Workqueue: events free_work [libcfs] [ 3428.203778] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3428.211507] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3428.224586] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3428.228790] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3428.233005] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3428.238345] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3428.245520] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3428.255480] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3428.261063] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3428.267432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3428.271201] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3428.275350] Call Trace: [ 3428.276551] [ 3428.277572] ? show_regs.cold.9+0x22/0x2f [ 3428.279815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3428.282493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3428.286564] ? watchdog+0x40/0x40 [ 3428.289392] ? hrtimer_interrupt+0x128/0x2c0 [ 3428.291714] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3428.300788] ? apic_timer_interrupt+0xf/0x20 [ 3428.303291] [ 3428.304377] ? on_each_cpu+0x51/0x120 [ 3428.307088] ? smp_call_function_many_cond+0x3db/0x540 [ 3428.311656] ? smp_call_function_many_cond+0x3a8/0x540 [ 3428.317230] ? load_new_mm_cr3+0x120/0x120 [ 3428.319695] ? load_new_mm_cr3+0x120/0x120 [ 3428.322711] on_each_cpu+0x51/0x120 [ 3428.327520] flush_tlb_kernel_range+0x93/0xb0 [ 3428.330423] free_unmap_vmap_area+0x63/0x70 [ 3428.335676] remove_vm_area+0xe7/0x110 [ 3428.337361] __vunmap+0x9c/0x400 [ 3428.339570] vfree+0x4d/0x100 [ 3428.346691] free_work+0x34/0x50 [libcfs] [ 3428.349606] process_one_work+0x2c8/0x700 [ 3428.355803] worker_thread+0x296/0x6e0 [ 3428.359579] ? rescuer_thread+0x570/0x570 [ 3428.361122] kthread+0x1d1/0x200 [ 3428.367730] ? set_kthread_struct+0x70/0x70 [ 3428.369358] ret_from_fork+0x1f/0x30 [ 3448.058922] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3448.069873] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3448.104998] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3448.113882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3448.124991] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3448.126405] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3448.144864] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3448.149178] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3448.153655] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3448.158391] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3448.161176] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3448.166017] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3448.169983] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3448.173895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3448.176812] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3448.180165] Call Trace: [ 3448.181397] [ 3448.182416] ? show_regs.cold.9+0x22/0x2f [ 3448.184267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3448.186777] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3448.189174] ? watchdog+0x40/0x40 [ 3448.190862] ? hrtimer_interrupt+0x128/0x2c0 [ 3448.193360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3448.197849] ? apic_timer_interrupt+0xf/0x20 [ 3448.199724] [ 3448.200699] ? on_each_cpu_cond_mask+0x46/0x150 [ 3448.203920] ? __x64_sys_fspick+0x210/0x210 [ 3448.207078] ? smp_call_function_many_cond+0x3d9/0x540 [ 3448.211522] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3448.216272] ? __x64_sys_fspick+0x210/0x210 [ 3448.217843] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3448.223608] on_each_cpu_cond_mask+0x46/0x150 [ 3448.230079] on_each_cpu_cond+0x1c/0x30 [ 3448.235747] invalidate_bh_lrus+0x27/0x30 [ 3448.239058] invalidate_bdev+0x5a/0x90 [ 3448.244002] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3448.248528] generic_shutdown_super+0xb1/0x1b0 [ 3448.252945] kill_block_super+0x29/0x80 [ 3448.254841] deactivate_locked_super+0x52/0xd0 [ 3448.257348] deactivate_super+0x83/0x90 [ 3448.259834] cleanup_mnt+0x5f/0xe0 [ 3448.261533] __cleanup_mnt+0x16/0x20 [ 3448.265036] task_work_run+0xc6/0x110 [ 3448.268327] exit_to_usermode_loop+0x1dd/0x1f0 [ 3448.270131] do_syscall_64+0x3d6/0x440 [ 3448.273047] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3448.275798] RIP: 0033:0x7fbb019648fb [ 3448.278023] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3448.286579] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3448.290662] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3448.292647] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3448.295403] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3448.300247] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3448.304433] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3452.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3452.062863] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3452.099634] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3452.106613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3452.111846] Workqueue: events free_work [libcfs] [ 3452.116712] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3452.119535] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3452.133456] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3452.138929] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3452.145047] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3452.151541] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3452.155025] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3452.161907] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3452.165956] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3452.169063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3452.171161] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3452.174920] Call Trace: [ 3452.178039] [ 3452.179115] ? show_regs.cold.9+0x22/0x2f [ 3452.180909] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3452.187215] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3452.192803] ? watchdog+0x40/0x40 [ 3452.195258] ? hrtimer_interrupt+0x128/0x2c0 [ 3452.196965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3452.199270] ? apic_timer_interrupt+0xf/0x20 [ 3452.205061] [ 3452.208942] ? on_each_cpu+0x51/0x120 [ 3452.210955] ? smp_call_function_many_cond+0x3db/0x540 [ 3452.218840] ? smp_call_function_many_cond+0x3a8/0x540 [ 3452.223491] ? load_new_mm_cr3+0x120/0x120 [ 3452.230047] ? load_new_mm_cr3+0x120/0x120 [ 3452.235016] on_each_cpu+0x51/0x120 [ 3452.240408] flush_tlb_kernel_range+0x93/0xb0 [ 3452.241884] free_unmap_vmap_area+0x63/0x70 [ 3452.243142] remove_vm_area+0xe7/0x110 [ 3452.247147] __vunmap+0x9c/0x400 [ 3452.250952] vfree+0x4d/0x100 [ 3452.253905] free_work+0x34/0x50 [libcfs] [ 3452.259122] process_one_work+0x2c8/0x700 [ 3452.265076] worker_thread+0x296/0x6e0 [ 3452.267970] ? rescuer_thread+0x570/0x570 [ 3452.270923] kthread+0x1d1/0x200 [ 3452.272572] ? set_kthread_struct+0x70/0x70 [ 3452.274812] ret_from_fork+0x1f/0x30 [ 3456.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3456.064371] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3456.091060] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3456.104394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3456.112477] Workqueue: events free_work [libcfs] [ 3456.116582] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3456.119840] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3456.129700] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3456.131822] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3456.137764] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3456.141083] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3456.143829] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3456.146254] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3456.151858] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3456.159187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3456.163232] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3456.172494] Call Trace: [ 3456.173166] [ 3456.173996] ? show_regs.cold.9+0x22/0x2f [ 3456.175389] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3456.177798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3456.179841] ? watchdog+0x40/0x40 [ 3456.182273] ? hrtimer_interrupt+0x128/0x2c0 [ 3456.185431] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3456.187785] ? apic_timer_interrupt+0xf/0x20 [ 3456.189980] [ 3456.190928] ? on_each_cpu+0x51/0x120 [ 3456.194159] ? smp_call_function_many_cond+0x3d9/0x540 [ 3456.197674] ? smp_call_function_many_cond+0x3a8/0x540 [ 3456.200063] ? load_new_mm_cr3+0x120/0x120 [ 3456.203170] ? load_new_mm_cr3+0x120/0x120 [ 3456.206641] on_each_cpu+0x51/0x120 [ 3456.209077] flush_tlb_kernel_range+0x93/0xb0 [ 3456.212402] free_unmap_vmap_area+0x63/0x70 [ 3456.214038] remove_vm_area+0xe7/0x110 [ 3456.216266] __vunmap+0x9c/0x400 [ 3456.220760] vfree+0x4d/0x100 [ 3456.222468] free_work+0x34/0x50 [libcfs] [ 3456.224698] process_one_work+0x2c8/0x700 [ 3456.226850] worker_thread+0x296/0x6e0 [ 3456.230009] ? rescuer_thread+0x570/0x570 [ 3456.231608] kthread+0x1d1/0x200 [ 3456.246592] ? set_kthread_struct+0x70/0x70 [ 3456.253296] ret_from_fork+0x1f/0x30 [ 3457.503032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1099s! [ 3457.506992] Showing busy workqueues and worker pools: [ 3457.510081] workqueue events: flags=0x0 [ 3457.522049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3457.534155] in-flight: 56293:free_work [libcfs] [ 3457.536239] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3457.552332] in-flight: 22277:free_work [libcfs] [ 3457.562321] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3457.569268] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3457.572180] pending: nfs4_renew_state [nfsv4] [ 3457.574073] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3457.576817] in-flight: 47596:free_work [libcfs] [ 3457.578966] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3457.582303] workqueue events_unbound: flags=0x2 [ 3457.584056] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3457.586092] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3457.589366] workqueue events_power_efficient: flags=0x80 [ 3457.591107] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3457.593697] pending: neigh_periodic_work, do_cache_clean [ 3457.595421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3457.597726] pending: check_lifetime [ 3457.599282] workqueue mm_percpu_wq: flags=0x8 [ 3457.600462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3457.603258] pending: vmstat_update [ 3457.604638] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3457.607103] pending: vmstat_update [ 3457.608329] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3457.611277] pending: vmstat_update [ 3457.612634] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3457.614680] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3457.616733] pending: cgroup_pidlist_destroy_work_fn [ 3457.618284] workqueue writeback: flags=0x4a [ 3457.620079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3457.621979] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3457.624973] workqueue xprtiod: flags=0xa [ 3457.626322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3457.628119] pending: xprt_autoclose [ 3457.629918] workqueue ipv6_addrconf: flags=0x40008 [ 3457.631961] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3457.635258] pending: addrconf_verify_work [ 3457.637841] workqueue knbd0-recv: flags=0x1a [ 3457.639458] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3457.642025] in-flight: 655:recv_work [ 3457.643404] workqueue ptlrpc_pinger: flags=0x2 [ 3457.644954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3457.647076] pending: ptlrpc_pinger_main [ptlrpc] [ 3457.649011] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1099s workers=5 idle: 53684 27131 54850 46584 [ 3457.652706] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1099s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3457.658023] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1099s workers=6 idle: 53660 37674 46417 53664 46418 [ 3457.662965] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3447s workers=2 idle: 657 [ 3476.059024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3476.064511] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3476.119014] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3476.133253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3476.151404] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3476.159919] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3476.188919] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3476.197614] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3476.203930] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3476.214474] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3476.221707] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3476.231073] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3476.242386] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3476.252798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3476.262942] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3476.265573] Call Trace: [ 3476.283440] [ 3476.284241] ? show_regs.cold.9+0x22/0x2f [ 3476.285744] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3476.297011] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3476.298700] ? watchdog+0x40/0x40 [ 3476.310224] ? hrtimer_interrupt+0x128/0x2c0 [ 3476.311857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3476.337552] ? apic_timer_interrupt+0xf/0x20 [ 3476.339146] [ 3476.339985] ? on_each_cpu_cond_mask+0x46/0x150 [ 3476.353063] ? __x64_sys_fspick+0x210/0x210 [ 3476.365048] ? smp_call_function_many_cond+0x3db/0x540 [ 3476.370069] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3476.379893] ? __x64_sys_fspick+0x210/0x210 [ 3476.387050] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3476.393318] on_each_cpu_cond_mask+0x46/0x150 [ 3476.400938] on_each_cpu_cond+0x1c/0x30 [ 3476.408380] invalidate_bh_lrus+0x27/0x30 [ 3476.412015] invalidate_bdev+0x5a/0x90 [ 3476.417811] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3476.422094] generic_shutdown_super+0xb1/0x1b0 [ 3476.430005] kill_block_super+0x29/0x80 [ 3476.431447] deactivate_locked_super+0x52/0xd0 [ 3476.440495] deactivate_super+0x83/0x90 [ 3476.441927] cleanup_mnt+0x5f/0xe0 [ 3476.443171] __cleanup_mnt+0x16/0x20 [ 3476.454750] task_work_run+0xc6/0x110 [ 3476.459907] exit_to_usermode_loop+0x1dd/0x1f0 [ 3476.468895] do_syscall_64+0x3d6/0x440 [ 3476.478232] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3476.480251] RIP: 0033:0x7fbb019648fb [ 3476.492903] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3476.516322] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3476.524563] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3476.531254] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3476.540571] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3476.550890] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3476.562594] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3484.055257] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3484.059014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3484.059373] Modules linked in: [ 3484.062551] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3484.062637] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3484.062644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3484.062686] Workqueue: events free_work [libcfs] [ 3484.062723] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3484.062731] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3484.062734] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3484.062738] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3484.062741] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3484.062743] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3484.062745] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3484.062747] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3484.062750] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3484.062753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3484.062755] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3484.062763] Call Trace: [ 3484.062768] [ 3484.062774] ? show_regs.cold.9+0x22/0x2f [ 3484.062783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3484.062788] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3484.062792] ? watchdog+0x40/0x40 [ 3484.062795] ? hrtimer_interrupt+0x128/0x2c0 [ 3484.062799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3484.062805] ? apic_timer_interrupt+0xf/0x20 [ 3484.062810] [ 3484.062811] ? on_each_cpu+0x51/0x120 [ 3484.062815] ? smp_call_function_many_cond+0x3db/0x540 [ 3484.062818] ? smp_call_function_many_cond+0x3a8/0x540 [ 3484.062821] ? load_new_mm_cr3+0x120/0x120 [ 3484.062826] ? load_new_mm_cr3+0x120/0x120 [ 3484.062829] on_each_cpu+0x51/0x120 [ 3484.062832] flush_tlb_kernel_range+0x93/0xb0 [ 3484.062837] free_unmap_vmap_area+0x63/0x70 [ 3484.062841] remove_vm_area+0xe7/0x110 [ 3484.062844] __vunmap+0x9c/0x400 [ 3484.062847] vfree+0x4d/0x100 [ 3484.062850] free_work+0x34/0x50 [libcfs] [ 3484.062875] process_one_work+0x2c8/0x700 [ 3484.062881] worker_thread+0x296/0x6e0 [ 3484.062885] ? rescuer_thread+0x570/0x570 [ 3484.062889] kthread+0x1d1/0x200 [ 3484.062893] ? set_kthread_struct+0x70/0x70 [ 3484.062896] ret_from_fork+0x1f/0x30 [ 3484.256121] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3484.277261] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3484.282609] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3484.286782] Workqueue: events free_work [libcfs] [ 3484.288845] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3484.291380] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3484.300420] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3484.304390] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3484.308193] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3484.311879] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3484.314690] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3484.318194] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3484.321489] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3484.323739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3484.326629] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3484.329847] Call Trace: [ 3484.331110] [ 3484.332082] ? show_regs.cold.9+0x22/0x2f [ 3484.333933] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3484.335984] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3484.337735] ? watchdog+0x40/0x40 [ 3484.339038] ? hrtimer_interrupt+0x128/0x2c0 [ 3484.340890] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3484.343033] ? apic_timer_interrupt+0xf/0x20 [ 3484.344871] [ 3484.345679] ? on_each_cpu+0x51/0x120 [ 3484.347217] ? smp_call_function_many_cond+0x3db/0x540 [ 3484.349534] ? smp_call_function_many_cond+0x3a8/0x540 [ 3484.351654] ? load_new_mm_cr3+0x120/0x120 [ 3484.353409] ? load_new_mm_cr3+0x120/0x120 [ 3484.354886] on_each_cpu+0x51/0x120 [ 3484.356428] flush_tlb_kernel_range+0x93/0xb0 [ 3484.358226] free_unmap_vmap_area+0x63/0x70 [ 3484.360028] remove_vm_area+0xe7/0x110 [ 3484.361667] __vunmap+0x9c/0x400 [ 3484.363053] vfree+0x4d/0x100 [ 3484.364340] free_work+0x34/0x50 [libcfs] [ 3484.365873] process_one_work+0x2c8/0x700 [ 3484.367421] worker_thread+0x296/0x6e0 [ 3484.368886] ? rescuer_thread+0x570/0x570 [ 3484.370606] kthread+0x1d1/0x200 [ 3484.372068] ? set_kthread_struct+0x70/0x70 [ 3484.373751] ret_from_fork+0x1f/0x30 [ 3487.713858] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1129s! [ 3487.719689] Showing busy workqueues and worker pools: [ 3487.728456] workqueue events: flags=0x0 [ 3487.732543] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3487.738002] in-flight: 56293:free_work [libcfs] [ 3487.740651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3487.749781] in-flight: 22277:free_work [libcfs] [ 3487.753278] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3487.761242] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3487.765840] pending: nfs4_renew_state [nfsv4] [ 3487.775720] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3487.785604] in-flight: 47596:free_work [libcfs] [ 3487.789542] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3487.796647] workqueue events_unbound: flags=0x2 [ 3487.807686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3487.820949] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3487.833874] workqueue events_power_efficient: flags=0x80 [ 3487.848468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3487.869756] pending: neigh_periodic_work, do_cache_clean [ 3487.871683] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3487.890511] pending: check_lifetime [ 3487.908179] workqueue mm_percpu_wq: flags=0x8 [ 3487.909706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3487.927519] pending: vmstat_update [ 3487.932456] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3487.943125] pending: vmstat_update [ 3487.956507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3487.958690] pending: vmstat_update [ 3487.969497] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3487.973550] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3487.988427] pending: cgroup_pidlist_destroy_work_fn [ 3487.998802] workqueue writeback: flags=0x4a [ 3488.001396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3488.009459] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3488.018985] workqueue xprtiod: flags=0xa [ 3488.020375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3488.030363] pending: xprt_autoclose [ 3488.035401] workqueue ipv6_addrconf: flags=0x40008 [ 3488.043718] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3488.048494] pending: addrconf_verify_work [ 3488.049940] workqueue knbd0-recv: flags=0x1a [ 3488.059435] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3488.075710] in-flight: 655:recv_work [ 3488.079375] workqueue ptlrpc_pinger: flags=0x2 [ 3488.087994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3488.090114] pending: ptlrpc_pinger_main [ptlrpc] [ 3488.103430] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1130s workers=5 idle: 53684 27131 54850 46584 [ 3488.114558] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1130s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3488.126001] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1130s workers=6 idle: 53660 37674 46417 53664 46418 [ 3488.132306] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3477s workers=2 idle: 657 [ 3497.763015] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 3497.766794] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 3497.772121] (detected by 2, t=1140033 jiffies, g=626849, q=1398) [ 3497.775182] Sending NMI from CPU 2 to CPUs 3: [ 3497.778354] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 3497.778362] rcu: rcu_sched kthread starved for 1139675 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3497.788434] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3497.795114] rcu: RCU grace-period kthread stack dump: [ 3497.798274] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3497.803964] Call Trace: [ 3497.804965] __schedule+0x351/0xcb0 [ 3497.806285] schedule+0xc0/0x180 [ 3497.808032] schedule_timeout+0xb4/0x190 [ 3497.810141] ? do_raw_spin_unlock+0x75/0x190 [ 3497.812601] ? __next_timer_interrupt+0x160/0x160 [ 3497.815266] ? prepare_to_swait_event+0xf3/0x140 [ 3497.817992] rcu_gp_kthread+0x8dd/0x15b0 [ 3497.819945] ? force_qs_rnp+0x2c0/0x2c0 [ 3497.821792] kthread+0x1d1/0x200 [ 3497.823243] ? set_kthread_struct+0x70/0x70 [ 3497.825165] ret_from_fork+0x1f/0x30 [ 3497.827030] rcu: Stack dump where RCU GP kthread last ran: [ 3497.829853] NMI backtrace for cpu 2 [ 3497.831906] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3497.838431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3497.843961] Workqueue: events free_work [libcfs] [ 3497.846693] Call Trace: [ 3497.847875] [ 3497.848854] ? dump_stack+0xbb/0x10e [ 3497.850323] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3497.852889] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3497.854944] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3497.857111] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3497.859230] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3497.861151] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3497.863292] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 3497.865399] ? update_process_times+0x6d/0xc0 [ 3497.867800] ? tick_sched_handle.isra.18+0x27/0x90 [ 3497.870246] ? tick_sched_timer+0x53/0xb0 [ 3497.872656] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3497.875216] ? tick_sched_handle.isra.18+0x90/0x90 [ 3497.877459] ? hrtimer_interrupt+0x128/0x2c0 [ 3497.880031] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3497.882445] ? apic_timer_interrupt+0xf/0x20 [ 3497.884921] [ 3497.885936] ? on_each_cpu+0x51/0x120 [ 3497.888029] ? smp_call_function_many_cond+0x3db/0x540 [ 3497.890802] ? smp_call_function_many_cond+0x3a8/0x540 [ 3497.893357] ? load_new_mm_cr3+0x120/0x120 [ 3497.895942] ? load_new_mm_cr3+0x120/0x120 [ 3497.898118] ? on_each_cpu+0x51/0x120 [ 3497.900010] ? flush_tlb_kernel_range+0x93/0xb0 [ 3497.902018] ? free_unmap_vmap_area+0x63/0x70 [ 3497.903820] ? remove_vm_area+0xe7/0x110 [ 3497.905263] ? __vunmap+0x9c/0x400 [ 3497.907401] ? vfree+0x4d/0x100 [ 3497.908904] ? free_work+0x34/0x50 [libcfs] [ 3497.910404] ? process_one_work+0x2c8/0x700 [ 3497.911844] ? worker_thread+0x296/0x6e0 [ 3497.913015] ? rescuer_thread+0x570/0x570 [ 3497.914115] ? kthread+0x1d1/0x200 [ 3497.915148] ? set_kthread_struct+0x70/0x70 [ 3497.916714] ? ret_from_fork+0x1f/0x30 [ 3504.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3504.065563] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3504.121225] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3504.129692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3504.145438] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3504.148529] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3504.163102] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3504.171072] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3504.176840] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3504.181633] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3504.189086] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3504.194251] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3504.197813] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3504.203036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3504.209946] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3504.217576] Call Trace: [ 3504.219028] [ 3504.220136] ? show_regs.cold.9+0x22/0x2f [ 3504.222910] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3504.235900] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3504.237632] ? watchdog+0x40/0x40 [ 3504.240135] ? hrtimer_interrupt+0x128/0x2c0 [ 3504.244767] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3504.249525] ? apic_timer_interrupt+0xf/0x20 [ 3504.251170] [ 3504.259902] ? on_each_cpu_cond_mask+0x46/0x150 [ 3504.267260] ? __x64_sys_fspick+0x210/0x210 [ 3504.276151] ? smp_call_function_many_cond+0x3db/0x540 [ 3504.279636] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3504.284474] ? __x64_sys_fspick+0x210/0x210 [ 3504.287256] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3504.296870] on_each_cpu_cond_mask+0x46/0x150 [ 3504.298492] on_each_cpu_cond+0x1c/0x30 [ 3504.311182] invalidate_bh_lrus+0x27/0x30 [ 3504.312310] invalidate_bdev+0x5a/0x90 [ 3504.313254] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3504.321056] generic_shutdown_super+0xb1/0x1b0 [ 3504.328391] kill_block_super+0x29/0x80 [ 3504.330858] deactivate_locked_super+0x52/0xd0 [ 3504.340685] deactivate_super+0x83/0x90 [ 3504.344998] cleanup_mnt+0x5f/0xe0 [ 3504.349787] __cleanup_mnt+0x16/0x20 [ 3504.351168] task_work_run+0xc6/0x110 [ 3504.352593] exit_to_usermode_loop+0x1dd/0x1f0 [ 3504.361536] do_syscall_64+0x3d6/0x440 [ 3504.365425] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3504.370346] RIP: 0033:0x7fbb019648fb [ 3504.373895] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3504.397932] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3504.403130] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3504.414101] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3504.424790] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3504.434892] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3504.437290] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3516.055469] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3516.061518] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3516.112386] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3516.128807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3516.136089] Workqueue: events free_work [libcfs] [ 3516.141086] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3516.148843] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3516.179745] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3516.190848] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3516.195931] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3516.208018] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3516.216931] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3516.224862] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3516.232825] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3516.240093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3516.243193] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3516.248517] Call Trace: [ 3516.250490] [ 3516.251333] ? show_regs.cold.9+0x22/0x2f [ 3516.253424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3516.257181] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3516.259919] ? watchdog+0x40/0x40 [ 3516.261745] ? hrtimer_interrupt+0x128/0x2c0 [ 3516.265145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3516.268537] ? apic_timer_interrupt+0xf/0x20 [ 3516.270149] [ 3516.270994] ? on_each_cpu+0x51/0x120 [ 3516.274259] ? smp_call_function_many_cond+0x3db/0x540 [ 3516.277489] ? smp_call_function_many_cond+0x3a8/0x540 [ 3516.279289] ? load_new_mm_cr3+0x120/0x120 [ 3516.284023] ? load_new_mm_cr3+0x120/0x120 [ 3516.288981] on_each_cpu+0x51/0x120 [ 3516.290158] flush_tlb_kernel_range+0x93/0xb0 [ 3516.294159] free_unmap_vmap_area+0x63/0x70 [ 3516.296959] remove_vm_area+0xe7/0x110 [ 3516.298935] __vunmap+0x9c/0x400 [ 3516.300842] vfree+0x4d/0x100 [ 3516.303016] free_work+0x34/0x50 [libcfs] [ 3516.305914] process_one_work+0x2c8/0x700 [ 3516.308173] worker_thread+0x296/0x6e0 [ 3516.310815] ? rescuer_thread+0x570/0x570 [ 3516.314492] kthread+0x1d1/0x200 [ 3516.317316] ? set_kthread_struct+0x70/0x70 [ 3516.322951] ret_from_fork+0x1f/0x30 [ 3518.433024] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1160s! [ 3518.436677] Showing busy workqueues and worker pools: [ 3518.441162] workqueue events: flags=0x0 [ 3518.442815] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3518.445672] in-flight: 56293:free_work [libcfs] [ 3518.447980] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3518.451125] in-flight: 22277:free_work [libcfs] [ 3518.453507] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3518.457835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3518.460734] pending: nfs4_renew_state [nfsv4] [ 3518.462309] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3518.465718] in-flight: 47596:free_work [libcfs] [ 3518.470735] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3518.475885] workqueue events_unbound: flags=0x2 [ 3518.478303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3518.482502] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3518.486976] workqueue events_power_efficient: flags=0x80 [ 3518.489820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3518.499831] pending: neigh_periodic_work, do_cache_clean [ 3518.504754] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3518.509793] pending: check_lifetime [ 3518.512872] workqueue mm_percpu_wq: flags=0x8 [ 3518.519027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3518.524122] pending: vmstat_update [ 3518.527042] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3518.532147] pending: vmstat_update [ 3518.534601] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3518.542736] pending: vmstat_update [ 3518.545284] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3518.548138] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3518.551288] pending: cgroup_pidlist_destroy_work_fn [ 3518.555790] workqueue writeback: flags=0x4a [ 3518.558693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3518.564877] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3518.571629] workqueue xprtiod: flags=0xa [ 3518.578062] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3518.587327] pending: xprt_autoclose [ 3518.589398] workqueue ipv6_addrconf: flags=0x40008 [ 3518.591233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3518.597893] pending: addrconf_verify_work [ 3518.602710] workqueue knbd0-recv: flags=0x1a [ 3518.606988] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3518.611950] in-flight: 655:recv_work [ 3518.614467] workqueue ptlrpc_pinger: flags=0x2 [ 3518.617436] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3518.622737] pending: ptlrpc_pinger_main [ptlrpc] [ 3518.627736] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1160s workers=5 idle: 53684 27131 54850 46584 [ 3518.637700] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1160s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3518.647262] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1160s workers=6 idle: 53660 37674 46417 53664 46418 [ 3518.655364] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3508s workers=2 idle: 657 [ 3524.061961] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3524.075829] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.148366] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.154204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3524.158250] Workqueue: events free_work [libcfs] [ 3524.159968] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3524.165333] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3524.176016] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3524.180669] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3524.184876] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3524.189582] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3524.195570] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3524.200692] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3524.205544] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3524.219735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.225930] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3524.232637] Call Trace: [ 3524.234833] [ 3524.238514] ? show_regs.cold.9+0x22/0x2f [ 3524.240790] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.242730] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.249538] ? watchdog+0x40/0x40 [ 3524.250972] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.255478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.257222] ? apic_timer_interrupt+0xf/0x20 [ 3524.260470] [ 3524.263728] ? on_each_cpu+0x51/0x120 [ 3524.265666] ? smp_call_function_many_cond+0x3db/0x540 [ 3524.267130] ? smp_call_function_many_cond+0x3a8/0x540 [ 3524.268980] ? load_new_mm_cr3+0x120/0x120 [ 3524.273420] ? load_new_mm_cr3+0x120/0x120 [ 3524.275544] on_each_cpu+0x51/0x120 [ 3524.277458] flush_tlb_kernel_range+0x93/0xb0 [ 3524.281736] free_unmap_vmap_area+0x63/0x70 [ 3524.283660] remove_vm_area+0xe7/0x110 [ 3524.285923] __vunmap+0x9c/0x400 [ 3524.287566] vfree+0x4d/0x100 [ 3524.290433] free_work+0x34/0x50 [libcfs] [ 3524.295480] process_one_work+0x2c8/0x700 [ 3524.297626] worker_thread+0x296/0x6e0 [ 3524.298837] ? rescuer_thread+0x570/0x570 [ 3524.303508] kthread+0x1d1/0x200 [ 3524.305576] ? set_kthread_struct+0x70/0x70 [ 3524.307494] ret_from_fork+0x1f/0x30 [ 3532.059113] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3532.071476] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3532.136842] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3532.151234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3532.167319] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3532.176559] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3532.190110] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3532.201751] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3532.212960] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3532.224896] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3532.232211] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3532.242778] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3532.255120] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3532.262782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3532.271320] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3532.281312] Call Trace: [ 3532.287447] [ 3532.289838] ? show_regs.cold.9+0x22/0x2f [ 3532.291844] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3532.295399] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3532.298766] ? watchdog+0x40/0x40 [ 3532.301169] ? hrtimer_interrupt+0x128/0x2c0 [ 3532.304786] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3532.311389] ? apic_timer_interrupt+0xf/0x20 [ 3532.317796] [ 3532.325289] ? on_each_cpu_cond_mask+0x46/0x150 [ 3532.328732] ? __x64_sys_fspick+0x210/0x210 [ 3532.331422] ? smp_call_function_many_cond+0x3d9/0x540 [ 3532.335119] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3532.339784] ? __x64_sys_fspick+0x210/0x210 [ 3532.341634] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3532.344296] on_each_cpu_cond_mask+0x46/0x150 [ 3532.352632] on_each_cpu_cond+0x1c/0x30 [ 3532.358277] invalidate_bh_lrus+0x27/0x30 [ 3532.362637] invalidate_bdev+0x5a/0x90 [ 3532.365841] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3532.369306] generic_shutdown_super+0xb1/0x1b0 [ 3532.371815] kill_block_super+0x29/0x80 [ 3532.373936] deactivate_locked_super+0x52/0xd0 [ 3532.376174] deactivate_super+0x83/0x90 [ 3532.378623] cleanup_mnt+0x5f/0xe0 [ 3532.380347] __cleanup_mnt+0x16/0x20 [ 3532.381756] task_work_run+0xc6/0x110 [ 3532.386820] exit_to_usermode_loop+0x1dd/0x1f0 [ 3532.388789] do_syscall_64+0x3d6/0x440 [ 3532.391832] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3532.395006] RIP: 0033:0x7fbb019648fb [ 3532.396235] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3532.407313] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3532.409987] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3532.416055] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3532.422838] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3532.427426] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3532.432603] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3544.055059] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3544.058485] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3544.081901] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3544.088329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3544.098336] Workqueue: events free_work [libcfs] [ 3544.106935] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3544.111733] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3544.126909] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3544.134043] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3544.139351] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3544.143863] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3544.147432] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3544.151450] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3544.157397] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3544.164975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3544.168083] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3544.172955] Call Trace: [ 3544.174185] [ 3544.175815] ? show_regs.cold.9+0x22/0x2f [ 3544.178590] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3544.184568] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3544.190158] ? watchdog+0x40/0x40 [ 3544.191677] ? hrtimer_interrupt+0x128/0x2c0 [ 3544.194537] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3544.200254] ? apic_timer_interrupt+0xf/0x20 [ 3544.205092] [ 3544.206185] ? on_each_cpu+0x51/0x120 [ 3544.210138] ? smp_call_function_many_cond+0x3db/0x540 [ 3544.216996] ? smp_call_function_many_cond+0x3a8/0x540 [ 3544.225968] ? load_new_mm_cr3+0x120/0x120 [ 3544.235370] ? load_new_mm_cr3+0x120/0x120 [ 3544.238396] on_each_cpu+0x51/0x120 [ 3544.242306] flush_tlb_kernel_range+0x93/0xb0 [ 3544.243913] free_unmap_vmap_area+0x63/0x70 [ 3544.250493] remove_vm_area+0xe7/0x110 [ 3544.256996] __vunmap+0x9c/0x400 [ 3544.261109] vfree+0x4d/0x100 [ 3544.263758] free_work+0x34/0x50 [libcfs] [ 3544.268443] process_one_work+0x2c8/0x700 [ 3544.271066] worker_thread+0x296/0x6e0 [ 3544.274020] ? rescuer_thread+0x570/0x570 [ 3544.276411] kthread+0x1d1/0x200 [ 3544.279326] ? set_kthread_struct+0x70/0x70 [ 3544.281842] ret_from_fork+0x1f/0x30 [ 3549.151032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1190s! [ 3549.168063] Showing busy workqueues and worker pools: [ 3549.178210] workqueue events: flags=0x0 [ 3549.186295] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3549.194047] in-flight: 56293:free_work [libcfs] [ 3549.197384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3549.206423] in-flight: 22277:free_work [libcfs] [ 3549.215423] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3549.223172] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3549.229274] pending: nfs4_renew_state [nfsv4] [ 3549.232883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3549.238158] in-flight: 47596:free_work [libcfs] [ 3549.240022] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3549.260048] workqueue events_unbound: flags=0x2 [ 3549.265280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3549.278777] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3549.289325] workqueue events_power_efficient: flags=0x80 [ 3549.296628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3549.307296] pending: neigh_periodic_work, do_cache_clean [ 3549.312501] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3549.323488] pending: check_lifetime [ 3549.328816] workqueue mm_percpu_wq: flags=0x8 [ 3549.331257] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3549.336704] pending: vmstat_update [ 3549.338873] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3549.342621] pending: vmstat_update [ 3549.346188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3549.351254] pending: vmstat_update [ 3549.354288] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3549.361074] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3549.365352] pending: cgroup_pidlist_destroy_work_fn [ 3549.367828] workqueue writeback: flags=0x4a [ 3549.371363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3549.381005] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3549.385444] workqueue xprtiod: flags=0xa [ 3549.389369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3549.396394] pending: xprt_autoclose [ 3549.399994] workqueue ipv6_addrconf: flags=0x40008 [ 3549.404109] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3549.419942] pending: addrconf_verify_work [ 3549.423199] workqueue knbd0-recv: flags=0x1a [ 3549.427111] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3549.432261] in-flight: 655:recv_work [ 3549.437224] workqueue ptlrpc_pinger: flags=0x2 [ 3549.441217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3549.457146] pending: ptlrpc_pinger_main [ptlrpc] [ 3549.464983] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1191s workers=5 idle: 53684 27131 54850 46584 [ 3549.471386] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1191s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3549.479457] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1191s workers=6 idle: 53660 37674 46417 53664 46418 [ 3549.485953] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3539s workers=2 idle: 657 [ 3552.059020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3552.066429] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.103377] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.112161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3552.118299] Workqueue: events free_work [libcfs] [ 3552.120655] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3552.123841] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3552.135377] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.143720] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3552.147433] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3552.151649] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3552.163821] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3552.168650] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3552.173893] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3552.180130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.184165] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3552.193180] Call Trace: [ 3552.194701] [ 3552.198539] ? show_regs.cold.9+0x22/0x2f [ 3552.206799] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.208550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.215324] ? watchdog+0x40/0x40 [ 3552.216598] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.227039] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.229650] ? apic_timer_interrupt+0xf/0x20 [ 3552.232049] [ 3552.233304] ? on_each_cpu+0x51/0x120 [ 3552.235206] ? smp_call_function_many_cond+0x3db/0x540 [ 3552.239626] ? smp_call_function_many_cond+0x3a8/0x540 [ 3552.245279] ? load_new_mm_cr3+0x120/0x120 [ 3552.247823] ? load_new_mm_cr3+0x120/0x120 [ 3552.253608] on_each_cpu+0x51/0x120 [ 3552.258378] flush_tlb_kernel_range+0x93/0xb0 [ 3552.260937] free_unmap_vmap_area+0x63/0x70 [ 3552.270513] remove_vm_area+0xe7/0x110 [ 3552.272321] __vunmap+0x9c/0x400 [ 3552.276019] vfree+0x4d/0x100 [ 3552.277736] free_work+0x34/0x50 [libcfs] [ 3552.279574] process_one_work+0x2c8/0x700 [ 3552.286415] worker_thread+0x296/0x6e0 [ 3552.289336] ? rescuer_thread+0x570/0x570 [ 3552.294704] kthread+0x1d1/0x200 [ 3552.297054] ? set_kthread_struct+0x70/0x70 [ 3552.301789] ret_from_fork+0x1f/0x30 [ 3560.063874] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3560.076169] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3560.138776] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3560.145721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3560.150369] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3560.157197] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3560.171871] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3560.176541] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3560.181732] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3560.185530] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3560.190285] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3560.193339] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3560.197231] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3560.201123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3560.205944] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3560.209870] Call Trace: [ 3560.211901] [ 3560.216319] ? show_regs.cold.9+0x22/0x2f [ 3560.217848] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3560.222339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3560.224954] ? watchdog+0x40/0x40 [ 3560.227039] ? hrtimer_interrupt+0x128/0x2c0 [ 3560.230663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3560.235140] ? apic_timer_interrupt+0xf/0x20 [ 3560.239248] [ 3560.242307] ? on_each_cpu_cond_mask+0x46/0x150 [ 3560.244460] ? __x64_sys_fspick+0x210/0x210 [ 3560.246330] ? smp_call_function_many_cond+0x3db/0x540 [ 3560.248383] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3560.249911] ? __x64_sys_fspick+0x210/0x210 [ 3560.255066] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3560.257021] on_each_cpu_cond_mask+0x46/0x150 [ 3560.266691] on_each_cpu_cond+0x1c/0x30 [ 3560.268174] invalidate_bh_lrus+0x27/0x30 [ 3560.274817] invalidate_bdev+0x5a/0x90 [ 3560.276047] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3560.281556] generic_shutdown_super+0xb1/0x1b0 [ 3560.283211] kill_block_super+0x29/0x80 [ 3560.284340] deactivate_locked_super+0x52/0xd0 [ 3560.289100] deactivate_super+0x83/0x90 [ 3560.291437] cleanup_mnt+0x5f/0xe0 [ 3560.293183] __cleanup_mnt+0x16/0x20 [ 3560.294997] task_work_run+0xc6/0x110 [ 3560.297655] exit_to_usermode_loop+0x1dd/0x1f0 [ 3560.299976] do_syscall_64+0x3d6/0x440 [ 3560.302181] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3560.304197] RIP: 0033:0x7fbb019648fb [ 3560.306181] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3560.318135] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3560.320952] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3560.325819] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3560.327887] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3560.333499] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3560.337328] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3576.056363] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3576.067382] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3576.109432] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3576.118911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3576.127970] Workqueue: events free_work [libcfs] [ 3576.132359] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3576.139527] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3576.162159] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3576.168024] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3576.174804] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3576.181124] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3576.184567] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3576.190020] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3576.196119] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3576.202248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3576.205564] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3576.211891] Call Trace: [ 3576.214739] [ 3576.215848] ? show_regs.cold.9+0x22/0x2f [ 3576.218775] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3576.222587] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3576.225349] ? watchdog+0x40/0x40 [ 3576.230601] ? hrtimer_interrupt+0x128/0x2c0 [ 3576.235531] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3576.239387] ? apic_timer_interrupt+0xf/0x20 [ 3576.243911] [ 3576.246779] ? on_each_cpu+0x51/0x120 [ 3576.249612] ? smp_call_function_many_cond+0x3db/0x540 [ 3576.253181] ? smp_call_function_many_cond+0x3a8/0x540 [ 3576.258315] ? load_new_mm_cr3+0x120/0x120 [ 3576.262198] ? load_new_mm_cr3+0x120/0x120 [ 3576.264115] on_each_cpu+0x51/0x120 [ 3576.268854] flush_tlb_kernel_range+0x93/0xb0 [ 3576.273214] free_unmap_vmap_area+0x63/0x70 [ 3576.277423] remove_vm_area+0xe7/0x110 [ 3576.281719] __vunmap+0x9c/0x400 [ 3576.283832] vfree+0x4d/0x100 [ 3576.286060] free_work+0x34/0x50 [libcfs] [ 3576.289572] process_one_work+0x2c8/0x700 [ 3576.291881] worker_thread+0x296/0x6e0 [ 3576.295880] ? rescuer_thread+0x570/0x570 [ 3576.299117] kthread+0x1d1/0x200 [ 3576.301866] ? set_kthread_struct+0x70/0x70 [ 3576.305224] ret_from_fork+0x1f/0x30 [ 3579.872124] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1221s! [ 3579.879198] Showing busy workqueues and worker pools: [ 3579.882458] workqueue events: flags=0x0 [ 3579.884731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3579.889002] in-flight: 56293:free_work [libcfs] [ 3579.891423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3579.896072] in-flight: 22277:free_work [libcfs] [ 3579.899192] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3579.906446] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3579.911961] pending: nfs4_renew_state [nfsv4] [ 3579.915438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3579.918968] in-flight: 47596:free_work [libcfs] [ 3579.921223] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3579.925810] workqueue events_unbound: flags=0x2 [ 3579.927811] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3579.930614] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3579.933740] workqueue events_power_efficient: flags=0x80 [ 3579.936544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3579.939635] pending: neigh_periodic_work, do_cache_clean [ 3579.941804] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3579.944937] pending: check_lifetime [ 3579.947189] workqueue mm_percpu_wq: flags=0x8 [ 3579.948890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3579.951527] pending: vmstat_update [ 3579.952986] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3579.955507] pending: vmstat_update [ 3579.957455] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3579.961096] pending: vmstat_update [ 3579.962787] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3579.965279] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3579.967315] pending: cgroup_pidlist_destroy_work_fn [ 3579.969372] workqueue writeback: flags=0x4a [ 3579.971328] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3579.974048] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3579.978026] workqueue xprtiod: flags=0xa [ 3579.979968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3579.982677] pending: xprt_autoclose [ 3579.984503] workqueue ipv6_addrconf: flags=0x40008 [ 3579.986953] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3579.990071] pending: addrconf_verify_work [ 3579.992187] workqueue knbd0-recv: flags=0x1a [ 3579.994383] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3579.997247] in-flight: 655:recv_work [ 3579.999160] workqueue ptlrpc_pinger: flags=0x2 [ 3580.001357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3580.004474] pending: ptlrpc_pinger_main [ptlrpc] [ 3580.007186] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1222s workers=5 idle: 53684 27131 54850 46584 [ 3580.011524] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1222s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3580.016077] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1222s workers=6 idle: 53660 37674 46417 53664 46418 [ 3580.020245] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3569s workers=2 idle: 657 [ 3580.059892] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3580.065371] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.098248] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.103630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3580.111030] Workqueue: events free_work [libcfs] [ 3580.112789] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3580.118800] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3580.135243] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.138740] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3580.148451] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3580.159654] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3580.165612] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3580.171928] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3580.176486] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3580.181403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.187547] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3580.191868] Call Trace: [ 3580.195594] [ 3580.196326] ? show_regs.cold.9+0x22/0x2f [ 3580.197514] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.200921] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.202625] ? watchdog+0x40/0x40 [ 3580.206173] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.211538] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.213316] ? apic_timer_interrupt+0xf/0x20 [ 3580.215500] [ 3580.216529] ? on_each_cpu+0x51/0x120 [ 3580.220563] ? smp_call_function_many_cond+0x3db/0x540 [ 3580.225441] ? smp_call_function_many_cond+0x3a8/0x540 [ 3580.228901] ? load_new_mm_cr3+0x120/0x120 [ 3580.233043] ? load_new_mm_cr3+0x120/0x120 [ 3580.239550] on_each_cpu+0x51/0x120 [ 3580.240979] flush_tlb_kernel_range+0x93/0xb0 [ 3580.242643] free_unmap_vmap_area+0x63/0x70 [ 3580.245278] remove_vm_area+0xe7/0x110 [ 3580.246941] __vunmap+0x9c/0x400 [ 3580.247993] vfree+0x4d/0x100 [ 3580.248989] free_work+0x34/0x50 [libcfs] [ 3580.250431] process_one_work+0x2c8/0x700 [ 3580.252745] worker_thread+0x296/0x6e0 [ 3580.254706] ? rescuer_thread+0x570/0x570 [ 3580.256349] kthread+0x1d1/0x200 [ 3580.258155] ? set_kthread_struct+0x70/0x70 [ 3580.260037] ret_from_fork+0x1f/0x30 [ 3588.058309] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3588.063669] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3588.117574] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3588.129585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3588.138698] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3588.141620] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3588.155572] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3588.160122] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3588.168248] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3588.172200] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3588.179062] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3588.188342] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3588.194312] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3588.202268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3588.206167] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3588.210917] Call Trace: [ 3588.212340] [ 3588.213463] ? show_regs.cold.9+0x22/0x2f [ 3588.216429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3588.219178] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3588.221431] ? watchdog+0x40/0x40 [ 3588.223152] ? hrtimer_interrupt+0x128/0x2c0 [ 3588.225247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3588.227529] ? apic_timer_interrupt+0xf/0x20 [ 3588.229906] [ 3588.230956] ? on_each_cpu_cond_mask+0x46/0x150 [ 3588.233079] ? __x64_sys_fspick+0x210/0x210 [ 3588.234915] ? smp_call_function_many_cond+0x3db/0x540 [ 3588.237145] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3588.239164] ? __x64_sys_fspick+0x210/0x210 [ 3588.241168] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3588.244063] on_each_cpu_cond_mask+0x46/0x150 [ 3588.245707] on_each_cpu_cond+0x1c/0x30 [ 3588.248127] invalidate_bh_lrus+0x27/0x30 [ 3588.250210] invalidate_bdev+0x5a/0x90 [ 3588.253040] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3588.259169] generic_shutdown_super+0xb1/0x1b0 [ 3588.260868] kill_block_super+0x29/0x80 [ 3588.264373] deactivate_locked_super+0x52/0xd0 [ 3588.267140] deactivate_super+0x83/0x90 [ 3588.270222] cleanup_mnt+0x5f/0xe0 [ 3588.273920] __cleanup_mnt+0x16/0x20 [ 3588.278953] task_work_run+0xc6/0x110 [ 3588.280531] exit_to_usermode_loop+0x1dd/0x1f0 [ 3588.282782] do_syscall_64+0x3d6/0x440 [ 3588.286785] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3588.292109] RIP: 0033:0x7fbb019648fb [ 3588.294132] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3588.312424] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3588.316427] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3588.319586] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3588.323138] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3588.326264] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3588.330398] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3604.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3604.061625] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3604.105814] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3604.116249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3604.124499] Workqueue: events free_work [libcfs] [ 3604.126888] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3604.133929] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3604.148693] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3604.154647] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3604.160221] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3604.166733] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3604.173456] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3604.186842] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3604.199586] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3604.209626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3604.214950] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3604.223535] Call Trace: [ 3604.224172] [ 3604.224962] ? show_regs.cold.9+0x22/0x2f [ 3604.230191] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3604.233567] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3604.236213] ? watchdog+0x40/0x40 [ 3604.241237] ? hrtimer_interrupt+0x128/0x2c0 [ 3604.244176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3604.248820] ? apic_timer_interrupt+0xf/0x20 [ 3604.250968] [ 3604.254931] ? on_each_cpu+0x51/0x120 [ 3604.257810] ? smp_call_function_many_cond+0x3d9/0x540 [ 3604.262502] ? smp_call_function_many_cond+0x3a8/0x540 [ 3604.266515] ? load_new_mm_cr3+0x120/0x120 [ 3604.268753] ? load_new_mm_cr3+0x120/0x120 [ 3604.270657] on_each_cpu+0x51/0x120 [ 3604.272637] flush_tlb_kernel_range+0x93/0xb0 [ 3604.274632] free_unmap_vmap_area+0x63/0x70 [ 3604.276940] remove_vm_area+0xe7/0x110 [ 3604.278992] __vunmap+0x9c/0x400 [ 3604.280927] vfree+0x4d/0x100 [ 3604.282865] free_work+0x34/0x50 [libcfs] [ 3604.285008] process_one_work+0x2c8/0x700 [ 3604.288708] worker_thread+0x296/0x6e0 [ 3604.291068] ? rescuer_thread+0x570/0x570 [ 3604.293470] kthread+0x1d1/0x200 [ 3604.295631] ? set_kthread_struct+0x70/0x70 [ 3604.297907] ret_from_fork+0x1f/0x30 [ 3608.059102] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3608.067617] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3608.098580] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3608.103676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3608.108598] Workqueue: events free_work [libcfs] [ 3608.111575] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3608.117063] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3608.128580] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3608.132659] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3608.136697] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3608.140028] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3608.143558] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3608.147822] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3608.152097] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3608.160673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3608.166651] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3608.171970] Call Trace: [ 3608.174615] [ 3608.175853] ? show_regs.cold.9+0x22/0x2f [ 3608.178843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3608.182621] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3608.187098] ? watchdog+0x40/0x40 [ 3608.191833] ? hrtimer_interrupt+0x128/0x2c0 [ 3608.195629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3608.198750] ? apic_timer_interrupt+0xf/0x20 [ 3608.200913] [ 3608.202921] ? on_each_cpu+0x51/0x120 [ 3608.204873] ? smp_call_function_many_cond+0x3db/0x540 [ 3608.207148] ? smp_call_function_many_cond+0x3a8/0x540 [ 3608.209680] ? load_new_mm_cr3+0x120/0x120 [ 3608.211616] ? load_new_mm_cr3+0x120/0x120 [ 3608.213318] on_each_cpu+0x51/0x120 [ 3608.214987] flush_tlb_kernel_range+0x93/0xb0 [ 3608.217029] free_unmap_vmap_area+0x63/0x70 [ 3608.219008] remove_vm_area+0xe7/0x110 [ 3608.220718] __vunmap+0x9c/0x400 [ 3608.221919] vfree+0x4d/0x100 [ 3608.222840] free_work+0x34/0x50 [libcfs] [ 3608.224222] process_one_work+0x2c8/0x700 [ 3608.225783] worker_thread+0x296/0x6e0 [ 3608.227001] ? rescuer_thread+0x570/0x570 [ 3608.228505] kthread+0x1d1/0x200 [ 3608.229839] ? set_kthread_struct+0x70/0x70 [ 3608.231766] ret_from_fork+0x1f/0x30 [ 3610.079770] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1251s! [ 3610.087887] Showing busy workqueues and worker pools: [ 3610.092452] workqueue events: flags=0x0 [ 3610.096151] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3610.101329] in-flight: 56293:free_work [libcfs] [ 3610.105918] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3610.110084] in-flight: 22277:free_work [libcfs] [ 3610.113158] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3610.118815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3610.123016] pending: nfs4_renew_state [nfsv4] [ 3610.125438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3610.128481] in-flight: 47596:free_work [libcfs] [ 3610.130702] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3610.134932] workqueue events_unbound: flags=0x2 [ 3610.137060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3610.140463] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3610.144954] workqueue events_power_efficient: flags=0x80 [ 3610.148037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3610.151057] pending: neigh_periodic_work, do_cache_clean [ 3610.154047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3610.159918] pending: check_lifetime [ 3610.162372] workqueue mm_percpu_wq: flags=0x8 [ 3610.169807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3610.175069] pending: vmstat_update [ 3610.177118] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3610.181020] pending: vmstat_update [ 3610.183079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3610.187010] pending: vmstat_update [ 3610.189440] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3610.192146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3610.198493] pending: cgroup_pidlist_destroy_work_fn [ 3610.202944] workqueue writeback: flags=0x4a [ 3610.207168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3610.212979] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3610.219907] workqueue xprtiod: flags=0xa [ 3610.223125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3610.229031] pending: xprt_autoclose [ 3610.231829] workqueue ipv6_addrconf: flags=0x40008 [ 3610.235016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3610.243377] pending: addrconf_verify_work [ 3610.246318] workqueue knbd0-recv: flags=0x1a [ 3610.258610] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3610.261116] in-flight: 655:recv_work [ 3610.267507] workqueue ptlrpc_pinger: flags=0x2 [ 3610.271027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3610.275318] pending: ptlrpc_pinger_main [ptlrpc] [ 3610.278470] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1252s workers=5 idle: 53684 27131 54850 46584 [ 3610.282617] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1252s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3610.288132] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1252s workers=6 idle: 53660 37674 46417 53664 46418 [ 3610.292543] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3600s workers=2 idle: 657 [ 3616.058009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3616.060137] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3616.077269] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3616.080521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3616.083621] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3616.086096] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3616.105810] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3616.115313] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3616.124676] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3616.127657] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3616.130898] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3616.134885] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3616.137514] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3616.143822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3616.147656] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3616.153978] Call Trace: [ 3616.155614] [ 3616.157781] ? show_regs.cold.9+0x22/0x2f [ 3616.161305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3616.165465] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3616.169193] ? watchdog+0x40/0x40 [ 3616.171498] ? hrtimer_interrupt+0x128/0x2c0 [ 3616.173159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3616.176544] ? apic_timer_interrupt+0xf/0x20 [ 3616.179060] [ 3616.185529] ? on_each_cpu_cond_mask+0x46/0x150 [ 3616.189119] ? __x64_sys_fspick+0x210/0x210 [ 3616.190687] ? smp_call_function_many_cond+0x3db/0x540 [ 3616.195961] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3616.198878] ? __x64_sys_fspick+0x210/0x210 [ 3616.201150] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3616.205409] on_each_cpu_cond_mask+0x46/0x150 [ 3616.209940] on_each_cpu_cond+0x1c/0x30 [ 3616.214753] invalidate_bh_lrus+0x27/0x30 [ 3616.220105] invalidate_bdev+0x5a/0x90 [ 3616.224030] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3616.227525] generic_shutdown_super+0xb1/0x1b0 [ 3616.230025] kill_block_super+0x29/0x80 [ 3616.231996] deactivate_locked_super+0x52/0xd0 [ 3616.235632] deactivate_super+0x83/0x90 [ 3616.239592] cleanup_mnt+0x5f/0xe0 [ 3616.242493] __cleanup_mnt+0x16/0x20 [ 3616.250594] task_work_run+0xc6/0x110 [ 3616.254545] exit_to_usermode_loop+0x1dd/0x1f0 [ 3616.260964] do_syscall_64+0x3d6/0x440 [ 3616.269658] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3616.277675] RIP: 0033:0x7fbb019648fb [ 3616.281273] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3616.295671] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3616.302087] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3616.314110] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3616.318438] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3616.324533] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3616.330017] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3636.056444] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3636.059017] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3636.066840] Modules linked in: [ 3636.071974] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3636.072055] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3636.072062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3636.072069] Workqueue: events free_work [libcfs] [ 3636.072107] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3636.072115] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3636.072119] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3636.072124] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3636.072126] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3636.072128] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3636.072130] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3636.072131] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3636.072134] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3636.072137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3636.072139] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3636.072148] Call Trace: [ 3636.072153] [ 3636.072157] ? show_regs.cold.9+0x22/0x2f [ 3636.072246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3636.072252] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3636.072257] ? watchdog+0x40/0x40 [ 3636.072261] ? hrtimer_interrupt+0x128/0x2c0 [ 3636.072264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3636.072271] ? apic_timer_interrupt+0xf/0x20 [ 3636.072275] [ 3636.072277] ? on_each_cpu+0x51/0x120 [ 3636.072282] ? smp_call_function_many_cond+0x3db/0x540 [ 3636.072286] ? smp_call_function_many_cond+0x3a8/0x540 [ 3636.072288] ? load_new_mm_cr3+0x120/0x120 [ 3636.072293] ? load_new_mm_cr3+0x120/0x120 [ 3636.072296] on_each_cpu+0x51/0x120 [ 3636.072299] flush_tlb_kernel_range+0x93/0xb0 [ 3636.072304] free_unmap_vmap_area+0x63/0x70 [ 3636.072416] remove_vm_area+0xe7/0x110 [ 3636.072420] __vunmap+0x9c/0x400 [ 3636.072424] vfree+0x4d/0x100 [ 3636.072428] free_work+0x34/0x50 [libcfs] [ 3636.072453] process_one_work+0x2c8/0x700 [ 3636.072460] worker_thread+0x296/0x6e0 [ 3636.072465] ? rescuer_thread+0x570/0x570 [ 3636.072468] kthread+0x1d1/0x200 [ 3636.072472] ? set_kthread_struct+0x70/0x70 [ 3636.072476] ret_from_fork+0x1f/0x30 [ 3636.452131] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3636.470266] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3636.474135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3636.477298] Workqueue: events free_work [libcfs] [ 3636.479157] RIP: 0010:smp_call_function_many_cond+0x3dd/0x540 [ 3636.481227] Code: 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 8b 11 <83> e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 01 eb [ 3636.488372] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3636.491471] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3636.494708] RDX: 0000000000000011 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3636.497599] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3636.500417] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3636.503213] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3636.506597] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3636.510861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3636.513432] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3636.517164] Call Trace: [ 3636.518498] [ 3636.519495] ? show_regs.cold.9+0x22/0x2f [ 3636.521223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3636.523331] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3636.525968] ? watchdog+0x40/0x40 [ 3636.527325] ? hrtimer_interrupt+0x128/0x2c0 [ 3636.528941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3636.532689] ? apic_timer_interrupt+0xf/0x20 [ 3636.534359] [ 3636.535417] ? on_each_cpu+0x51/0x120 [ 3636.542616] ? smp_call_function_many_cond+0x3dd/0x540 [ 3636.547437] ? smp_call_function_many_cond+0x3a8/0x540 [ 3636.550035] ? load_new_mm_cr3+0x120/0x120 [ 3636.552315] ? load_new_mm_cr3+0x120/0x120 [ 3636.554308] on_each_cpu+0x51/0x120 [ 3636.556226] flush_tlb_kernel_range+0x93/0xb0 [ 3636.558743] free_unmap_vmap_area+0x63/0x70 [ 3636.561276] remove_vm_area+0xe7/0x110 [ 3636.563506] __vunmap+0x9c/0x400 [ 3636.566472] vfree+0x4d/0x100 [ 3636.570288] free_work+0x34/0x50 [libcfs] [ 3636.571791] process_one_work+0x2c8/0x700 [ 3636.576876] worker_thread+0x296/0x6e0 [ 3636.582580] ? rescuer_thread+0x570/0x570 [ 3636.584542] kthread+0x1d1/0x200 [ 3636.588040] ? set_kthread_struct+0x70/0x70 [ 3636.589858] ret_from_fork+0x1f/0x30 [ 3640.799028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1282s! [ 3640.808963] Showing busy workqueues and worker pools: [ 3640.816895] workqueue events: flags=0x0 [ 3640.826039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3640.838618] in-flight: 56293:free_work [libcfs] [ 3640.845647] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3640.862617] in-flight: 22277:free_work [libcfs] [ 3640.865711] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3640.881068] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3640.892861] pending: nfs4_renew_state [nfsv4] [ 3640.894807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3640.905852] in-flight: 47596:free_work [libcfs] [ 3640.909218] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3640.924817] workqueue events_unbound: flags=0x2 [ 3640.934940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3640.949020] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3640.959195] workqueue events_power_efficient: flags=0x80 [ 3640.967813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3640.975850] pending: neigh_periodic_work, do_cache_clean [ 3640.979381] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3640.987957] pending: check_lifetime [ 3640.992303] workqueue mm_percpu_wq: flags=0x8 [ 3640.999598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3641.004547] pending: vmstat_update [ 3641.005983] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3641.021382] pending: vmstat_update [ 3641.024799] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3641.032426] pending: vmstat_update [ 3641.038275] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3641.047453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3641.062931] pending: cgroup_pidlist_destroy_work_fn [ 3641.066791] workqueue writeback: flags=0x4a [ 3641.071048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3641.078426] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3641.082566] workqueue xprtiod: flags=0xa [ 3641.084013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3641.087918] pending: xprt_autoclose [ 3641.095042] workqueue ipv6_addrconf: flags=0x40008 [ 3641.098824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3641.108083] pending: addrconf_verify_work [ 3641.116948] workqueue knbd0-recv: flags=0x1a [ 3641.123738] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3641.131291] in-flight: 655:recv_work [ 3641.141802] workqueue ptlrpc_pinger: flags=0x2 [ 3641.147792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3641.153769] pending: ptlrpc_pinger_main [ptlrpc] [ 3641.160728] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1283s workers=5 idle: 53684 27131 54850 46584 [ 3641.176026] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1283s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3641.188628] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1283s workers=6 idle: 53660 37674 46417 53664 46418 [ 3641.202033] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3631s workers=2 idle: 657 [ 3644.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3644.061851] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3644.154731] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3644.176151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3644.184775] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3644.189690] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3644.213976] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3644.223344] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3644.241187] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3644.254934] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3644.257644] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3644.273878] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3644.281198] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3644.291324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3644.304943] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3644.317961] Call Trace: [ 3644.325537] [ 3644.326381] ? show_regs.cold.9+0x22/0x2f [ 3644.327973] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3644.355938] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3644.372148] ? watchdog+0x40/0x40 [ 3644.377757] ? hrtimer_interrupt+0x128/0x2c0 [ 3644.383914] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3644.390766] ? apic_timer_interrupt+0xf/0x20 [ 3644.397272] [ 3644.403356] ? on_each_cpu_cond_mask+0x46/0x150 [ 3644.406250] ? __x64_sys_fspick+0x210/0x210 [ 3644.407311] ? smp_call_function_many_cond+0x3d9/0x540 [ 3644.417161] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3644.427507] ? __x64_sys_fspick+0x210/0x210 [ 3644.431877] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3644.446873] on_each_cpu_cond_mask+0x46/0x150 [ 3644.454193] on_each_cpu_cond+0x1c/0x30 [ 3644.457007] invalidate_bh_lrus+0x27/0x30 [ 3644.459777] invalidate_bdev+0x5a/0x90 [ 3644.463198] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3644.472982] generic_shutdown_super+0xb1/0x1b0 [ 3644.477580] kill_block_super+0x29/0x80 [ 3644.479349] deactivate_locked_super+0x52/0xd0 [ 3644.481121] deactivate_super+0x83/0x90 [ 3644.490830] cleanup_mnt+0x5f/0xe0 [ 3644.494765] __cleanup_mnt+0x16/0x20 [ 3644.497046] task_work_run+0xc6/0x110 [ 3644.500075] exit_to_usermode_loop+0x1dd/0x1f0 [ 3644.504281] do_syscall_64+0x3d6/0x440 [ 3644.508026] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3644.512732] RIP: 0033:0x7fbb019648fb [ 3644.514770] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3644.535753] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3644.545959] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3644.556127] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3644.565892] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3644.576648] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3644.585884] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3664.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3664.063106] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3664.091664] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3664.097025] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3664.101385] Workqueue: events free_work [libcfs] [ 3664.103653] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3664.106422] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3664.116124] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3664.119935] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3664.123495] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3664.126683] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3664.129630] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3664.133150] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3664.137326] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3664.141514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3664.144384] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3664.148003] Call Trace: [ 3664.149431] [ 3664.150843] ? show_regs.cold.9+0x22/0x2f [ 3664.152932] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3664.156491] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3664.160519] ? watchdog+0x40/0x40 [ 3664.163632] ? hrtimer_interrupt+0x128/0x2c0 [ 3664.165596] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3664.168212] ? apic_timer_interrupt+0xf/0x20 [ 3664.170463] [ 3664.171652] ? on_each_cpu+0x51/0x120 [ 3664.173553] ? smp_call_function_many_cond+0x3d9/0x540 [ 3664.175848] ? smp_call_function_many_cond+0x3a8/0x540 [ 3664.178121] ? load_new_mm_cr3+0x120/0x120 [ 3664.179710] ? load_new_mm_cr3+0x120/0x120 [ 3664.181846] on_each_cpu+0x51/0x120 [ 3664.183140] flush_tlb_kernel_range+0x93/0xb0 [ 3664.187159] free_unmap_vmap_area+0x63/0x70 [ 3664.193402] remove_vm_area+0xe7/0x110 [ 3664.196430] __vunmap+0x9c/0x400 [ 3664.197830] vfree+0x4d/0x100 [ 3664.205714] free_work+0x34/0x50 [libcfs] [ 3664.209441] process_one_work+0x2c8/0x700 [ 3664.214695] worker_thread+0x296/0x6e0 [ 3664.217779] ? rescuer_thread+0x570/0x570 [ 3664.219364] kthread+0x1d1/0x200 [ 3664.221478] ? set_kthread_struct+0x70/0x70 [ 3664.224582] ret_from_fork+0x1f/0x30 [ 3668.056201] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3668.063575] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3668.121423] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3668.137704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3668.146736] Workqueue: events free_work [libcfs] [ 3668.155122] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3668.162378] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3668.177329] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3668.183374] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3668.186870] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3668.196298] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3668.202617] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3668.210346] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3668.215627] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3668.221671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3668.227903] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3668.233064] Call Trace: [ 3668.234019] [ 3668.238291] ? show_regs.cold.9+0x22/0x2f [ 3668.244364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3668.250840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3668.254483] ? watchdog+0x40/0x40 [ 3668.256647] ? hrtimer_interrupt+0x128/0x2c0 [ 3668.262680] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3668.266612] ? apic_timer_interrupt+0xf/0x20 [ 3668.273358] [ 3668.274673] ? on_each_cpu+0x51/0x120 [ 3668.279579] ? smp_call_function_many_cond+0x3d9/0x540 [ 3668.280870] ? smp_call_function_many_cond+0x3a8/0x540 [ 3668.282680] ? load_new_mm_cr3+0x120/0x120 [ 3668.287774] ? load_new_mm_cr3+0x120/0x120 [ 3668.292524] on_each_cpu+0x51/0x120 [ 3668.295597] flush_tlb_kernel_range+0x93/0xb0 [ 3668.299916] free_unmap_vmap_area+0x63/0x70 [ 3668.304835] remove_vm_area+0xe7/0x110 [ 3668.307746] __vunmap+0x9c/0x400 [ 3668.310736] vfree+0x4d/0x100 [ 3668.312540] free_work+0x34/0x50 [libcfs] [ 3668.316815] process_one_work+0x2c8/0x700 [ 3668.323706] worker_thread+0x296/0x6e0 [ 3668.330770] ? rescuer_thread+0x570/0x570 [ 3668.332756] kthread+0x1d1/0x200 [ 3668.335079] ? set_kthread_struct+0x70/0x70 [ 3668.338118] ret_from_fork+0x1f/0x30 [ 3671.521027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1313s! [ 3671.528494] Showing busy workqueues and worker pools: [ 3671.532183] workqueue events: flags=0x0 [ 3671.537273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3671.541981] in-flight: 56293:free_work [libcfs] [ 3671.554485] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3671.563357] in-flight: 22277:free_work [libcfs] [ 3671.567255] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3671.576279] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3671.588817] pending: nfs4_renew_state [nfsv4] [ 3671.590711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3671.599460] in-flight: 47596:free_work [libcfs] [ 3671.603464] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3671.611728] workqueue events_unbound: flags=0x2 [ 3671.616462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3671.618833] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3671.632568] workqueue events_power_efficient: flags=0x80 [ 3671.642325] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3671.650158] pending: neigh_periodic_work, do_cache_clean [ 3671.656224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3671.661608] pending: check_lifetime [ 3671.664292] workqueue mm_percpu_wq: flags=0x8 [ 3671.668807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3671.684272] pending: vmstat_update [ 3671.685653] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3671.691840] pending: vmstat_update [ 3671.695356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3671.700361] pending: vmstat_update [ 3671.702904] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3671.707715] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3671.722325] pending: cgroup_pidlist_destroy_work_fn [ 3671.727597] workqueue writeback: flags=0x4a [ 3671.730371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3671.740995] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3671.745562] workqueue xprtiod: flags=0xa [ 3671.749368] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3671.752910] pending: xprt_autoclose [ 3671.760244] workqueue ipv6_addrconf: flags=0x40008 [ 3671.768622] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3671.780370] pending: addrconf_verify_work [ 3671.787247] workqueue knbd0-recv: flags=0x1a [ 3671.795546] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3671.804045] in-flight: 655:recv_work [ 3671.807302] workqueue ptlrpc_pinger: flags=0x2 [ 3671.809495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3671.813021] pending: ptlrpc_pinger_main [ptlrpc] [ 3671.817360] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1314s workers=5 idle: 53684 27131 54850 46584 [ 3671.827478] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1314s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3671.842465] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1314s workers=6 idle: 53660 37674 46417 53664 46418 [ 3671.848964] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3661s workers=2 idle: 657 [ 3672.062513] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3672.066609] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3672.106650] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3672.117396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3672.127540] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3672.131139] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3672.146823] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3672.153728] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3672.157585] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3672.164794] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3672.173142] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3672.179283] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3672.188319] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3672.199700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3672.203407] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3672.207554] Call Trace: [ 3672.216481] [ 3672.219969] ? show_regs.cold.9+0x22/0x2f [ 3672.226288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3672.232612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3672.239315] ? watchdog+0x40/0x40 [ 3672.244493] ? hrtimer_interrupt+0x128/0x2c0 [ 3672.248978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3672.250832] ? apic_timer_interrupt+0xf/0x20 [ 3672.260496] [ 3672.262569] ? on_each_cpu_cond_mask+0x46/0x150 [ 3672.265534] ? __x64_sys_fspick+0x210/0x210 [ 3672.269492] ? smp_call_function_many_cond+0x3db/0x540 [ 3672.273469] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3672.276448] ? __x64_sys_fspick+0x210/0x210 [ 3672.281170] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3672.285404] on_each_cpu_cond_mask+0x46/0x150 [ 3672.288251] on_each_cpu_cond+0x1c/0x30 [ 3672.290298] invalidate_bh_lrus+0x27/0x30 [ 3672.292431] invalidate_bdev+0x5a/0x90 [ 3672.294441] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3672.297581] generic_shutdown_super+0xb1/0x1b0 [ 3672.300202] kill_block_super+0x29/0x80 [ 3672.302437] deactivate_locked_super+0x52/0xd0 [ 3672.304745] deactivate_super+0x83/0x90 [ 3672.306801] cleanup_mnt+0x5f/0xe0 [ 3672.309266] __cleanup_mnt+0x16/0x20 [ 3672.311240] task_work_run+0xc6/0x110 [ 3672.313310] exit_to_usermode_loop+0x1dd/0x1f0 [ 3672.316120] do_syscall_64+0x3d6/0x440 [ 3672.317764] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3672.319782] RIP: 0033:0x7fbb019648fb [ 3672.321113] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3672.329548] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3672.333258] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3672.336159] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3672.339668] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3672.343401] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3672.346756] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3677.768464] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 3677.773089] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 3677.781549] (detected by 2, t=1320038 jiffies, g=626849, q=1398) [ 3677.788332] Sending NMI from CPU 2 to CPUs 3: [ 3677.802579] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 3677.802591] rcu: rcu_sched kthread starved for 1319680 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 3677.821519] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3677.828738] rcu: RCU grace-period kthread stack dump: [ 3677.831734] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3677.843104] Call Trace: [ 3677.846523] __schedule+0x351/0xcb0 [ 3677.852945] schedule+0xc0/0x180 [ 3677.855829] schedule_timeout+0xb4/0x190 [ 3677.858084] ? do_raw_spin_unlock+0x75/0x190 [ 3677.859160] ? __next_timer_interrupt+0x160/0x160 [ 3677.860272] ? prepare_to_swait_event+0xf3/0x140 [ 3677.867655] rcu_gp_kthread+0x8dd/0x15b0 [ 3677.869236] ? force_qs_rnp+0x2c0/0x2c0 [ 3677.878364] kthread+0x1d1/0x200 [ 3677.882120] ? set_kthread_struct+0x70/0x70 [ 3677.885590] ret_from_fork+0x1f/0x30 [ 3677.887474] rcu: Stack dump where RCU GP kthread last ran: [ 3677.892689] Sending NMI from CPU 2 to CPUs 0: [ 3677.895100] NMI backtrace for cpu 0 [ 3677.895105] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3677.895108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3677.895110] Workqueue: events free_work [libcfs] [ 3677.895112] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3677.895114] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3677.895116] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 [ 3677.895118] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3677.895120] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3677.895121] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3677.895122] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3677.895123] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3677.895125] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3677.895126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3677.895127] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3677.895128] Call Trace: [ 3677.895129] [ 3677.895130] ? show_regs.cold.9+0x22/0x2f [ 3677.895131] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3677.895133] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3677.895134] ? nmi_handle+0xbd/0x210 [ 3677.895134] ? default_do_nmi+0x4b/0x1b0 [ 3677.895135] ? do_nmi+0x280/0x380 [ 3677.895136] ? end_repeat_nmi+0x16/0x1f [ 3677.895137] ? on_each_cpu+0x51/0x120 [ 3677.895138] ? smp_call_function_many_cond+0x3d9/0x540 [ 3677.895139] ? smp_call_function_many_cond+0x3d9/0x540 [ 3677.895141] ? smp_call_function_many_cond+0x3d9/0x540 [ 3677.895142] [ 3677.895143] ? load_new_mm_cr3+0x120/0x120 [ 3677.895144] ? load_new_mm_cr3+0x120/0x120 [ 3677.895145] on_each_cpu+0x51/0x120 [ 3677.895146] flush_tlb_kernel_range+0x93/0xb0 [ 3677.895146] free_unmap_vmap_area+0x63/0x70 [ 3677.895147] remove_vm_area+0xe7/0x110 [ 3677.895149] __vunmap+0x9c/0x400 [ 3677.895149] vfree+0x4d/0x100 [ 3677.895150] free_work+0x34/0x50 [libcfs] [ 3677.895152] process_one_work+0x2c8/0x700 [ 3677.895153] worker_thread+0x296/0x6e0 [ 3677.895154] ? rescuer_thread+0x570/0x570 [ 3677.895155] kthread+0x1d1/0x200 [ 3677.895156] ? set_kthread_struct+0x70/0x70 [ 3677.895157] ret_from_fork+0x1f/0x30 [ 3700.061011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3700.067702] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3700.106544] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3700.116452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3700.122118] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3700.128214] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3700.139406] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3700.143802] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3700.148384] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3700.155448] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3700.161885] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3700.166860] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3700.170898] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3700.178524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3700.183058] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3700.187435] Call Trace: [ 3700.188754] [ 3700.189554] ? show_regs.cold.9+0x22/0x2f [ 3700.192233] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3700.195088] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3700.197184] ? watchdog+0x40/0x40 [ 3700.199025] ? hrtimer_interrupt+0x128/0x2c0 [ 3700.201187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3700.203773] ? apic_timer_interrupt+0xf/0x20 [ 3700.205803] [ 3700.206870] ? on_each_cpu_cond_mask+0x46/0x150 [ 3700.209375] ? __x64_sys_fspick+0x210/0x210 [ 3700.211789] ? smp_call_function_many_cond+0x3db/0x540 [ 3700.213848] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3700.217372] ? __x64_sys_fspick+0x210/0x210 [ 3700.220988] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3700.223006] on_each_cpu_cond_mask+0x46/0x150 [ 3700.227262] on_each_cpu_cond+0x1c/0x30 [ 3700.228953] invalidate_bh_lrus+0x27/0x30 [ 3700.230585] invalidate_bdev+0x5a/0x90 [ 3700.232058] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3700.235332] generic_shutdown_super+0xb1/0x1b0 [ 3700.238721] kill_block_super+0x29/0x80 [ 3700.241134] deactivate_locked_super+0x52/0xd0 [ 3700.245485] deactivate_super+0x83/0x90 [ 3700.248044] cleanup_mnt+0x5f/0xe0 [ 3700.249516] __cleanup_mnt+0x16/0x20 [ 3700.250703] task_work_run+0xc6/0x110 [ 3700.251957] exit_to_usermode_loop+0x1dd/0x1f0 [ 3700.258633] do_syscall_64+0x3d6/0x440 [ 3700.261021] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3700.263026] RIP: 0033:0x7fbb019648fb [ 3700.266596] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3700.277888] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3700.281446] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3700.285183] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3700.291301] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3700.296175] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3700.300256] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3702.240437] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1343s! [ 3702.249416] Showing busy workqueues and worker pools: [ 3702.251390] workqueue events: flags=0x0 [ 3702.255396] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3702.258043] in-flight: 56293:free_work [libcfs] [ 3702.264119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3702.271008] in-flight: 22277:free_work [libcfs] [ 3702.274054] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3702.286393] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3702.290108] pending: nfs4_renew_state [nfsv4] [ 3702.291639] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3702.294124] in-flight: 47596:free_work [libcfs] [ 3702.295693] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3702.298895] workqueue events_unbound: flags=0x2 [ 3702.300081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3702.302139] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3702.305640] workqueue events_power_efficient: flags=0x80 [ 3702.307615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3702.309823] pending: neigh_periodic_work, do_cache_clean [ 3702.311788] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3702.314594] pending: check_lifetime [ 3702.317903] workqueue mm_percpu_wq: flags=0x8 [ 3702.319877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3702.323017] pending: vmstat_update [ 3702.324768] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3702.327631] pending: vmstat_update [ 3702.328957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3702.331597] pending: vmstat_update [ 3702.333013] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3702.335458] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3702.338144] pending: cgroup_pidlist_destroy_work_fn [ 3702.340472] workqueue writeback: flags=0x4a [ 3702.342233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3702.345290] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3702.348645] workqueue xprtiod: flags=0xa [ 3702.354510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3702.359145] pending: xprt_autoclose [ 3702.361372] workqueue ipv6_addrconf: flags=0x40008 [ 3702.366699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3702.371725] pending: addrconf_verify_work [ 3702.377866] workqueue knbd0-recv: flags=0x1a [ 3702.380552] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3702.387946] in-flight: 655:recv_work [ 3702.389997] workqueue ptlrpc_pinger: flags=0x2 [ 3702.391713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3702.395884] pending: ptlrpc_pinger_main [ptlrpc] [ 3702.404853] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1344s workers=5 idle: 53684 27131 54850 46584 [ 3702.412887] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1344s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3702.419416] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1344s workers=6 idle: 53660 37674 46417 53664 46418 [ 3702.430682] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3692s workers=2 idle: 657 [ 3704.059973] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3704.068361] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3704.127742] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3704.147303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3704.153925] Workqueue: events free_work [libcfs] [ 3704.160273] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3704.166420] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3704.186902] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3704.197389] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3704.209712] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3704.216756] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3704.220691] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3704.227745] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3704.231947] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3704.238470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3704.246145] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3704.262448] Call Trace: [ 3704.267341] [ 3704.270123] ? show_regs.cold.9+0x22/0x2f [ 3704.273371] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3704.276085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3704.278743] ? watchdog+0x40/0x40 [ 3704.280567] ? hrtimer_interrupt+0x128/0x2c0 [ 3704.283491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3704.287106] ? apic_timer_interrupt+0xf/0x20 [ 3704.293651] [ 3704.296882] ? on_each_cpu+0x51/0x120 [ 3704.302636] ? smp_call_function_many_cond+0x3db/0x540 [ 3704.311352] ? smp_call_function_many_cond+0x3a8/0x540 [ 3704.316375] ? load_new_mm_cr3+0x120/0x120 [ 3704.326127] ? load_new_mm_cr3+0x120/0x120 [ 3704.329445] on_each_cpu+0x51/0x120 [ 3704.333795] flush_tlb_kernel_range+0x93/0xb0 [ 3704.338627] free_unmap_vmap_area+0x63/0x70 [ 3704.343581] remove_vm_area+0xe7/0x110 [ 3704.345001] __vunmap+0x9c/0x400 [ 3704.347600] vfree+0x4d/0x100 [ 3704.351446] free_work+0x34/0x50 [libcfs] [ 3704.353779] process_one_work+0x2c8/0x700 [ 3704.355902] worker_thread+0x296/0x6e0 [ 3704.361787] ? rescuer_thread+0x570/0x570 [ 3704.365220] kthread+0x1d1/0x200 [ 3704.369805] ? set_kthread_struct+0x70/0x70 [ 3704.377984] ret_from_fork+0x1f/0x30 [ 3728.055492] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3728.058009] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3728.071340] Modules linked in: [ 3728.074543] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3728.074747] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3728.074754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3728.074757] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3728.074769] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3728.074774] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3728.074778] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3728.074781] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3728.074783] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3728.074785] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3728.074788] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3728.074791] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3728.074794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3728.074797] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3728.074805] Call Trace: [ 3728.074810] [ 3728.074815] ? show_regs.cold.9+0x22/0x2f [ 3728.074824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3728.074830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3728.074835] ? watchdog+0x40/0x40 [ 3728.074839] ? hrtimer_interrupt+0x128/0x2c0 [ 3728.074843] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3728.074853] ? apic_timer_interrupt+0xf/0x20 [ 3728.074858] [ 3728.074860] ? on_each_cpu_cond_mask+0x46/0x150 [ 3728.074864] ? __x64_sys_fspick+0x210/0x210 [ 3728.074884] ? smp_call_function_many_cond+0x3db/0x540 [ 3728.074888] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3728.074892] ? __x64_sys_fspick+0x210/0x210 [ 3728.074897] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3728.074899] on_each_cpu_cond_mask+0x46/0x150 [ 3728.074903] on_each_cpu_cond+0x1c/0x30 [ 3728.074906] invalidate_bh_lrus+0x27/0x30 [ 3728.074909] invalidate_bdev+0x5a/0x90 [ 3728.074917] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3728.075045] generic_shutdown_super+0xb1/0x1b0 [ 3728.075053] kill_block_super+0x29/0x80 [ 3728.075058] deactivate_locked_super+0x52/0xd0 [ 3728.075063] deactivate_super+0x83/0x90 [ 3728.075067] cleanup_mnt+0x5f/0xe0 [ 3728.075075] __cleanup_mnt+0x16/0x20 [ 3728.075079] task_work_run+0xc6/0x110 [ 3728.075088] exit_to_usermode_loop+0x1dd/0x1f0 [ 3728.075095] do_syscall_64+0x3d6/0x440 [ 3728.075099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3728.075104] RIP: 0033:0x7fbb019648fb [ 3728.075124] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3728.075127] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3728.075131] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3728.075133] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3728.075135] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3728.075138] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3728.075140] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3728.278109] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3728.334196] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3728.346248] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3728.356568] Workqueue: events free_work [libcfs] [ 3728.363489] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3728.373335] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3728.389629] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3728.393524] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3728.398099] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3728.401844] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3728.405628] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3728.408292] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3728.416974] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3728.424571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3728.430038] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3728.445535] Call Trace: [ 3728.452253] [ 3728.454188] ? show_regs.cold.9+0x22/0x2f [ 3728.457313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3728.461913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3728.465027] ? watchdog+0x40/0x40 [ 3728.467962] ? hrtimer_interrupt+0x128/0x2c0 [ 3728.475047] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3728.478826] ? apic_timer_interrupt+0xf/0x20 [ 3728.488708] [ 3728.489480] ? on_each_cpu+0x51/0x120 [ 3728.490815] ? smp_call_function_many_cond+0x3db/0x540 [ 3728.506550] ? smp_call_function_many_cond+0x3a8/0x540 [ 3728.519886] ? load_new_mm_cr3+0x120/0x120 [ 3728.523303] ? load_new_mm_cr3+0x120/0x120 [ 3728.528572] on_each_cpu+0x51/0x120 [ 3728.534073] flush_tlb_kernel_range+0x93/0xb0 [ 3728.538095] free_unmap_vmap_area+0x63/0x70 [ 3728.539683] remove_vm_area+0xe7/0x110 [ 3728.548016] __vunmap+0x9c/0x400 [ 3728.550040] vfree+0x4d/0x100 [ 3728.551722] free_work+0x34/0x50 [libcfs] [ 3728.567632] process_one_work+0x2c8/0x700 [ 3728.569874] worker_thread+0x296/0x6e0 [ 3728.574115] ? rescuer_thread+0x570/0x570 [ 3728.578937] kthread+0x1d1/0x200 [ 3728.598412] ? set_kthread_struct+0x70/0x70 [ 3728.600005] ret_from_fork+0x1f/0x30 [ 3732.065631] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3732.085381] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.142097] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.150371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3732.156871] Workqueue: events free_work [libcfs] [ 3732.159819] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3732.164502] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3732.176071] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3732.180134] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3732.184205] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3732.187521] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3732.192141] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3732.196898] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3732.202288] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3732.208091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.211548] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3732.218599] Call Trace: [ 3732.219591] [ 3732.233890] ? show_regs.cold.9+0x22/0x2f [ 3732.240288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.249300] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.253380] ? watchdog+0x40/0x40 [ 3732.256255] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.259391] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.262716] ? apic_timer_interrupt+0xf/0x20 [ 3732.265877] [ 3732.266996] ? on_each_cpu+0x51/0x120 [ 3732.268456] ? smp_call_function_many_cond+0x3db/0x540 [ 3732.273026] ? smp_call_function_many_cond+0x3a8/0x540 [ 3732.276642] ? load_new_mm_cr3+0x120/0x120 [ 3732.279893] ? load_new_mm_cr3+0x120/0x120 [ 3732.283007] on_each_cpu+0x51/0x120 [ 3732.290090] flush_tlb_kernel_range+0x93/0xb0 [ 3732.301295] free_unmap_vmap_area+0x63/0x70 [ 3732.304695] remove_vm_area+0xe7/0x110 [ 3732.311082] __vunmap+0x9c/0x400 [ 3732.314073] vfree+0x4d/0x100 [ 3732.317345] free_work+0x34/0x50 [libcfs] [ 3732.323166] process_one_work+0x2c8/0x700 [ 3732.330233] worker_thread+0x296/0x6e0 [ 3732.337398] ? rescuer_thread+0x570/0x570 [ 3732.341462] kthread+0x1d1/0x200 [ 3732.345938] ? set_kthread_struct+0x70/0x70 [ 3732.347433] ret_from_fork+0x1f/0x30 [ 3732.449469] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1374s! [ 3732.459131] Showing busy workqueues and worker pools: [ 3732.467479] workqueue events: flags=0x0 [ 3732.474046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3732.480471] in-flight: 56293:free_work [libcfs] [ 3732.483968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3732.491096] in-flight: 22277:free_work [libcfs] [ 3732.497058] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3732.512241] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3732.516956] pending: nfs4_renew_state [nfsv4] [ 3732.522820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3732.530056] in-flight: 47596:free_work [libcfs] [ 3732.537434] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3732.551070] workqueue events_unbound: flags=0x2 [ 3732.554626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3732.560468] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3732.570635] workqueue events_power_efficient: flags=0x80 [ 3732.574527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3732.577965] pending: neigh_periodic_work, do_cache_clean [ 3732.581698] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3732.586078] pending: check_lifetime [ 3732.588834] workqueue mm_percpu_wq: flags=0x8 [ 3732.591654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3732.596045] pending: vmstat_update [ 3732.598048] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3732.602117] pending: vmstat_update [ 3732.604468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3732.608051] pending: vmstat_update [ 3732.610146] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3732.612401] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3732.616463] pending: cgroup_pidlist_destroy_work_fn [ 3732.619747] workqueue writeback: flags=0x4a [ 3732.621928] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3732.623978] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3732.626875] workqueue xprtiod: flags=0xa [ 3732.628994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3732.632194] pending: xprt_autoclose [ 3732.634288] workqueue ipv6_addrconf: flags=0x40008 [ 3732.636715] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3732.639840] pending: addrconf_verify_work [ 3732.642878] workqueue knbd0-recv: flags=0x1a [ 3732.644571] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3732.656146] in-flight: 655:recv_work [ 3732.661960] workqueue ptlrpc_pinger: flags=0x2 [ 3732.666748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3732.669159] pending: ptlrpc_pinger_main [ptlrpc] [ 3732.683490] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1374s workers=5 idle: 53684 27131 54850 46584 [ 3732.692832] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1374s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3732.709462] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1374s workers=6 idle: 53660 37674 46417 53664 46418 [ 3732.718497] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3722s workers=2 idle: 657 [ 3756.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3756.062329] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3756.090007] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3756.094935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3756.098801] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3756.106022] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3756.126480] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3756.132527] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3756.140845] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3756.146894] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3756.150052] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3756.154196] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3756.157526] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3756.161162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3756.165994] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3756.175582] Call Trace: [ 3756.180780] [ 3756.182469] ? show_regs.cold.9+0x22/0x2f [ 3756.186884] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3756.194497] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3756.196332] ? watchdog+0x40/0x40 [ 3756.199563] ? hrtimer_interrupt+0x128/0x2c0 [ 3756.206487] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3756.209429] ? apic_timer_interrupt+0xf/0x20 [ 3756.210716] [ 3756.211534] ? on_each_cpu_cond_mask+0x46/0x150 [ 3756.213973] ? __x64_sys_fspick+0x210/0x210 [ 3756.219483] ? smp_call_function_many_cond+0x3db/0x540 [ 3756.220982] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3756.223560] ? __x64_sys_fspick+0x210/0x210 [ 3756.224758] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3756.229661] on_each_cpu_cond_mask+0x46/0x150 [ 3756.231014] on_each_cpu_cond+0x1c/0x30 [ 3756.235436] invalidate_bh_lrus+0x27/0x30 [ 3756.237969] invalidate_bdev+0x5a/0x90 [ 3756.239186] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3756.241836] generic_shutdown_super+0xb1/0x1b0 [ 3756.243833] kill_block_super+0x29/0x80 [ 3756.245186] deactivate_locked_super+0x52/0xd0 [ 3756.246910] deactivate_super+0x83/0x90 [ 3756.254520] cleanup_mnt+0x5f/0xe0 [ 3756.256563] __cleanup_mnt+0x16/0x20 [ 3756.258558] task_work_run+0xc6/0x110 [ 3756.260561] exit_to_usermode_loop+0x1dd/0x1f0 [ 3756.263473] do_syscall_64+0x3d6/0x440 [ 3756.265554] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3756.267771] RIP: 0033:0x7fbb019648fb [ 3756.269439] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3756.276581] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3756.279648] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3756.287671] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3756.292980] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3756.296865] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3756.300620] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3760.055301] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3760.060856] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3760.064070] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.064154] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.064162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3760.064169] Workqueue: events free_work [libcfs] [ 3760.064216] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3760.064225] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3760.064230] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3760.064246] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3760.064248] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3760.064250] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3760.064253] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3760.064255] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3760.064257] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3760.064260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.064262] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3760.064270] Call Trace: [ 3760.064275] [ 3760.064281] ? show_regs.cold.9+0x22/0x2f [ 3760.064292] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.064298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.064302] ? watchdog+0x40/0x40 [ 3760.064307] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.064311] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.064319] ? apic_timer_interrupt+0xf/0x20 [ 3760.064323] [ 3760.064326] ? on_each_cpu+0x51/0x120 [ 3760.064331] ? smp_call_function_many_cond+0x3db/0x540 [ 3760.064334] ? smp_call_function_many_cond+0x3a8/0x540 [ 3760.064337] ? load_new_mm_cr3+0x120/0x120 [ 3760.064342] ? load_new_mm_cr3+0x120/0x120 [ 3760.064345] on_each_cpu+0x51/0x120 [ 3760.064349] flush_tlb_kernel_range+0x93/0xb0 [ 3760.064355] free_unmap_vmap_area+0x63/0x70 [ 3760.064360] remove_vm_area+0xe7/0x110 [ 3760.064363] __vunmap+0x9c/0x400 [ 3760.064366] vfree+0x4d/0x100 [ 3760.064370] free_work+0x34/0x50 [libcfs] [ 3760.064394] process_one_work+0x2c8/0x700 [ 3760.064402] worker_thread+0x296/0x6e0 [ 3760.064406] ? rescuer_thread+0x570/0x570 [ 3760.064409] kthread+0x1d1/0x200 [ 3760.064413] ? set_kthread_struct+0x70/0x70 [ 3760.064417] ret_from_fork+0x1f/0x30 [ 3760.309678] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.356096] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.362601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3760.367683] Workqueue: events free_work [libcfs] [ 3760.370575] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3760.373575] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3760.382878] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3760.386689] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3760.389827] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3760.392862] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3760.398988] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3760.401859] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3760.404942] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3760.408844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.411259] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3760.414228] Call Trace: [ 3760.415064] [ 3760.416060] ? show_regs.cold.9+0x22/0x2f [ 3760.418046] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.419881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.421558] ? watchdog+0x40/0x40 [ 3760.423380] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.425285] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.427394] ? apic_timer_interrupt+0xf/0x20 [ 3760.429262] [ 3760.430261] ? on_each_cpu+0x51/0x120 [ 3760.431700] ? smp_call_function_many_cond+0x3db/0x540 [ 3760.433940] ? smp_call_function_many_cond+0x3a8/0x540 [ 3760.436503] ? load_new_mm_cr3+0x120/0x120 [ 3760.438587] ? load_new_mm_cr3+0x120/0x120 [ 3760.440327] on_each_cpu+0x51/0x120 [ 3760.441945] flush_tlb_kernel_range+0x93/0xb0 [ 3760.443839] free_unmap_vmap_area+0x63/0x70 [ 3760.445506] remove_vm_area+0xe7/0x110 [ 3760.447096] __vunmap+0x9c/0x400 [ 3760.448447] vfree+0x4d/0x100 [ 3760.449664] free_work+0x34/0x50 [libcfs] [ 3760.451297] process_one_work+0x2c8/0x700 [ 3760.452927] worker_thread+0x296/0x6e0 [ 3760.454623] ? rescuer_thread+0x570/0x570 [ 3760.456364] kthread+0x1d1/0x200 [ 3760.457756] ? set_kthread_struct+0x70/0x70 [ 3760.459559] ret_from_fork+0x1f/0x30 [ 3763.167373] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1404s! [ 3763.178477] Showing busy workqueues and worker pools: [ 3763.184334] workqueue events: flags=0x0 [ 3763.186814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3763.195347] in-flight: 56293:free_work [libcfs] [ 3763.197572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3763.201504] in-flight: 22277:free_work [libcfs] [ 3763.212542] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3763.225251] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3763.229266] pending: nfs4_renew_state [nfsv4] [ 3763.234401] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3763.238518] in-flight: 47596:free_work [libcfs] [ 3763.242105] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3763.248397] workqueue events_unbound: flags=0x2 [ 3763.251250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3763.254836] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3763.262336] workqueue events_power_efficient: flags=0x80 [ 3763.266286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3763.278758] pending: neigh_periodic_work, do_cache_clean [ 3763.283057] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3763.286340] pending: check_lifetime [ 3763.295761] workqueue mm_percpu_wq: flags=0x8 [ 3763.306511] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3763.319633] pending: vmstat_update [ 3763.322695] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3763.328563] pending: vmstat_update [ 3763.332957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3763.338864] pending: vmstat_update [ 3763.344044] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3763.354653] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3763.368153] pending: cgroup_pidlist_destroy_work_fn [ 3763.375852] workqueue writeback: flags=0x4a [ 3763.381016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3763.389849] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3763.395612] workqueue xprtiod: flags=0xa [ 3763.397671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3763.401156] pending: xprt_autoclose [ 3763.404045] workqueue ipv6_addrconf: flags=0x40008 [ 3763.407247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3763.411256] pending: addrconf_verify_work [ 3763.413661] workqueue knbd0-recv: flags=0x1a [ 3763.421018] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3763.425847] in-flight: 655:recv_work [ 3763.435833] workqueue ptlrpc_pinger: flags=0x2 [ 3763.441426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3763.447012] pending: ptlrpc_pinger_main [ptlrpc] [ 3763.449303] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1405s workers=5 idle: 53684 27131 54850 46584 [ 3763.460853] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1405s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3763.472808] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1405s workers=6 idle: 53660 37674 46417 53664 46418 [ 3763.479219] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3753s workers=2 idle: 657 [ 3784.058016] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3784.060489] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3784.076674] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3784.080648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3784.083209] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3784.084997] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3784.090423] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3784.093260] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3784.098404] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3784.101057] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3784.104045] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3784.106899] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3784.109547] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3784.112936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3784.114735] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3784.117019] Call Trace: [ 3784.117803] [ 3784.118403] ? show_regs.cold.9+0x22/0x2f [ 3784.119754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3784.121264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3784.122830] ? watchdog+0x40/0x40 [ 3784.123834] ? hrtimer_interrupt+0x128/0x2c0 [ 3784.125147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3784.126651] ? apic_timer_interrupt+0xf/0x20 [ 3784.127977] [ 3784.128637] ? on_each_cpu_cond_mask+0x46/0x150 [ 3784.129993] ? __x64_sys_fspick+0x210/0x210 [ 3784.131549] ? smp_call_function_many_cond+0x3d9/0x540 [ 3784.133611] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3784.135083] ? __x64_sys_fspick+0x210/0x210 [ 3784.136345] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3784.137790] on_each_cpu_cond_mask+0x46/0x150 [ 3784.139117] on_each_cpu_cond+0x1c/0x30 [ 3784.140274] invalidate_bh_lrus+0x27/0x30 [ 3784.141630] invalidate_bdev+0x5a/0x90 [ 3784.142762] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3784.144419] generic_shutdown_super+0xb1/0x1b0 [ 3784.145876] kill_block_super+0x29/0x80 [ 3784.147553] deactivate_locked_super+0x52/0xd0 [ 3784.149085] deactivate_super+0x83/0x90 [ 3784.150339] cleanup_mnt+0x5f/0xe0 [ 3784.151544] __cleanup_mnt+0x16/0x20 [ 3784.152878] task_work_run+0xc6/0x110 [ 3784.154080] exit_to_usermode_loop+0x1dd/0x1f0 [ 3784.155892] do_syscall_64+0x3d6/0x440 [ 3784.157045] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3784.158889] RIP: 0033:0x7fbb019648fb [ 3784.160095] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3784.166011] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3784.168307] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3784.170701] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3784.172910] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3784.175010] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3784.177179] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3788.055048] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3788.061449] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3788.066954] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [ 3788.073556] Modules linked in: [ 3788.073558] intel_rapl_msr intel_rapl_common sb_edac rapl [ 3788.097804] zfs(O) [ 3788.098673] pcspkr [ 3788.106781] spl(O) [ 3788.107734] i2c_piix4 [ 3788.108453] lustre(O) [ 3788.109294] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3788.109318] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3788.109324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3788.109331] Workqueue: events free_work [libcfs] [ 3788.109440] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3788.109449] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3788.109453] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3788.109458] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3788.109460] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3788.109462] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3788.109464] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3788.109467] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3788.109470] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3788.109473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3788.109475] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3788.109483] Call Trace: [ 3788.109488] [ 3788.109493] ? show_regs.cold.9+0x22/0x2f [ 3788.109502] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3788.109508] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3788.109512] ? watchdog+0x40/0x40 [ 3788.109517] ? hrtimer_interrupt+0x128/0x2c0 [ 3788.109521] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3788.109530] ? apic_timer_interrupt+0xf/0x20 [ 3788.109534] [ 3788.109536] ? on_each_cpu+0x51/0x120 [ 3788.109540] ? smp_call_function_many_cond+0x3db/0x540 [ 3788.109544] ? smp_call_function_many_cond+0x3a8/0x540 [ 3788.109548] ? load_new_mm_cr3+0x120/0x120 [ 3788.109553] ? load_new_mm_cr3+0x120/0x120 [ 3788.109557] on_each_cpu+0x51/0x120 [ 3788.109561] flush_tlb_kernel_range+0x93/0xb0 [ 3788.109567] free_unmap_vmap_area+0x63/0x70 [ 3788.109586] remove_vm_area+0xe7/0x110 [ 3788.109590] __vunmap+0x9c/0x400 [ 3788.109594] vfree+0x4d/0x100 [ 3788.109597] free_work+0x34/0x50 [libcfs] [ 3788.109623] process_one_work+0x2c8/0x700 [ 3788.109630] worker_thread+0x296/0x6e0 [ 3788.109635] ? rescuer_thread+0x570/0x570 [ 3788.109638] kthread+0x1d1/0x200 [ 3788.109642] ? set_kthread_struct+0x70/0x70 [ 3788.109645] ret_from_fork+0x1f/0x30 [ 3788.321481] osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3788.363306] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3788.375121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3788.381541] Workqueue: events free_work [libcfs] [ 3788.385617] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3788.388732] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3788.402301] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3788.412474] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3788.422848] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3788.426815] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3788.430800] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3788.440237] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3788.445689] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3788.453687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3788.460662] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3788.464179] Call Trace: [ 3788.465932] [ 3788.466658] ? show_regs.cold.9+0x22/0x2f [ 3788.468013] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3788.470285] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3788.475576] ? watchdog+0x40/0x40 [ 3788.480023] ? hrtimer_interrupt+0x128/0x2c0 [ 3788.482015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3788.485938] ? apic_timer_interrupt+0xf/0x20 [ 3788.487826] [ 3788.488373] ? on_each_cpu+0x51/0x120 [ 3788.490222] ? smp_call_function_many_cond+0x3db/0x540 [ 3788.492611] ? smp_call_function_many_cond+0x3a8/0x540 [ 3788.499829] ? load_new_mm_cr3+0x120/0x120 [ 3788.503828] ? load_new_mm_cr3+0x120/0x120 [ 3788.511265] on_each_cpu+0x51/0x120 [ 3788.512469] flush_tlb_kernel_range+0x93/0xb0 [ 3788.515627] free_unmap_vmap_area+0x63/0x70 [ 3788.517787] remove_vm_area+0xe7/0x110 [ 3788.519886] __vunmap+0x9c/0x400 [ 3788.521787] vfree+0x4d/0x100 [ 3788.523599] free_work+0x34/0x50 [libcfs] [ 3788.527145] process_one_work+0x2c8/0x700 [ 3788.529339] worker_thread+0x296/0x6e0 [ 3788.530747] ? rescuer_thread+0x570/0x570 [ 3788.533911] kthread+0x1d1/0x200 [ 3788.536151] ? set_kthread_struct+0x70/0x70 [ 3788.538544] ret_from_fork+0x1f/0x30 [ 3793.887105] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1435s! [ 3793.891767] Showing busy workqueues and worker pools: [ 3793.897668] workqueue events: flags=0x0 [ 3793.898697] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3793.900347] in-flight: 56293:free_work [libcfs] [ 3793.909198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3793.917997] in-flight: 22277:free_work [libcfs] [ 3793.923381] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3793.928452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3793.935622] pending: nfs4_renew_state [nfsv4] [ 3793.937993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3793.947531] in-flight: 47596:free_work [libcfs] [ 3793.953968] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3793.962558] workqueue events_unbound: flags=0x2 [ 3793.965605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3793.970549] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3793.975654] workqueue events_power_efficient: flags=0x80 [ 3793.977849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3793.981817] pending: neigh_periodic_work, do_cache_clean [ 3793.985471] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3793.995860] pending: check_lifetime [ 3793.999070] workqueue mm_percpu_wq: flags=0x8 [ 3794.003702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3794.008688] pending: vmstat_update [ 3794.011400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3794.024085] pending: vmstat_update [ 3794.031284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3794.044992] pending: vmstat_update [ 3794.048605] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3794.054894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3794.061595] pending: cgroup_pidlist_destroy_work_fn [ 3794.069554] workqueue writeback: flags=0x4a [ 3794.075892] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3794.079429] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3794.088415] workqueue xprtiod: flags=0xa [ 3794.094541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3794.102168] pending: xprt_autoclose [ 3794.107846] workqueue ipv6_addrconf: flags=0x40008 [ 3794.122182] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3794.124495] pending: addrconf_verify_work [ 3794.130666] workqueue knbd0-recv: flags=0x1a [ 3794.132012] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3794.142635] in-flight: 655:recv_work [ 3794.146697] workqueue ptlrpc_pinger: flags=0x2 [ 3794.148176] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3794.153207] pending: ptlrpc_pinger_main [ptlrpc] [ 3794.159077] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1436s workers=5 idle: 53684 27131 54850 46584 [ 3794.166677] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1436s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3794.177483] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1436s workers=6 idle: 53660 37674 46417 53664 46418 [ 3794.186400] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3783s workers=2 idle: 657 [ 3812.058213] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3812.067019] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3812.142359] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3812.152799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3812.166815] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3812.171475] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3812.205746] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3812.218784] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3812.229454] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3812.236553] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3812.247476] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3812.250249] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3812.262127] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3812.274148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3812.281076] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3812.288261] Call Trace: [ 3812.289077] [ 3812.289839] ? show_regs.cold.9+0x22/0x2f [ 3812.298089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3812.303943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3812.305716] ? watchdog+0x40/0x40 [ 3812.306974] ? hrtimer_interrupt+0x128/0x2c0 [ 3812.315262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3812.316537] ? apic_timer_interrupt+0xf/0x20 [ 3812.325255] [ 3812.329807] ? on_each_cpu_cond_mask+0x46/0x150 [ 3812.334614] ? __x64_sys_fspick+0x210/0x210 [ 3812.336983] ? smp_call_function_many_cond+0x3db/0x540 [ 3812.347548] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3812.349040] ? __x64_sys_fspick+0x210/0x210 [ 3812.360001] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3812.366451] on_each_cpu_cond_mask+0x46/0x150 [ 3812.368885] on_each_cpu_cond+0x1c/0x30 [ 3812.374612] invalidate_bh_lrus+0x27/0x30 [ 3812.381325] invalidate_bdev+0x5a/0x90 [ 3812.383449] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3812.391451] generic_shutdown_super+0xb1/0x1b0 [ 3812.395551] kill_block_super+0x29/0x80 [ 3812.400789] deactivate_locked_super+0x52/0xd0 [ 3812.408486] deactivate_super+0x83/0x90 [ 3812.410732] cleanup_mnt+0x5f/0xe0 [ 3812.412592] __cleanup_mnt+0x16/0x20 [ 3812.413960] task_work_run+0xc6/0x110 [ 3812.423955] exit_to_usermode_loop+0x1dd/0x1f0 [ 3812.426688] do_syscall_64+0x3d6/0x440 [ 3812.432238] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3812.436283] RIP: 0033:0x7fbb019648fb [ 3812.442989] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3812.459032] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3812.463992] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3812.470980] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3812.475167] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3812.479584] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3812.484194] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3816.060057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3816.067557] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3816.106266] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3816.115706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3816.121409] Workqueue: events free_work [libcfs] [ 3816.122730] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3816.127701] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3816.147103] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3816.152325] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3816.158706] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3816.164566] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3816.176180] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3816.183512] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3816.189574] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3816.198887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3816.202587] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3816.211446] Call Trace: [ 3816.216038] [ 3816.219216] ? show_regs.cold.9+0x22/0x2f [ 3816.220718] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3816.229075] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3816.237735] ? watchdog+0x40/0x40 [ 3816.245323] ? hrtimer_interrupt+0x128/0x2c0 [ 3816.249422] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3816.258049] ? apic_timer_interrupt+0xf/0x20 [ 3816.261896] [ 3816.266047] ? on_each_cpu+0x51/0x120 [ 3816.274412] ? smp_call_function_many_cond+0x3db/0x540 [ 3816.276025] ? smp_call_function_many_cond+0x3a8/0x540 [ 3816.285136] ? load_new_mm_cr3+0x120/0x120 [ 3816.295015] ? load_new_mm_cr3+0x120/0x120 [ 3816.297938] on_each_cpu+0x51/0x120 [ 3816.299635] flush_tlb_kernel_range+0x93/0xb0 [ 3816.303061] free_unmap_vmap_area+0x63/0x70 [ 3816.305170] remove_vm_area+0xe7/0x110 [ 3816.309614] __vunmap+0x9c/0x400 [ 3816.315421] vfree+0x4d/0x100 [ 3816.317083] free_work+0x34/0x50 [libcfs] [ 3816.328059] process_one_work+0x2c8/0x700 [ 3816.334390] worker_thread+0x296/0x6e0 [ 3816.336138] ? rescuer_thread+0x570/0x570 [ 3816.338567] kthread+0x1d1/0x200 [ 3816.340490] ? set_kthread_struct+0x70/0x70 [ 3816.353944] ret_from_fork+0x1f/0x30 [ 3820.056482] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3820.060789] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3820.154691] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3820.162958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3820.172001] Workqueue: events free_work [libcfs] [ 3820.175575] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3820.182291] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3820.195996] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3820.200905] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3820.208512] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3820.220225] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3820.225152] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3820.229106] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3820.232552] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3820.237459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3820.240708] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3820.246835] Call Trace: [ 3820.247715] [ 3820.249386] ? show_regs.cold.9+0x22/0x2f [ 3820.251300] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3820.255467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3820.259681] ? watchdog+0x40/0x40 [ 3820.260946] ? hrtimer_interrupt+0x128/0x2c0 [ 3820.262504] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3820.271608] ? apic_timer_interrupt+0xf/0x20 [ 3820.275531] [ 3820.276486] ? on_each_cpu+0x51/0x120 [ 3820.279564] ? smp_call_function_many_cond+0x3db/0x540 [ 3820.289365] ? smp_call_function_many_cond+0x3a8/0x540 [ 3820.294155] ? load_new_mm_cr3+0x120/0x120 [ 3820.295640] ? load_new_mm_cr3+0x120/0x120 [ 3820.305699] on_each_cpu+0x51/0x120 [ 3820.306994] flush_tlb_kernel_range+0x93/0xb0 [ 3820.310274] free_unmap_vmap_area+0x63/0x70 [ 3820.313757] remove_vm_area+0xe7/0x110 [ 3820.324835] __vunmap+0x9c/0x400 [ 3820.328037] vfree+0x4d/0x100 [ 3820.329215] free_work+0x34/0x50 [libcfs] [ 3820.344639] process_one_work+0x2c8/0x700 [ 3820.348012] worker_thread+0x296/0x6e0 [ 3820.357910] ? rescuer_thread+0x570/0x570 [ 3820.365891] kthread+0x1d1/0x200 [ 3820.370415] ? set_kthread_struct+0x70/0x70 [ 3820.373271] ret_from_fork+0x1f/0x30 [ 3824.607026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1466s! [ 3824.610884] Showing busy workqueues and worker pools: [ 3824.613242] workqueue events: flags=0x0 [ 3824.614702] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3824.617264] in-flight: 56293:free_work [libcfs] [ 3824.618966] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3824.621879] in-flight: 22277:free_work [libcfs] [ 3824.624723] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3824.634839] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3824.640851] pending: nfs4_renew_state [nfsv4] [ 3824.643767] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3824.653886] in-flight: 47596:free_work [libcfs] [ 3824.656369] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3824.673137] workqueue events_unbound: flags=0x2 [ 3824.674948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3824.681094] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3824.687231] workqueue events_power_efficient: flags=0x80 [ 3824.696992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3824.705162] pending: neigh_periodic_work, do_cache_clean [ 3824.707453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3824.712992] pending: check_lifetime [ 3824.717259] workqueue mm_percpu_wq: flags=0x8 [ 3824.720067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3824.723925] pending: vmstat_update [ 3824.728484] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3824.733821] pending: vmstat_update [ 3824.735873] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3824.740252] pending: vmstat_update [ 3824.743320] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3824.749443] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3824.755437] pending: cgroup_pidlist_destroy_work_fn [ 3824.761481] workqueue writeback: flags=0x4a [ 3824.764152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3824.766600] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3824.785445] workqueue xprtiod: flags=0xa [ 3824.791887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3824.796251] pending: xprt_autoclose [ 3824.798296] workqueue ipv6_addrconf: flags=0x40008 [ 3824.804137] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3824.809891] pending: addrconf_verify_work [ 3824.811558] workqueue knbd0-recv: flags=0x1a [ 3824.813816] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3824.821126] in-flight: 655:recv_work [ 3824.823422] workqueue ptlrpc_pinger: flags=0x2 [ 3824.826051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3824.835990] pending: ptlrpc_pinger_main [ptlrpc] [ 3824.842795] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1467s workers=5 idle: 53684 27131 54850 46584 [ 3824.853529] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1467s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3824.863115] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1467s workers=6 idle: 53660 37674 46417 53664 46418 [ 3824.867372] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3814s workers=2 idle: 657 [ 3840.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 3840.061117] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3840.077570] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3840.080491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3840.082786] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3840.084323] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3840.089528] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3840.092365] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3840.094130] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3840.096501] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3840.098955] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3840.101307] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3840.103004] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3840.105023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3840.106567] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3840.109439] Call Trace: [ 3840.110502] [ 3840.111074] ? show_regs.cold.9+0x22/0x2f [ 3840.112130] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3840.113513] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3840.114584] ? watchdog+0x40/0x40 [ 3840.115438] ? hrtimer_interrupt+0x128/0x2c0 [ 3840.116523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3840.117708] ? apic_timer_interrupt+0xf/0x20 [ 3840.118827] [ 3840.119486] ? on_each_cpu_cond_mask+0x46/0x150 [ 3840.120627] ? __x64_sys_fspick+0x210/0x210 [ 3840.121646] ? smp_call_function_many_cond+0x3db/0x540 [ 3840.122952] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3840.124160] ? __x64_sys_fspick+0x210/0x210 [ 3840.125390] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3840.126842] on_each_cpu_cond_mask+0x46/0x150 [ 3840.128788] on_each_cpu_cond+0x1c/0x30 [ 3840.130415] invalidate_bh_lrus+0x27/0x30 [ 3840.132058] invalidate_bdev+0x5a/0x90 [ 3840.133625] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3840.135844] generic_shutdown_super+0xb1/0x1b0 [ 3840.137798] kill_block_super+0x29/0x80 [ 3840.139381] deactivate_locked_super+0x52/0xd0 [ 3840.141279] deactivate_super+0x83/0x90 [ 3840.143102] cleanup_mnt+0x5f/0xe0 [ 3840.144654] __cleanup_mnt+0x16/0x20 [ 3840.146116] task_work_run+0xc6/0x110 [ 3840.147788] exit_to_usermode_loop+0x1dd/0x1f0 [ 3840.149666] do_syscall_64+0x3d6/0x440 [ 3840.151344] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3840.153485] RIP: 0033:0x7fbb019648fb [ 3840.154868] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3840.160943] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3840.165185] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3840.169919] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3840.173988] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3840.177171] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3840.182401] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3844.059070] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3844.062592] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3844.085651] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3844.091349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3844.094957] Workqueue: events free_work [libcfs] [ 3844.097028] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3844.100095] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3844.109526] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3844.112990] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3844.116663] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3844.119930] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3844.123062] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3844.126437] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3844.129755] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3844.133716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3844.136445] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3844.139255] Call Trace: [ 3844.140563] [ 3844.141548] ? show_regs.cold.9+0x22/0x2f [ 3844.143227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3844.145560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3844.147607] ? watchdog+0x40/0x40 [ 3844.149109] ? hrtimer_interrupt+0x128/0x2c0 [ 3844.151053] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3844.153636] ? apic_timer_interrupt+0xf/0x20 [ 3844.155584] [ 3844.156579] ? on_each_cpu+0x51/0x120 [ 3844.158376] ? smp_call_function_many_cond+0x3db/0x540 [ 3844.160718] ? smp_call_function_many_cond+0x3a8/0x540 [ 3844.162846] ? load_new_mm_cr3+0x120/0x120 [ 3844.164872] ? load_new_mm_cr3+0x120/0x120 [ 3844.166972] on_each_cpu+0x51/0x120 [ 3844.168646] flush_tlb_kernel_range+0x93/0xb0 [ 3844.170650] free_unmap_vmap_area+0x63/0x70 [ 3844.172432] remove_vm_area+0xe7/0x110 [ 3844.174072] __vunmap+0x9c/0x400 [ 3844.175542] vfree+0x4d/0x100 [ 3844.176746] free_work+0x34/0x50 [libcfs] [ 3844.178521] process_one_work+0x2c8/0x700 [ 3844.180283] worker_thread+0x296/0x6e0 [ 3844.181836] ? rescuer_thread+0x570/0x570 [ 3844.183816] kthread+0x1d1/0x200 [ 3844.185304] ? set_kthread_struct+0x70/0x70 [ 3844.187487] ret_from_fork+0x1f/0x30 [ 3852.055005] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3852.062406] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3852.087123] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3852.093430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3852.098817] Workqueue: events free_work [libcfs] [ 3852.101591] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3852.104214] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3852.114810] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3852.119469] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3852.124953] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3852.129771] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3852.132477] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3852.137855] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3852.142436] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3852.146734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3852.149225] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3852.154210] Call Trace: [ 3852.156912] [ 3852.158081] ? show_regs.cold.9+0x22/0x2f [ 3852.160320] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3852.163559] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3852.167762] ? watchdog+0x40/0x40 [ 3852.169806] ? hrtimer_interrupt+0x128/0x2c0 [ 3852.174293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3852.177092] ? apic_timer_interrupt+0xf/0x20 [ 3852.179960] [ 3852.181048] ? on_each_cpu+0x51/0x120 [ 3852.182518] ? smp_call_function_many_cond+0x3d9/0x540 [ 3852.190253] ? smp_call_function_many_cond+0x3a8/0x540 [ 3852.198328] ? load_new_mm_cr3+0x120/0x120 [ 3852.202692] ? load_new_mm_cr3+0x120/0x120 [ 3852.205583] on_each_cpu+0x51/0x120 [ 3852.208236] flush_tlb_kernel_range+0x93/0xb0 [ 3852.210012] free_unmap_vmap_area+0x63/0x70 [ 3852.211651] remove_vm_area+0xe7/0x110 [ 3852.215177] __vunmap+0x9c/0x400 [ 3852.217740] vfree+0x4d/0x100 [ 3852.220855] free_work+0x34/0x50 [libcfs] [ 3852.229023] process_one_work+0x2c8/0x700 [ 3852.231819] worker_thread+0x296/0x6e0 [ 3852.232999] ? rescuer_thread+0x570/0x570 [ 3852.241114] kthread+0x1d1/0x200 [ 3852.243916] ? set_kthread_struct+0x70/0x70 [ 3852.248335] ret_from_fork+0x1f/0x30 [ 3855.328093] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1497s! [ 3855.333802] Showing busy workqueues and worker pools: [ 3855.337222] workqueue events: flags=0x0 [ 3855.342515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.346374] in-flight: 56293:free_work [libcfs] [ 3855.350699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3855.354715] in-flight: 22277:free_work [libcfs] [ 3855.357496] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3855.362544] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.371371] pending: nfs4_renew_state [nfsv4] [ 3855.375229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3855.379569] in-flight: 47596:free_work [libcfs] [ 3855.383642] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3855.389438] workqueue events_unbound: flags=0x2 [ 3855.392823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3855.397195] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3855.402207] workqueue events_power_efficient: flags=0x80 [ 3855.405570] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3855.410354] pending: neigh_periodic_work, do_cache_clean [ 3855.413765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.417450] pending: check_lifetime [ 3855.419030] workqueue mm_percpu_wq: flags=0x8 [ 3855.422631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.426341] pending: vmstat_update [ 3855.428378] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.432459] pending: vmstat_update [ 3855.434423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.438521] pending: vmstat_update [ 3855.440646] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3855.443689] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3855.448325] pending: cgroup_pidlist_destroy_work_fn [ 3855.453784] workqueue writeback: flags=0x4a [ 3855.457363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3855.463430] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3855.468189] workqueue xprtiod: flags=0xa [ 3855.471100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3855.474768] pending: xprt_autoclose [ 3855.477526] workqueue ipv6_addrconf: flags=0x40008 [ 3855.479962] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3855.484359] pending: addrconf_verify_work [ 3855.489711] workqueue knbd0-recv: flags=0x1a [ 3855.495608] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3855.501515] in-flight: 655:recv_work [ 3855.508794] workqueue ptlrpc_pinger: flags=0x2 [ 3855.512815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3855.517678] pending: ptlrpc_pinger_main [ptlrpc] [ 3855.523779] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1497s workers=5 idle: 53684 27131 54850 46584 [ 3855.530749] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1497s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3855.536626] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1497s workers=6 idle: 53660 37674 46417 53664 46418 [ 3855.543631] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3845s workers=2 idle: 657 [ 3857.773573] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 3857.776566] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 3857.786449] (detected by 2, t=1500043 jiffies, g=626849, q=1398) [ 3857.796844] Sending NMI from CPU 2 to CPUs 3: [ 3857.809152] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 3857.809164] rcu: rcu_sched kthread starved for 1499685 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3857.832689] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3857.837630] rcu: RCU grace-period kthread stack dump: [ 3857.843176] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3857.859084] Call Trace: [ 3857.865514] __schedule+0x351/0xcb0 [ 3857.867721] schedule+0xc0/0x180 [ 3857.870215] schedule_timeout+0xb4/0x190 [ 3857.878670] ? do_raw_spin_unlock+0x75/0x190 [ 3857.883181] ? __next_timer_interrupt+0x160/0x160 [ 3857.886987] ? prepare_to_swait_event+0xf3/0x140 [ 3857.890951] rcu_gp_kthread+0x8dd/0x15b0 [ 3857.895820] ? force_qs_rnp+0x2c0/0x2c0 [ 3857.899421] kthread+0x1d1/0x200 [ 3857.901908] ? set_kthread_struct+0x70/0x70 [ 3857.903912] ret_from_fork+0x1f/0x30 [ 3857.906746] rcu: Stack dump where RCU GP kthread last ran: [ 3857.913164] NMI backtrace for cpu 2 [ 3857.916820] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3857.929112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3857.937997] Workqueue: events free_work [libcfs] [ 3857.945549] Call Trace: [ 3857.947499] [ 3857.951379] ? dump_stack+0xbb/0x10e [ 3857.953514] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3857.956745] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3857.959632] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3857.964757] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3857.971639] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3857.976649] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3857.982605] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 3857.989325] ? update_process_times+0x6d/0xc0 [ 3857.991694] ? tick_sched_handle.isra.18+0x27/0x90 [ 3857.994665] ? tick_sched_timer+0x53/0xb0 [ 3857.996158] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3858.001484] ? tick_sched_handle.isra.18+0x90/0x90 [ 3858.003284] ? hrtimer_interrupt+0x128/0x2c0 [ 3858.008637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3858.012171] ? apic_timer_interrupt+0xf/0x20 [ 3858.014327] [ 3858.015408] ? on_each_cpu+0x51/0x120 [ 3858.019576] ? smp_call_function_many_cond+0x3db/0x540 [ 3858.022119] ? smp_call_function_many_cond+0x3a8/0x540 [ 3858.024503] ? load_new_mm_cr3+0x120/0x120 [ 3858.031294] ? load_new_mm_cr3+0x120/0x120 [ 3858.032881] ? on_each_cpu+0x51/0x120 [ 3858.038567] ? flush_tlb_kernel_range+0x93/0xb0 [ 3858.041728] ? free_unmap_vmap_area+0x63/0x70 [ 3858.049535] ? remove_vm_area+0xe7/0x110 [ 3858.051998] ? __vunmap+0x9c/0x400 [ 3858.053732] ? vfree+0x4d/0x100 [ 3858.055649] ? free_work+0x34/0x50 [libcfs] [ 3858.058744] ? process_one_work+0x2c8/0x700 [ 3858.061242] ? worker_thread+0x296/0x6e0 [ 3858.063951] ? rescuer_thread+0x570/0x570 [ 3858.065542] ? kthread+0x1d1/0x200 [ 3858.070357] ? set_kthread_struct+0x70/0x70 [ 3858.072767] ? ret_from_fork+0x1f/0x30 [ 3868.058626] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3868.062288] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3868.120535] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3868.135860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3868.152956] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3868.158286] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3868.183081] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3868.195414] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3868.208783] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3868.212112] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3868.215569] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3868.219255] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3868.225041] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3868.231539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3868.237790] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3868.243284] Call Trace: [ 3868.245384] [ 3868.246519] ? show_regs.cold.9+0x22/0x2f [ 3868.250555] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3868.254596] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3868.258036] ? watchdog+0x40/0x40 [ 3868.260068] ? hrtimer_interrupt+0x128/0x2c0 [ 3868.265301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3868.269153] ? apic_timer_interrupt+0xf/0x20 [ 3868.272834] [ 3868.274004] ? on_each_cpu_cond_mask+0x46/0x150 [ 3868.278813] ? __x64_sys_fspick+0x210/0x210 [ 3868.281988] ? smp_call_function_many_cond+0x3db/0x540 [ 3868.286191] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3868.290182] ? __x64_sys_fspick+0x210/0x210 [ 3868.293372] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3868.296960] on_each_cpu_cond_mask+0x46/0x150 [ 3868.300155] on_each_cpu_cond+0x1c/0x30 [ 3868.303134] invalidate_bh_lrus+0x27/0x30 [ 3868.306462] invalidate_bdev+0x5a/0x90 [ 3868.308937] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3868.313320] generic_shutdown_super+0xb1/0x1b0 [ 3868.316692] kill_block_super+0x29/0x80 [ 3868.318032] deactivate_locked_super+0x52/0xd0 [ 3868.319717] deactivate_super+0x83/0x90 [ 3868.324878] cleanup_mnt+0x5f/0xe0 [ 3868.328226] __cleanup_mnt+0x16/0x20 [ 3868.330815] task_work_run+0xc6/0x110 [ 3868.332287] exit_to_usermode_loop+0x1dd/0x1f0 [ 3868.335864] do_syscall_64+0x3d6/0x440 [ 3868.339871] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3868.343875] RIP: 0033:0x7fbb019648fb [ 3868.345876] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3868.355401] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3868.359758] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3868.363041] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3868.366826] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3868.370972] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3868.375069] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3880.055241] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3880.063327] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3880.129329] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3880.142298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3880.153598] Workqueue: events free_work [libcfs] [ 3880.157537] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3880.162434] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3880.175127] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3880.179317] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3880.183899] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3880.188255] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3880.192799] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3880.198478] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3880.207826] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3880.220028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3880.230184] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3880.248380] Call Trace: [ 3880.252296] [ 3880.254081] ? show_regs.cold.9+0x22/0x2f [ 3880.255636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3880.267765] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3880.270923] ? watchdog+0x40/0x40 [ 3880.273769] ? hrtimer_interrupt+0x128/0x2c0 [ 3880.279256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3880.284509] ? apic_timer_interrupt+0xf/0x20 [ 3880.294474] [ 3880.298244] ? on_each_cpu+0x51/0x120 [ 3880.303673] ? smp_call_function_many_cond+0x3db/0x540 [ 3880.308893] ? smp_call_function_many_cond+0x3a8/0x540 [ 3880.317351] ? load_new_mm_cr3+0x120/0x120 [ 3880.323342] ? load_new_mm_cr3+0x120/0x120 [ 3880.329335] on_each_cpu+0x51/0x120 [ 3880.335676] flush_tlb_kernel_range+0x93/0xb0 [ 3880.343914] free_unmap_vmap_area+0x63/0x70 [ 3880.346353] remove_vm_area+0xe7/0x110 [ 3880.349720] __vunmap+0x9c/0x400 [ 3880.355151] vfree+0x4d/0x100 [ 3880.356757] free_work+0x34/0x50 [libcfs] [ 3880.360918] process_one_work+0x2c8/0x700 [ 3880.364318] worker_thread+0x296/0x6e0 [ 3880.368797] ? rescuer_thread+0x570/0x570 [ 3880.370417] kthread+0x1d1/0x200 [ 3880.372458] ? set_kthread_struct+0x70/0x70 [ 3880.374687] ret_from_fork+0x1f/0x30 [ 3884.059595] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 3884.066588] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3884.106765] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3884.114130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3884.120587] Workqueue: events free_work [libcfs] [ 3884.123877] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3884.127550] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3884.145211] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3884.148701] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3884.156093] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3884.160764] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3884.167780] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3884.177923] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3884.184407] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3884.187889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3884.189958] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3884.194527] Call Trace: [ 3884.197011] [ 3884.198874] ? show_regs.cold.9+0x22/0x2f [ 3884.202254] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3884.205199] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3884.208767] ? watchdog+0x40/0x40 [ 3884.210741] ? hrtimer_interrupt+0x128/0x2c0 [ 3884.214636] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3884.220865] ? apic_timer_interrupt+0xf/0x20 [ 3884.227179] [ 3884.227925] ? on_each_cpu+0x51/0x120 [ 3884.231622] ? smp_call_function_many_cond+0x3db/0x540 [ 3884.236875] ? smp_call_function_many_cond+0x3a8/0x540 [ 3884.247824] ? load_new_mm_cr3+0x120/0x120 [ 3884.252950] ? load_new_mm_cr3+0x120/0x120 [ 3884.254592] on_each_cpu+0x51/0x120 [ 3884.259181] flush_tlb_kernel_range+0x93/0xb0 [ 3884.261633] free_unmap_vmap_area+0x63/0x70 [ 3884.264442] remove_vm_area+0xe7/0x110 [ 3884.267690] __vunmap+0x9c/0x400 [ 3884.270548] vfree+0x4d/0x100 [ 3884.272407] free_work+0x34/0x50 [libcfs] [ 3884.274587] process_one_work+0x2c8/0x700 [ 3884.276597] worker_thread+0x296/0x6e0 [ 3884.278481] ? rescuer_thread+0x570/0x570 [ 3884.281023] kthread+0x1d1/0x200 [ 3884.284859] ? set_kthread_struct+0x70/0x70 [ 3884.286436] ret_from_fork+0x1f/0x30 [ 3886.052136] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1527s! [ 3886.057759] Showing busy workqueues and worker pools: [ 3886.060038] workqueue events: flags=0x0 [ 3886.061143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.065801] in-flight: 56293:free_work [libcfs] [ 3886.070693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3886.074254] in-flight: 22277:free_work [libcfs] [ 3886.076926] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3886.082053] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.086174] pending: nfs4_renew_state [nfsv4] [ 3886.089530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3886.092306] in-flight: 47596:free_work [libcfs] [ 3886.093941] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3886.105717] workqueue events_unbound: flags=0x2 [ 3886.108932] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3886.112795] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3886.117736] workqueue events_power_efficient: flags=0x80 [ 3886.120216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3886.123955] pending: neigh_periodic_work, do_cache_clean [ 3886.127113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.134077] pending: check_lifetime [ 3886.138198] workqueue mm_percpu_wq: flags=0x8 [ 3886.140493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.142977] pending: vmstat_update [ 3886.150723] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.156044] pending: vmstat_update [ 3886.159360] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.164930] pending: vmstat_update [ 3886.166933] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3886.170628] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3886.174826] pending: cgroup_pidlist_destroy_work_fn [ 3886.177222] workqueue writeback: flags=0x4a [ 3886.179726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3886.183117] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3886.187894] workqueue xprtiod: flags=0xa [ 3886.190205] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3886.194174] pending: xprt_autoclose [ 3886.196183] workqueue ipv6_addrconf: flags=0x40008 [ 3886.198982] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3886.203982] pending: addrconf_verify_work [ 3886.206839] workqueue knbd0-recv: flags=0x1a [ 3886.209228] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3886.213527] in-flight: 655:recv_work [ 3886.215828] workqueue ptlrpc_pinger: flags=0x2 [ 3886.218167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3886.221871] pending: ptlrpc_pinger_main [ptlrpc] [ 3886.225189] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1528s workers=5 idle: 53684 27131 54850 46584 [ 3886.230978] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1528s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3886.237912] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1528s workers=6 idle: 53660 37674 46417 53664 46418 [ 3886.248192] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3876s workers=2 idle: 657 [ 3896.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3896.061622] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3896.093005] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3896.103519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3896.112636] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3896.116928] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3896.134954] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3896.143820] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3896.146657] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3896.153742] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3896.159482] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3896.162727] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3896.166477] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3896.170191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3896.172796] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3896.175921] Call Trace: [ 3896.176968] [ 3896.177885] ? show_regs.cold.9+0x22/0x2f [ 3896.179760] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3896.181790] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3896.185131] ? watchdog+0x40/0x40 [ 3896.188265] ? hrtimer_interrupt+0x128/0x2c0 [ 3896.192354] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3896.195029] ? apic_timer_interrupt+0xf/0x20 [ 3896.196546] [ 3896.197600] ? on_each_cpu_cond_mask+0x46/0x150 [ 3896.199569] ? __x64_sys_fspick+0x210/0x210 [ 3896.202745] ? smp_call_function_many_cond+0x3db/0x540 [ 3896.205037] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3896.207960] ? __x64_sys_fspick+0x210/0x210 [ 3896.209699] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3896.212787] on_each_cpu_cond_mask+0x46/0x150 [ 3896.214889] on_each_cpu_cond+0x1c/0x30 [ 3896.216459] invalidate_bh_lrus+0x27/0x30 [ 3896.218334] invalidate_bdev+0x5a/0x90 [ 3896.219886] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3896.222146] generic_shutdown_super+0xb1/0x1b0 [ 3896.224213] kill_block_super+0x29/0x80 [ 3896.226005] deactivate_locked_super+0x52/0xd0 [ 3896.227950] deactivate_super+0x83/0x90 [ 3896.229718] cleanup_mnt+0x5f/0xe0 [ 3896.231425] __cleanup_mnt+0x16/0x20 [ 3896.236472] task_work_run+0xc6/0x110 [ 3896.239326] exit_to_usermode_loop+0x1dd/0x1f0 [ 3896.245757] do_syscall_64+0x3d6/0x440 [ 3896.247121] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3896.251363] RIP: 0033:0x7fbb019648fb [ 3896.252825] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3896.265368] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3896.269899] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3896.273651] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3896.278401] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3896.282329] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3896.288846] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3912.055015] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 3912.059997] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.065586] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3912.089242] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.089256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3912.089264] Workqueue: events free_work [libcfs] [ 3912.098242] Modules linked in: [ 3912.105078] [ 3912.105088] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3912.105107] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3912.105111] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3912.105118] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3912.105120] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3912.105123] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3912.105125] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3912.105127] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3912.105129] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3912.105132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.105135] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3912.105144] Call Trace: [ 3912.105150] [ 3912.105158] ? show_regs.cold.9+0x22/0x2f [ 3912.105168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.105175] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.105181] ? watchdog+0x40/0x40 [ 3912.105186] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.105190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.105200] ? apic_timer_interrupt+0xf/0x20 [ 3912.105205] [ 3912.105208] ? on_each_cpu+0x51/0x120 [ 3912.105211] ? smp_call_function_many_cond+0x3db/0x540 [ 3912.105215] ? smp_call_function_many_cond+0x3a8/0x540 [ 3912.105218] ? load_new_mm_cr3+0x120/0x120 [ 3912.105223] ? load_new_mm_cr3+0x120/0x120 [ 3912.105226] on_each_cpu+0x51/0x120 [ 3912.105230] flush_tlb_kernel_range+0x93/0xb0 [ 3912.105235] free_unmap_vmap_area+0x63/0x70 [ 3912.105240] remove_vm_area+0xe7/0x110 [ 3912.105243] __vunmap+0x9c/0x400 [ 3912.105246] vfree+0x4d/0x100 [ 3912.105253] free_work+0x34/0x50 [libcfs] [ 3912.105289] process_one_work+0x2c8/0x700 [ 3912.105296] worker_thread+0x296/0x6e0 [ 3912.105302] ? rescuer_thread+0x570/0x570 [ 3912.105306] kthread+0x1d1/0x200 [ 3912.105311] ? set_kthread_struct+0x70/0x70 [ 3912.105314] ret_from_fork+0x1f/0x30 [ 3912.306634] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.338026] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.345266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3912.351646] Workqueue: events free_work [libcfs] [ 3912.357717] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3912.360055] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3912.372055] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3912.376050] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3912.379965] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3912.383600] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3912.390842] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3912.398241] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3912.402153] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3912.406682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.415020] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3912.419955] Call Trace: [ 3912.420794] [ 3912.425340] ? show_regs.cold.9+0x22/0x2f [ 3912.426884] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.433114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.437014] ? watchdog+0x40/0x40 [ 3912.439539] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.443089] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.446902] ? apic_timer_interrupt+0xf/0x20 [ 3912.449827] [ 3912.450854] ? on_each_cpu+0x51/0x120 [ 3912.452522] ? smp_call_function_many_cond+0x3db/0x540 [ 3912.454975] ? smp_call_function_many_cond+0x3a8/0x540 [ 3912.457582] ? load_new_mm_cr3+0x120/0x120 [ 3912.459617] ? load_new_mm_cr3+0x120/0x120 [ 3912.461826] on_each_cpu+0x51/0x120 [ 3912.463581] flush_tlb_kernel_range+0x93/0xb0 [ 3912.466026] free_unmap_vmap_area+0x63/0x70 [ 3912.468388] remove_vm_area+0xe7/0x110 [ 3912.470372] __vunmap+0x9c/0x400 [ 3912.472150] vfree+0x4d/0x100 [ 3912.473490] free_work+0x34/0x50 [libcfs] [ 3912.475788] process_one_work+0x2c8/0x700 [ 3912.477910] worker_thread+0x296/0x6e0 [ 3912.480243] ? rescuer_thread+0x570/0x570 [ 3912.482928] kthread+0x1d1/0x200 [ 3912.484614] ? set_kthread_struct+0x70/0x70 [ 3912.486831] ret_from_fork+0x1f/0x30 [ 3916.257365] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1557s! [ 3916.261338] Showing busy workqueues and worker pools: [ 3916.272254] workqueue events: flags=0x0 [ 3916.279906] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.284260] in-flight: 56293:free_work [libcfs] [ 3916.293866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3916.303246] in-flight: 22277:free_work [libcfs] [ 3916.312131] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3916.320867] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.327932] pending: nfs4_renew_state [nfsv4] [ 3916.334912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3916.345323] in-flight: 47596:free_work [libcfs] [ 3916.353086] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3916.363268] workqueue events_unbound: flags=0x2 [ 3916.371236] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3916.382038] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3916.395524] workqueue events_power_efficient: flags=0x80 [ 3916.403814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3916.408075] pending: neigh_periodic_work, do_cache_clean [ 3916.410455] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.416155] pending: check_lifetime [ 3916.419287] workqueue mm_percpu_wq: flags=0x8 [ 3916.422309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.429074] pending: vmstat_update [ 3916.430547] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.437328] pending: vmstat_update [ 3916.441200] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.450018] pending: vmstat_update [ 3916.456166] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3916.462408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3916.472241] pending: cgroup_pidlist_destroy_work_fn [ 3916.480044] workqueue writeback: flags=0x4a [ 3916.486874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3916.490020] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3916.498947] workqueue xprtiod: flags=0xa [ 3916.503141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3916.509998] pending: xprt_autoclose [ 3916.514230] workqueue ipv6_addrconf: flags=0x40008 [ 3916.518118] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3916.521113] pending: addrconf_verify_work [ 3916.523236] workqueue knbd0-recv: flags=0x1a [ 3916.524905] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3916.529612] in-flight: 655:recv_work [ 3916.533282] workqueue ptlrpc_pinger: flags=0x2 [ 3916.537394] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3916.543855] pending: ptlrpc_pinger_main [ptlrpc] [ 3916.546614] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1558s workers=5 idle: 53684 27131 54850 46584 [ 3916.555979] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1558s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3916.564402] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1558s workers=6 idle: 53660 37674 46417 53664 46418 [ 3916.576016] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3906s workers=2 idle: 657 [ 3924.061709] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3924.069909] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3924.116679] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3924.125855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3924.134922] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3924.137469] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3924.159483] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3924.170269] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3924.174841] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3924.183305] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3924.190853] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3924.197873] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3924.208321] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3924.214966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3924.217364] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3924.224228] Call Trace: [ 3924.225303] [ 3924.226256] ? show_regs.cold.9+0x22/0x2f [ 3924.227764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3924.229084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3924.231223] ? watchdog+0x40/0x40 [ 3924.233013] ? hrtimer_interrupt+0x128/0x2c0 [ 3924.235222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3924.237270] ? apic_timer_interrupt+0xf/0x20 [ 3924.238859] [ 3924.239763] ? on_each_cpu_cond_mask+0x46/0x150 [ 3924.241660] ? __x64_sys_fspick+0x210/0x210 [ 3924.243093] ? smp_call_function_many_cond+0x3db/0x540 [ 3924.245343] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3924.247381] ? __x64_sys_fspick+0x210/0x210 [ 3924.249403] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3924.251628] on_each_cpu_cond_mask+0x46/0x150 [ 3924.253347] on_each_cpu_cond+0x1c/0x30 [ 3924.255289] invalidate_bh_lrus+0x27/0x30 [ 3924.258341] invalidate_bdev+0x5a/0x90 [ 3924.269939] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3924.278820] generic_shutdown_super+0xb1/0x1b0 [ 3924.282982] kill_block_super+0x29/0x80 [ 3924.286347] deactivate_locked_super+0x52/0xd0 [ 3924.292343] deactivate_super+0x83/0x90 [ 3924.298137] cleanup_mnt+0x5f/0xe0 [ 3924.303114] __cleanup_mnt+0x16/0x20 [ 3924.304400] task_work_run+0xc6/0x110 [ 3924.308142] exit_to_usermode_loop+0x1dd/0x1f0 [ 3924.313792] do_syscall_64+0x3d6/0x440 [ 3924.316179] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3924.320193] RIP: 0033:0x7fbb019648fb [ 3924.323879] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3924.350074] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3924.357912] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3924.364862] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3924.369929] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3924.374270] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3924.378270] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3940.059008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3940.061783] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3940.080233] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3940.084371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3940.087416] Workqueue: events free_work [libcfs] [ 3940.089007] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3940.091037] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3940.097512] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3940.100056] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3940.102602] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3940.105442] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3940.107903] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3940.110591] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3940.113190] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3940.116483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3940.118528] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3940.121086] Call Trace: [ 3940.121962] [ 3940.124226] ? show_regs.cold.9+0x22/0x2f [ 3940.125532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3940.127142] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3940.130502] ? watchdog+0x40/0x40 [ 3940.131675] ? hrtimer_interrupt+0x128/0x2c0 [ 3940.134673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3940.137270] ? apic_timer_interrupt+0xf/0x20 [ 3940.139211] [ 3940.140346] ? on_each_cpu+0x51/0x120 [ 3940.141909] ? smp_call_function_many_cond+0x3db/0x540 [ 3940.144218] ? smp_call_function_many_cond+0x3a8/0x540 [ 3940.146656] ? load_new_mm_cr3+0x120/0x120 [ 3940.148676] ? load_new_mm_cr3+0x120/0x120 [ 3940.150608] on_each_cpu+0x51/0x120 [ 3940.154046] flush_tlb_kernel_range+0x93/0xb0 [ 3940.158252] free_unmap_vmap_area+0x63/0x70 [ 3940.161572] remove_vm_area+0xe7/0x110 [ 3940.163236] __vunmap+0x9c/0x400 [ 3940.165217] vfree+0x4d/0x100 [ 3940.167371] free_work+0x34/0x50 [libcfs] [ 3940.169676] process_one_work+0x2c8/0x700 [ 3940.171610] worker_thread+0x296/0x6e0 [ 3940.174027] ? rescuer_thread+0x570/0x570 [ 3940.176408] kthread+0x1d1/0x200 [ 3940.178490] ? set_kthread_struct+0x70/0x70 [ 3940.179875] ret_from_fork+0x1f/0x30 [ 3944.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3944.058782] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3944.120355] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3944.127443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3944.135189] Workqueue: events free_work [libcfs] [ 3944.138692] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3944.140848] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3944.169317] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3944.177345] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3944.184788] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3944.190949] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3944.199975] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3944.210478] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3944.216950] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3944.226713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3944.230904] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3944.247881] Call Trace: [ 3944.251319] [ 3944.254309] ? show_regs.cold.9+0x22/0x2f [ 3944.258121] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3944.261139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3944.265758] ? watchdog+0x40/0x40 [ 3944.271194] ? hrtimer_interrupt+0x128/0x2c0 [ 3944.276957] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3944.284777] ? apic_timer_interrupt+0xf/0x20 [ 3944.293368] [ 3944.294176] ? on_each_cpu+0x51/0x120 [ 3944.300654] ? smp_call_function_many_cond+0x3db/0x540 [ 3944.303818] ? smp_call_function_many_cond+0x3a8/0x540 [ 3944.309396] ? load_new_mm_cr3+0x120/0x120 [ 3944.314215] ? load_new_mm_cr3+0x120/0x120 [ 3944.319351] on_each_cpu+0x51/0x120 [ 3944.322902] flush_tlb_kernel_range+0x93/0xb0 [ 3944.325363] free_unmap_vmap_area+0x63/0x70 [ 3944.330430] remove_vm_area+0xe7/0x110 [ 3944.333105] __vunmap+0x9c/0x400 [ 3944.335334] vfree+0x4d/0x100 [ 3944.339474] free_work+0x34/0x50 [libcfs] [ 3944.347324] process_one_work+0x2c8/0x700 [ 3944.353234] worker_thread+0x296/0x6e0 [ 3944.358381] ? rescuer_thread+0x570/0x570 [ 3944.363084] kthread+0x1d1/0x200 [ 3944.368303] ? set_kthread_struct+0x70/0x70 [ 3944.372259] ret_from_fork+0x1f/0x30 [ 3946.978662] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1588s! [ 3946.991390] Showing busy workqueues and worker pools: [ 3947.000054] workqueue events: flags=0x0 [ 3947.003255] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.009988] in-flight: 56293:free_work [libcfs] [ 3947.026587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3947.034564] in-flight: 22277:free_work [libcfs] [ 3947.042927] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3947.057990] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.062876] pending: nfs4_renew_state [nfsv4] [ 3947.066312] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3947.072271] in-flight: 47596:free_work [libcfs] [ 3947.074736] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3947.079664] workqueue events_unbound: flags=0x2 [ 3947.081832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3947.085026] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3947.089088] workqueue events_power_efficient: flags=0x80 [ 3947.092750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.095946] pending: neigh_periodic_work, do_cache_clean [ 3947.098892] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.102197] pending: check_lifetime [ 3947.104800] workqueue mm_percpu_wq: flags=0x8 [ 3947.106155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.108001] pending: vmstat_update [ 3947.112918] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.118006] pending: vmstat_update [ 3947.120870] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.124388] pending: vmstat_update [ 3947.132733] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3947.141961] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3947.156046] pending: cgroup_pidlist_destroy_work_fn [ 3947.157625] workqueue writeback: flags=0x4a [ 3947.168882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3947.174258] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3947.182792] workqueue xprtiod: flags=0xa [ 3947.186676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3947.201114] pending: xprt_autoclose [ 3947.202359] workqueue ipv6_addrconf: flags=0x40008 [ 3947.215904] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3947.228439] pending: addrconf_verify_work [ 3947.233578] workqueue knbd0-recv: flags=0x1a [ 3947.237989] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3947.253747] in-flight: 655:recv_work [ 3947.254940] workqueue ptlrpc_pinger: flags=0x2 [ 3947.256770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3947.259983] pending: ptlrpc_pinger_main [ptlrpc] [ 3947.263717] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1589s workers=5 idle: 53684 27131 54850 46584 [ 3947.277845] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1589s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3947.285043] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1589s workers=6 idle: 53660 37674 46417 53664 46418 [ 3947.290438] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3937s workers=2 idle: 657 [ 3952.059960] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3952.066443] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3952.130170] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3952.145956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3952.159009] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3952.163046] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3952.185418] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3952.195227] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3952.206285] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3952.227742] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3952.245839] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3952.248335] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3952.261856] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3952.274371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3952.277776] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3952.286895] Call Trace: [ 3952.290279] [ 3952.292179] ? show_regs.cold.9+0x22/0x2f [ 3952.295566] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3952.311351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3952.320403] ? watchdog+0x40/0x40 [ 3952.325842] ? hrtimer_interrupt+0x128/0x2c0 [ 3952.327322] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3952.336438] ? apic_timer_interrupt+0xf/0x20 [ 3952.341483] [ 3952.344502] ? on_each_cpu_cond_mask+0x46/0x150 [ 3952.351210] ? __x64_sys_fspick+0x210/0x210 [ 3952.363886] ? smp_call_function_many_cond+0x3db/0x540 [ 3952.365197] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3952.372330] ? __x64_sys_fspick+0x210/0x210 [ 3952.375271] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3952.381414] on_each_cpu_cond_mask+0x46/0x150 [ 3952.389726] on_each_cpu_cond+0x1c/0x30 [ 3952.393690] invalidate_bh_lrus+0x27/0x30 [ 3952.398249] invalidate_bdev+0x5a/0x90 [ 3952.400248] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3952.409879] generic_shutdown_super+0xb1/0x1b0 [ 3952.417392] kill_block_super+0x29/0x80 [ 3952.423331] deactivate_locked_super+0x52/0xd0 [ 3952.427669] deactivate_super+0x83/0x90 [ 3952.433908] cleanup_mnt+0x5f/0xe0 [ 3952.436759] __cleanup_mnt+0x16/0x20 [ 3952.439122] task_work_run+0xc6/0x110 [ 3952.443716] exit_to_usermode_loop+0x1dd/0x1f0 [ 3952.451160] do_syscall_64+0x3d6/0x440 [ 3952.456337] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3952.462151] RIP: 0033:0x7fbb019648fb [ 3952.472381] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3952.494329] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3952.503182] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3952.518342] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3952.529729] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3952.543875] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3952.560050] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3968.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3968.063213] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3968.087318] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3968.101176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3968.109659] Workqueue: events free_work [libcfs] [ 3968.111246] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 3968.113639] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 3968.121712] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3968.124858] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3968.127724] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3968.139066] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3968.141920] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3968.144442] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3968.147623] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3968.151171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3968.153894] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3968.163151] Call Trace: [ 3968.164613] [ 3968.165577] ? show_regs.cold.9+0x22/0x2f [ 3968.167188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3968.172920] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3968.178070] ? watchdog+0x40/0x40 [ 3968.180297] ? hrtimer_interrupt+0x128/0x2c0 [ 3968.183071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3968.191318] ? apic_timer_interrupt+0xf/0x20 [ 3968.193839] [ 3968.195135] ? on_each_cpu+0x51/0x120 [ 3968.197082] ? smp_call_function_many_cond+0x3d9/0x540 [ 3968.199775] ? smp_call_function_many_cond+0x3a8/0x540 [ 3968.203360] ? load_new_mm_cr3+0x120/0x120 [ 3968.204844] ? load_new_mm_cr3+0x120/0x120 [ 3968.206360] on_each_cpu+0x51/0x120 [ 3968.209163] flush_tlb_kernel_range+0x93/0xb0 [ 3968.212420] free_unmap_vmap_area+0x63/0x70 [ 3968.215352] remove_vm_area+0xe7/0x110 [ 3968.216893] __vunmap+0x9c/0x400 [ 3968.219984] vfree+0x4d/0x100 [ 3968.221955] free_work+0x34/0x50 [libcfs] [ 3968.223346] process_one_work+0x2c8/0x700 [ 3968.224982] worker_thread+0x296/0x6e0 [ 3968.228028] ? rescuer_thread+0x570/0x570 [ 3968.230548] kthread+0x1d1/0x200 [ 3968.233030] ? set_kthread_struct+0x70/0x70 [ 3968.235793] ret_from_fork+0x1f/0x30 [ 3972.057867] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 3972.065091] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3972.179918] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3972.199346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3972.224183] Workqueue: events free_work [libcfs] [ 3972.225996] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3972.240036] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3972.296998] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3972.313856] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 3972.336890] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 3972.348815] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 3972.362902] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3972.388688] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 3972.411898] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 3972.430179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3972.459033] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 3972.464936] Call Trace: [ 3972.475966] [ 3972.477791] ? show_regs.cold.9+0x22/0x2f [ 3972.479272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3972.498029] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3972.509967] ? watchdog+0x40/0x40 [ 3972.516794] ? hrtimer_interrupt+0x128/0x2c0 [ 3972.519995] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3972.521687] ? apic_timer_interrupt+0xf/0x20 [ 3972.526778] [ 3972.527918] ? on_each_cpu+0x51/0x120 [ 3972.530929] ? smp_call_function_many_cond+0x3db/0x540 [ 3972.536855] ? smp_call_function_many_cond+0x3a8/0x540 [ 3972.543042] ? load_new_mm_cr3+0x120/0x120 [ 3972.547680] ? load_new_mm_cr3+0x120/0x120 [ 3972.553176] on_each_cpu+0x51/0x120 [ 3972.556154] flush_tlb_kernel_range+0x93/0xb0 [ 3972.557899] free_unmap_vmap_area+0x63/0x70 [ 3972.561965] remove_vm_area+0xe7/0x110 [ 3972.566830] __vunmap+0x9c/0x400 [ 3972.570137] vfree+0x4d/0x100 [ 3972.572816] free_work+0x34/0x50 [libcfs] [ 3972.578317] process_one_work+0x2c8/0x700 [ 3972.582069] worker_thread+0x296/0x6e0 [ 3972.584883] ? rescuer_thread+0x570/0x570 [ 3972.590063] kthread+0x1d1/0x200 [ 3972.592027] ? set_kthread_struct+0x70/0x70 [ 3972.594869] ret_from_fork+0x1f/0x30 [ 3977.695031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1619s! [ 3977.703056] Showing busy workqueues and worker pools: [ 3977.706212] workqueue events: flags=0x0 [ 3977.709026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3977.716679] in-flight: 56293:free_work [libcfs] [ 3977.724922] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3977.729458] in-flight: 22277:free_work [libcfs] [ 3977.732535] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 3977.737864] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3977.743555] pending: nfs4_renew_state [nfsv4] [ 3977.749729] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3977.756582] in-flight: 47596:free_work [libcfs] [ 3977.762637] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3977.766742] workqueue events_unbound: flags=0x2 [ 3977.778342] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 3977.780832] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3977.787046] workqueue events_power_efficient: flags=0x80 [ 3977.800693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3977.817639] pending: neigh_periodic_work, do_cache_clean [ 3977.820523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3977.826389] pending: check_lifetime [ 3977.828077] workqueue mm_percpu_wq: flags=0x8 [ 3977.831018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3977.839559] pending: vmstat_update [ 3977.841133] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3977.852472] pending: vmstat_update [ 3977.859230] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3977.863859] pending: vmstat_update [ 3977.865425] workqueue cgroup_pidlist_destroy: flags=0x0 [ 3977.871657] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3977.876399] pending: cgroup_pidlist_destroy_work_fn [ 3977.879806] workqueue writeback: flags=0x4a [ 3977.881756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3977.885452] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 3977.889945] workqueue xprtiod: flags=0xa [ 3977.892453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3977.899534] pending: xprt_autoclose [ 3977.901669] workqueue ipv6_addrconf: flags=0x40008 [ 3977.903930] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3977.906607] pending: addrconf_verify_work [ 3977.908908] workqueue knbd0-recv: flags=0x1a [ 3977.911353] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3977.917899] in-flight: 655:recv_work [ 3977.921311] workqueue ptlrpc_pinger: flags=0x2 [ 3977.924335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3977.928551] pending: ptlrpc_pinger_main [ptlrpc] [ 3977.931797] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1620s workers=5 idle: 53684 27131 54850 46584 [ 3977.941542] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1620s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 3977.950714] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1620s workers=6 idle: 53660 37674 46417 53664 46418 [ 3977.955069] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3967s workers=2 idle: 657 [ 3980.064726] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 3980.071827] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3980.110977] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3980.124410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3980.133982] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3980.139814] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3980.159162] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3980.170988] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 3980.172897] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 3980.186186] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 3980.192964] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 3980.197929] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 3980.202541] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 3980.207959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3980.211720] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 3980.215932] Call Trace: [ 3980.221941] [ 3980.223177] ? show_regs.cold.9+0x22/0x2f [ 3980.226364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3980.230258] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3980.233499] ? watchdog+0x40/0x40 [ 3980.235834] ? hrtimer_interrupt+0x128/0x2c0 [ 3980.238970] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3980.241800] ? apic_timer_interrupt+0xf/0x20 [ 3980.244028] [ 3980.245916] ? on_each_cpu_cond_mask+0x46/0x150 [ 3980.248042] ? __x64_sys_fspick+0x210/0x210 [ 3980.251075] ? smp_call_function_many_cond+0x3db/0x540 [ 3980.254958] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3980.257726] ? __x64_sys_fspick+0x210/0x210 [ 3980.260204] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 3980.263152] on_each_cpu_cond_mask+0x46/0x150 [ 3980.265904] on_each_cpu_cond+0x1c/0x30 [ 3980.268139] invalidate_bh_lrus+0x27/0x30 [ 3980.270774] invalidate_bdev+0x5a/0x90 [ 3980.272923] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 3980.275994] generic_shutdown_super+0xb1/0x1b0 [ 3980.277924] kill_block_super+0x29/0x80 [ 3980.279533] deactivate_locked_super+0x52/0xd0 [ 3980.281989] deactivate_super+0x83/0x90 [ 3980.284156] cleanup_mnt+0x5f/0xe0 [ 3980.286260] __cleanup_mnt+0x16/0x20 [ 3980.288723] task_work_run+0xc6/0x110 [ 3980.290835] exit_to_usermode_loop+0x1dd/0x1f0 [ 3980.293187] do_syscall_64+0x3d6/0x440 [ 3980.296003] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3980.299068] RIP: 0033:0x7fbb019648fb [ 3980.301787] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 3980.317960] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 3980.332023] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 3980.340130] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 3980.350902] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 3980.360251] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 3980.369670] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 3996.059040] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 3996.068664] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3996.109325] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3996.124419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 3996.135623] Workqueue: events free_work [libcfs] [ 3996.140271] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 3996.144466] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 3996.157629] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3996.168490] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 3996.173765] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 3996.180132] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 3996.185707] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 3996.188031] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 3996.192658] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 3996.197649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3996.200663] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 3996.204399] Call Trace: [ 3996.205744] [ 3996.207149] ? show_regs.cold.9+0x22/0x2f [ 3996.210456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3996.213895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3996.217963] ? watchdog+0x40/0x40 [ 3996.223166] ? hrtimer_interrupt+0x128/0x2c0 [ 3996.225736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3996.230450] ? apic_timer_interrupt+0xf/0x20 [ 3996.235560] [ 3996.237506] ? on_each_cpu+0x51/0x120 [ 3996.240609] ? smp_call_function_many_cond+0x3db/0x540 [ 3996.250423] ? smp_call_function_many_cond+0x3a8/0x540 [ 3996.252633] ? load_new_mm_cr3+0x120/0x120 [ 3996.262180] ? load_new_mm_cr3+0x120/0x120 [ 3996.263743] on_each_cpu+0x51/0x120 [ 3996.273154] flush_tlb_kernel_range+0x93/0xb0 [ 3996.275730] free_unmap_vmap_area+0x63/0x70 [ 3996.286208] remove_vm_area+0xe7/0x110 [ 3996.290448] __vunmap+0x9c/0x400 [ 3996.303467] vfree+0x4d/0x100 [ 3996.306551] free_work+0x34/0x50 [libcfs] [ 3996.313408] process_one_work+0x2c8/0x700 [ 3996.316321] worker_thread+0x296/0x6e0 [ 3996.319663] ? rescuer_thread+0x570/0x570 [ 3996.322641] kthread+0x1d1/0x200 [ 3996.324260] ? set_kthread_struct+0x70/0x70 [ 3996.325771] ret_from_fork+0x1f/0x30 [ 4004.055223] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4004.062823] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4004.092698] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4004.110835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4004.117252] Workqueue: events free_work [libcfs] [ 4004.123027] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4004.124475] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4004.142839] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4004.147073] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4004.148967] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4004.150652] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4004.159094] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4004.164345] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4004.168628] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4004.173850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4004.179527] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4004.184774] Call Trace: [ 4004.187258] [ 4004.188324] ? show_regs.cold.9+0x22/0x2f [ 4004.191576] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4004.198023] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4004.200081] ? watchdog+0x40/0x40 [ 4004.202320] ? hrtimer_interrupt+0x128/0x2c0 [ 4004.204453] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4004.208272] ? apic_timer_interrupt+0xf/0x20 [ 4004.211369] [ 4004.213501] ? on_each_cpu+0x51/0x120 [ 4004.215606] ? smp_call_function_many_cond+0x3db/0x540 [ 4004.221649] ? smp_call_function_many_cond+0x3a8/0x540 [ 4004.227539] ? load_new_mm_cr3+0x120/0x120 [ 4004.229784] ? load_new_mm_cr3+0x120/0x120 [ 4004.232626] on_each_cpu+0x51/0x120 [ 4004.234493] flush_tlb_kernel_range+0x93/0xb0 [ 4004.236680] free_unmap_vmap_area+0x63/0x70 [ 4004.238874] remove_vm_area+0xe7/0x110 [ 4004.241087] __vunmap+0x9c/0x400 [ 4004.242858] vfree+0x4d/0x100 [ 4004.244186] free_work+0x34/0x50 [libcfs] [ 4004.246542] process_one_work+0x2c8/0x700 [ 4004.248386] worker_thread+0x296/0x6e0 [ 4004.254300] ? rescuer_thread+0x570/0x570 [ 4004.257113] kthread+0x1d1/0x200 [ 4004.261386] ? set_kthread_struct+0x70/0x70 [ 4004.263030] ret_from_fork+0x1f/0x30 [ 4008.058009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4008.062224] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4008.090026] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4008.097484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4008.106760] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4008.110306] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4008.130016] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4008.144255] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4008.151033] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4008.158207] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4008.169871] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4008.179581] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4008.192541] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4008.199047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4008.203170] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4008.207680] Call Trace: [ 4008.213619] [ 4008.216009] ? show_regs.cold.9+0x22/0x2f [ 4008.217582] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4008.222084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4008.225667] ? watchdog+0x40/0x40 [ 4008.227839] ? hrtimer_interrupt+0x128/0x2c0 [ 4008.229464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4008.232184] ? apic_timer_interrupt+0xf/0x20 [ 4008.235636] [ 4008.238018] ? on_each_cpu_cond_mask+0x46/0x150 [ 4008.242191] ? __x64_sys_fspick+0x210/0x210 [ 4008.246427] ? smp_call_function_many_cond+0x3d9/0x540 [ 4008.250088] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4008.252995] ? __x64_sys_fspick+0x210/0x210 [ 4008.260109] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4008.267139] on_each_cpu_cond_mask+0x46/0x150 [ 4008.271253] on_each_cpu_cond+0x1c/0x30 [ 4008.276007] invalidate_bh_lrus+0x27/0x30 [ 4008.283318] invalidate_bdev+0x5a/0x90 [ 4008.290256] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4008.298012] generic_shutdown_super+0xb1/0x1b0 [ 4008.302071] kill_block_super+0x29/0x80 [ 4008.305561] deactivate_locked_super+0x52/0xd0 [ 4008.309113] deactivate_super+0x83/0x90 [ 4008.314273] cleanup_mnt+0x5f/0xe0 [ 4008.315673] __cleanup_mnt+0x16/0x20 [ 4008.319791] task_work_run+0xc6/0x110 [ 4008.322666] exit_to_usermode_loop+0x1dd/0x1f0 [ 4008.329160] do_syscall_64+0x3d6/0x440 [ 4008.331464] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4008.333500] RIP: 0033:0x7fbb019648fb [ 4008.341947] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4008.356815] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4008.364342] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4008.372118] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4008.378611] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4008.384057] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4008.389244] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4008.422526] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1650s! [ 4008.434968] Showing busy workqueues and worker pools: [ 4008.441883] workqueue events: flags=0x0 [ 4008.443931] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.451384] in-flight: 56293:free_work [libcfs] [ 4008.456008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4008.465851] in-flight: 22277:free_work [libcfs] [ 4008.468935] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4008.474531] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.483308] pending: nfs4_renew_state [nfsv4] [ 4008.486210] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4008.489576] in-flight: 47596:free_work [libcfs] [ 4008.492000] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4008.496253] workqueue events_unbound: flags=0x2 [ 4008.498577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4008.501684] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4008.505133] workqueue events_power_efficient: flags=0x80 [ 4008.507583] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4008.511207] pending: neigh_periodic_work, do_cache_clean [ 4008.515284] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.519178] pending: check_lifetime [ 4008.520560] workqueue mm_percpu_wq: flags=0x8 [ 4008.522056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.526885] pending: vmstat_update [ 4008.529722] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.533684] pending: vmstat_update [ 4008.537099] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4008.541828] pending: vmstat_update [ 4008.543750] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4008.546277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4008.550068] pending: cgroup_pidlist_destroy_work_fn [ 4008.553516] workqueue writeback: flags=0x4a [ 4008.556249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4008.560474] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4008.563773] workqueue xprtiod: flags=0xa [ 4008.566683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4008.568843] pending: xprt_autoclose [ 4008.571219] workqueue ipv6_addrconf: flags=0x40008 [ 4008.574068] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4008.577761] pending: addrconf_verify_work [ 4008.582704] workqueue knbd0-recv: flags=0x1a [ 4008.587114] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4008.592350] in-flight: 655:recv_work [ 4008.594163] workqueue ptlrpc_pinger: flags=0x2 [ 4008.595509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4008.599302] pending: ptlrpc_pinger_main [ptlrpc] [ 4008.602370] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1650s workers=5 idle: 53684 27131 54850 46584 [ 4008.610099] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1650s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4008.615510] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1650s workers=6 idle: 53660 37674 46417 53664 46418 [ 4008.621687] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3998s workers=2 idle: 657 [ 4024.067195] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4024.077158] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4024.129779] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4024.144080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4024.153457] Workqueue: events free_work [libcfs] [ 4024.158772] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4024.166923] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4024.183667] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4024.200077] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4024.202286] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4024.211879] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4024.218361] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4024.224898] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4024.231604] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4024.236801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4024.240781] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4024.248963] Call Trace: [ 4024.250338] [ 4024.251163] ? show_regs.cold.9+0x22/0x2f [ 4024.253958] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4024.257188] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4024.259891] ? watchdog+0x40/0x40 [ 4024.261692] ? hrtimer_interrupt+0x128/0x2c0 [ 4024.266255] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4024.269681] ? apic_timer_interrupt+0xf/0x20 [ 4024.275427] [ 4024.276626] ? on_each_cpu+0x51/0x120 [ 4024.280534] ? smp_call_function_many_cond+0x3d9/0x540 [ 4024.286005] ? smp_call_function_many_cond+0x3a8/0x540 [ 4024.288104] ? load_new_mm_cr3+0x120/0x120 [ 4024.291942] ? load_new_mm_cr3+0x120/0x120 [ 4024.297690] on_each_cpu+0x51/0x120 [ 4024.299119] flush_tlb_kernel_range+0x93/0xb0 [ 4024.303762] free_unmap_vmap_area+0x63/0x70 [ 4024.306798] remove_vm_area+0xe7/0x110 [ 4024.309253] __vunmap+0x9c/0x400 [ 4024.310469] vfree+0x4d/0x100 [ 4024.314826] free_work+0x34/0x50 [libcfs] [ 4024.317107] process_one_work+0x2c8/0x700 [ 4024.321534] worker_thread+0x296/0x6e0 [ 4024.324595] ? rescuer_thread+0x570/0x570 [ 4024.329454] kthread+0x1d1/0x200 [ 4024.331668] ? set_kthread_struct+0x70/0x70 [ 4024.333884] ret_from_fork+0x1f/0x30 [ 4036.058317] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4036.065952] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4036.077000] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4036.077093] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4036.077100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4036.077107] Workqueue: events free_work [libcfs] [ 4036.077147] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4036.077156] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4036.077160] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4036.077165] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4036.077168] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4036.077170] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4036.077172] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4036.077174] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4036.077178] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4036.077181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4036.077183] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4036.079834] Call Trace: [ 4036.079847] [ 4036.079853] ? show_regs.cold.9+0x22/0x2f [ 4036.079870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4036.079879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4036.079883] ? watchdog+0x40/0x40 [ 4036.079886] ? hrtimer_interrupt+0x128/0x2c0 [ 4036.079890] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4036.079897] ? apic_timer_interrupt+0xf/0x20 [ 4036.079901] [ 4036.079903] ? on_each_cpu+0x51/0x120 [ 4036.079908] ? smp_call_function_many_cond+0x3db/0x540 [ 4036.079912] ? smp_call_function_many_cond+0x3a8/0x540 [ 4036.079916] ? load_new_mm_cr3+0x120/0x120 [ 4036.079921] ? load_new_mm_cr3+0x120/0x120 [ 4036.079925] on_each_cpu+0x51/0x120 [ 4036.079928] flush_tlb_kernel_range+0x93/0xb0 [ 4036.079933] free_unmap_vmap_area+0x63/0x70 [ 4036.079938] remove_vm_area+0xe7/0x110 [ 4036.079941] __vunmap+0x9c/0x400 [ 4036.079945] vfree+0x4d/0x100 [ 4036.079950] free_work+0x34/0x50 [libcfs] [ 4036.079984] process_one_work+0x2c8/0x700 [ 4036.079990] worker_thread+0x296/0x6e0 [ 4036.079994] ? rescuer_thread+0x570/0x570 [ 4036.079998] kthread+0x1d1/0x200 [ 4036.080013] ? set_kthread_struct+0x70/0x70 [ 4036.080018] ret_from_fork+0x1f/0x30 [ 4036.409627] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4036.440771] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4036.446153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4036.458709] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4036.470966] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4036.489296] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4036.493965] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4036.500641] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4036.513174] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4036.525703] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4036.536564] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4036.545589] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4036.554283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4036.557632] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4036.561153] Call Trace: [ 4036.561930] [ 4036.567660] ? show_regs.cold.9+0x22/0x2f [ 4036.570513] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4036.583619] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4036.594179] ? watchdog+0x40/0x40 [ 4036.595440] ? hrtimer_interrupt+0x128/0x2c0 [ 4036.597053] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4036.603119] ? apic_timer_interrupt+0xf/0x20 [ 4036.612043] [ 4036.614099] ? on_each_cpu_cond_mask+0x46/0x150 [ 4036.620665] ? __x64_sys_fspick+0x210/0x210 [ 4036.637672] ? smp_call_function_many_cond+0x3db/0x540 [ 4036.643906] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4036.658291] ? __x64_sys_fspick+0x210/0x210 [ 4036.659793] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4036.683900] on_each_cpu_cond_mask+0x46/0x150 [ 4036.687219] on_each_cpu_cond+0x1c/0x30 [ 4036.691951] invalidate_bh_lrus+0x27/0x30 [ 4036.695997] invalidate_bdev+0x5a/0x90 [ 4036.698712] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4036.702973] generic_shutdown_super+0xb1/0x1b0 [ 4036.712180] kill_block_super+0x29/0x80 [ 4036.717137] deactivate_locked_super+0x52/0xd0 [ 4036.718713] deactivate_super+0x83/0x90 [ 4036.721126] cleanup_mnt+0x5f/0xe0 [ 4036.723494] __cleanup_mnt+0x16/0x20 [ 4036.726326] task_work_run+0xc6/0x110 [ 4036.729549] exit_to_usermode_loop+0x1dd/0x1f0 [ 4036.735750] do_syscall_64+0x3d6/0x440 [ 4036.737243] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4036.749101] RIP: 0033:0x7fbb019648fb [ 4036.765485] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4036.785164] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4036.790538] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4036.794926] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4036.798862] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4036.801945] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4036.805958] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4037.778017] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4037.781444] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 4037.787777] (detected by 2, t=1680048 jiffies, g=626849, q=1398) [ 4037.791516] Sending NMI from CPU 2 to CPUs 3: [ 4037.794715] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 4037.794724] rcu: rcu_sched kthread starved for 1679690 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4037.809891] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4037.815060] rcu: RCU grace-period kthread stack dump: [ 4037.817706] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4037.825440] Call Trace: [ 4037.826910] __schedule+0x351/0xcb0 [ 4037.830795] schedule+0xc0/0x180 [ 4037.832122] schedule_timeout+0xb4/0x190 [ 4037.839887] ? do_raw_spin_unlock+0x75/0x190 [ 4037.842942] ? __next_timer_interrupt+0x160/0x160 [ 4037.848646] ? prepare_to_swait_event+0xf3/0x140 [ 4037.850529] rcu_gp_kthread+0x8dd/0x15b0 [ 4037.856495] ? force_qs_rnp+0x2c0/0x2c0 [ 4037.860980] kthread+0x1d1/0x200 [ 4037.862383] ? set_kthread_struct+0x70/0x70 [ 4037.864113] ret_from_fork+0x1f/0x30 [ 4037.869773] rcu: Stack dump where RCU GP kthread last ran: [ 4037.873615] NMI backtrace for cpu 2 [ 4037.876411] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4037.885514] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4037.893633] Workqueue: events free_work [libcfs] [ 4037.896614] Call Trace: [ 4037.898504] [ 4037.899658] ? dump_stack+0xbb/0x10e [ 4037.907197] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4037.910453] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4037.913797] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4037.918592] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4037.926525] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4037.929642] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4037.937452] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 4037.943901] ? update_process_times+0x6d/0xc0 [ 4037.947658] ? tick_sched_handle.isra.18+0x27/0x90 [ 4037.951883] ? tick_sched_timer+0x53/0xb0 [ 4037.955866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4037.958757] ? tick_sched_handle.isra.18+0x90/0x90 [ 4037.962633] ? hrtimer_interrupt+0x128/0x2c0 [ 4037.965311] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4037.967663] ? apic_timer_interrupt+0xf/0x20 [ 4037.970280] [ 4037.971081] ? on_each_cpu+0x51/0x120 [ 4037.973841] ? smp_call_function_many_cond+0x3db/0x540 [ 4037.976784] ? smp_call_function_many_cond+0x3a8/0x540 [ 4037.979654] ? load_new_mm_cr3+0x120/0x120 [ 4037.982443] ? load_new_mm_cr3+0x120/0x120 [ 4037.984705] ? on_each_cpu+0x51/0x120 [ 4037.986652] ? flush_tlb_kernel_range+0x93/0xb0 [ 4037.988602] ? free_unmap_vmap_area+0x63/0x70 [ 4037.992220] ? remove_vm_area+0xe7/0x110 [ 4037.994910] ? __vunmap+0x9c/0x400 [ 4038.000534] ? vfree+0x4d/0x100 [ 4038.003541] ? free_work+0x34/0x50 [libcfs] [ 4038.007712] ? process_one_work+0x2c8/0x700 [ 4038.011510] ? worker_thread+0x296/0x6e0 [ 4038.014327] ? rescuer_thread+0x570/0x570 [ 4038.019683] ? kthread+0x1d1/0x200 [ 4038.022286] ? set_kthread_struct+0x70/0x70 [ 4038.024019] ? ret_from_fork+0x1f/0x30 [ 4039.135206] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1680s! [ 4039.139975] Showing busy workqueues and worker pools: [ 4039.142929] workqueue events: flags=0x0 [ 4039.145502] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4039.149453] in-flight: 56293:free_work [libcfs] [ 4039.152828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4039.156932] in-flight: 22277:free_work [libcfs] [ 4039.160041] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4039.166236] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4039.169878] pending: nfs4_renew_state [nfsv4] [ 4039.173370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4039.177308] in-flight: 47596:free_work [libcfs] [ 4039.179977] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4039.184677] workqueue events_unbound: flags=0x2 [ 4039.187870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4039.192341] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4039.196901] workqueue events_power_efficient: flags=0x80 [ 4039.201070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4039.206408] pending: neigh_periodic_work, do_cache_clean [ 4039.209374] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4039.213213] pending: check_lifetime [ 4039.215319] workqueue mm_percpu_wq: flags=0x8 [ 4039.217425] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4039.222315] pending: vmstat_update [ 4039.224560] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4039.227978] pending: vmstat_update [ 4039.229874] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4039.232145] pending: vmstat_update [ 4039.237387] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4039.241215] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4039.246537] pending: cgroup_pidlist_destroy_work_fn [ 4039.251330] workqueue writeback: flags=0x4a [ 4039.253513] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4039.256686] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4039.260229] workqueue xprtiod: flags=0xa [ 4039.262471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4039.266560] pending: xprt_autoclose [ 4039.269477] workqueue ipv6_addrconf: flags=0x40008 [ 4039.272836] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4039.277644] pending: addrconf_verify_work [ 4039.281844] workqueue knbd0-recv: flags=0x1a [ 4039.287172] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4039.291724] in-flight: 655:recv_work [ 4039.295742] workqueue ptlrpc_pinger: flags=0x2 [ 4039.299810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4039.304863] pending: ptlrpc_pinger_main [ptlrpc] [ 4039.308266] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1681s workers=5 idle: 53684 27131 54850 46584 [ 4039.319565] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1681s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4039.331493] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1681s workers=6 idle: 53660 37674 46417 53664 46418 [ 4039.335694] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4029s workers=2 idle: 657 [ 4064.056955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4064.059007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4064.064887] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.064970] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.064977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4064.064984] Workqueue: events free_work [libcfs] [ 4064.065021] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4064.067361] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4064.067821] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.067902] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.067907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4064.067910] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4064.067920] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4064.067924] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4064.068198] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4064.068201] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4064.068203] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4064.068205] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4064.068208] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4064.068211] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4064.068214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.068216] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4064.068224] Call Trace: [ 4064.068229] [ 4064.068234] ? show_regs.cold.9+0x22/0x2f [ 4064.068244] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.068249] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.068254] ? watchdog+0x40/0x40 [ 4064.068258] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.068262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.068269] ? apic_timer_interrupt+0xf/0x20 [ 4064.068274] [ 4064.068276] ? on_each_cpu_cond_mask+0x46/0x150 [ 4064.068279] ? __x64_sys_fspick+0x210/0x210 [ 4064.068286] ? smp_call_function_many_cond+0x3db/0x540 [ 4064.068290] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4064.068293] ? __x64_sys_fspick+0x210/0x210 [ 4064.068297] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4064.068299] on_each_cpu_cond_mask+0x46/0x150 [ 4064.068304] on_each_cpu_cond+0x1c/0x30 [ 4064.068307] invalidate_bh_lrus+0x27/0x30 [ 4064.068311] invalidate_bdev+0x5a/0x90 [ 4064.068317] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4064.071725] generic_shutdown_super+0xb1/0x1b0 [ 4064.071737] kill_block_super+0x29/0x80 [ 4064.072044] deactivate_locked_super+0x52/0xd0 [ 4064.072049] deactivate_super+0x83/0x90 [ 4064.072055] cleanup_mnt+0x5f/0xe0 [ 4064.072061] __cleanup_mnt+0x16/0x20 [ 4064.072066] task_work_run+0xc6/0x110 [ 4064.072073] exit_to_usermode_loop+0x1dd/0x1f0 [ 4064.072080] do_syscall_64+0x3d6/0x440 [ 4064.072085] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4064.072093] RIP: 0033:0x7fbb019648fb [ 4064.072100] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4064.072105] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4064.072112] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4064.072115] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4064.072117] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4064.072119] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4064.072121] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4064.072847] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.072930] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.072937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4064.072943] Workqueue: events free_work [libcfs] [ 4064.072987] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4064.072995] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4064.072999] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4064.073004] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4064.073006] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4064.073008] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4064.073010] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4064.073012] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4064.073016] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4064.073019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.073021] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4064.073030] Call Trace: [ 4064.073034] [ 4064.073037] ? show_regs.cold.9+0x22/0x2f [ 4064.073047] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.073053] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.073057] ? watchdog+0x40/0x40 [ 4064.073061] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.073065] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.073073] ? apic_timer_interrupt+0xf/0x20 [ 4064.073077] [ 4064.073079] ? on_each_cpu+0x51/0x120 [ 4064.073083] ? smp_call_function_many_cond+0x3db/0x540 [ 4064.073086] ? smp_call_function_many_cond+0x3a8/0x540 [ 4064.073089] ? load_new_mm_cr3+0x120/0x120 [ 4064.073094] ? load_new_mm_cr3+0x120/0x120 [ 4064.073098] on_each_cpu+0x51/0x120 [ 4064.073102] flush_tlb_kernel_range+0x93/0xb0 [ 4064.073107] free_unmap_vmap_area+0x63/0x70 [ 4064.073111] remove_vm_area+0xe7/0x110 [ 4064.073115] __vunmap+0x9c/0x400 [ 4064.073118] vfree+0x4d/0x100 [ 4064.073122] free_work+0x34/0x50 [libcfs] [ 4064.073158] process_one_work+0x2c8/0x700 [ 4064.073164] worker_thread+0x296/0x6e0 [ 4064.073169] ? rescuer_thread+0x570/0x570 [ 4064.073172] kthread+0x1d1/0x200 [ 4064.073176] ? set_kthread_struct+0x70/0x70 [ 4064.073180] ret_from_fork+0x1f/0x30 [ 4064.825025] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4064.844054] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4064.848851] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4064.857250] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4064.866351] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4064.874330] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4064.881924] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4064.892805] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4064.897036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.900625] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4064.903996] Call Trace: [ 4064.904975] [ 4064.906695] ? show_regs.cold.9+0x22/0x2f [ 4064.908856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.917626] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.920354] ? watchdog+0x40/0x40 [ 4064.922372] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.926192] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.931882] ? apic_timer_interrupt+0xf/0x20 [ 4064.936686] [ 4064.939335] ? on_each_cpu+0x51/0x120 [ 4064.943924] ? smp_call_function_many_cond+0x3db/0x540 [ 4064.950425] ? smp_call_function_many_cond+0x3a8/0x540 [ 4064.955745] ? load_new_mm_cr3+0x120/0x120 [ 4064.960906] ? load_new_mm_cr3+0x120/0x120 [ 4064.965811] on_each_cpu+0x51/0x120 [ 4064.969000] flush_tlb_kernel_range+0x93/0xb0 [ 4064.974310] free_unmap_vmap_area+0x63/0x70 [ 4064.977672] remove_vm_area+0xe7/0x110 [ 4064.981232] __vunmap+0x9c/0x400 [ 4064.983898] vfree+0x4d/0x100 [ 4064.986269] free_work+0x34/0x50 [libcfs] [ 4064.988754] process_one_work+0x2c8/0x700 [ 4064.992042] worker_thread+0x296/0x6e0 [ 4064.995263] ? rescuer_thread+0x570/0x570 [ 4064.997933] kthread+0x1d1/0x200 [ 4065.000141] ? set_kthread_struct+0x70/0x70 [ 4065.003676] ret_from_fork+0x1f/0x30 [ 4069.347269] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1711s! [ 4069.350342] Showing busy workqueues and worker pools: [ 4069.362030] workqueue events: flags=0x0 [ 4069.371690] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.382843] in-flight: 56293:free_work [libcfs] [ 4069.385373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4069.398231] in-flight: 22277:free_work [libcfs] [ 4069.410191] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4069.419786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.429843] pending: nfs4_renew_state [nfsv4] [ 4069.436263] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4069.447227] in-flight: 47596:free_work [libcfs] [ 4069.451975] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4069.465289] workqueue events_unbound: flags=0x2 [ 4069.471529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4069.494476] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4069.502649] workqueue events_power_efficient: flags=0x80 [ 4069.511763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4069.519220] pending: neigh_periodic_work, do_cache_clean [ 4069.524530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.529514] pending: check_lifetime [ 4069.533641] workqueue mm_percpu_wq: flags=0x8 [ 4069.536565] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.545633] pending: vmstat_update [ 4069.548675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.558533] pending: vmstat_update [ 4069.566243] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4069.570446] pending: vmstat_update [ 4069.575668] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4069.582233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4069.585816] pending: cgroup_pidlist_destroy_work_fn [ 4069.589158] workqueue writeback: flags=0x4a [ 4069.592837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4069.600494] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4069.606186] workqueue xprtiod: flags=0xa [ 4069.608821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4069.620187] pending: xprt_autoclose [ 4069.621772] workqueue ipv6_addrconf: flags=0x40008 [ 4069.623009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4069.634613] pending: addrconf_verify_work [ 4069.637529] workqueue knbd0-recv: flags=0x1a [ 4069.645364] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4069.658600] in-flight: 655:recv_work [ 4069.662469] workqueue ptlrpc_pinger: flags=0x2 [ 4069.667492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4069.671588] pending: ptlrpc_pinger_main [ptlrpc] [ 4069.678567] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1711s workers=5 idle: 53684 27131 54850 46584 [ 4069.685284] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1711s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4069.696850] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1711s workers=6 idle: 53660 37674 46417 53664 46418 [ 4069.711706] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4059s workers=2 idle: 657 [ 4092.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4092.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4092.064007] Modules linked in: [ 4092.067388] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4092.067466] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4092.067473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4092.067479] Workqueue: events free_work [libcfs] [ 4092.067520] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4092.067528] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4092.067531] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4092.067536] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4092.067538] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4092.067540] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4092.067542] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4092.067544] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4092.067547] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4092.067549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4092.067551] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4092.067561] Call Trace: [ 4092.067565] [ 4092.067570] ? show_regs.cold.9+0x22/0x2f [ 4092.067580] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4092.067586] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4092.067590] ? watchdog+0x40/0x40 [ 4092.067595] ? hrtimer_interrupt+0x128/0x2c0 [ 4092.067598] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4092.067607] ? apic_timer_interrupt+0xf/0x20 [ 4092.067611] [ 4092.067613] ? on_each_cpu+0x51/0x120 [ 4092.067617] ? smp_call_function_many_cond+0x3d9/0x540 [ 4092.067620] ? smp_call_function_many_cond+0x3a8/0x540 [ 4092.067624] ? load_new_mm_cr3+0x120/0x120 [ 4092.067628] ? load_new_mm_cr3+0x120/0x120 [ 4092.067632] on_each_cpu+0x51/0x120 [ 4092.067637] flush_tlb_kernel_range+0x93/0xb0 [ 4092.067642] free_unmap_vmap_area+0x63/0x70 [ 4092.067648] remove_vm_area+0xe7/0x110 [ 4092.067651] __vunmap+0x9c/0x400 [ 4092.067654] vfree+0x4d/0x100 [ 4092.067658] free_work+0x34/0x50 [libcfs] [ 4092.067682] process_one_work+0x2c8/0x700 [ 4092.067689] worker_thread+0x296/0x6e0 [ 4092.067693] ? rescuer_thread+0x570/0x570 [ 4092.067696] kthread+0x1d1/0x200 [ 4092.067701] ? set_kthread_struct+0x70/0x70 [ 4092.067704] ret_from_fork+0x1f/0x30 [ 4092.274248] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4092.316950] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4092.325923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4092.332917] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4092.339491] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4092.350335] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4092.360189] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4092.368878] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4092.380589] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4092.389663] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4092.393786] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4092.404535] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4092.411475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4092.419658] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4092.427163] Call Trace: [ 4092.430022] [ 4092.432537] ? show_regs.cold.9+0x22/0x2f [ 4092.435564] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4092.438695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4092.442530] ? watchdog+0x40/0x40 [ 4092.443873] ? hrtimer_interrupt+0x128/0x2c0 [ 4092.449572] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4092.451989] ? apic_timer_interrupt+0xf/0x20 [ 4092.454698] [ 4092.457347] ? on_each_cpu_cond_mask+0x46/0x150 [ 4092.460579] ? __x64_sys_fspick+0x210/0x210 [ 4092.464322] ? smp_call_function_many_cond+0x3db/0x540 [ 4092.466818] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4092.468470] ? __x64_sys_fspick+0x210/0x210 [ 4092.469999] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4092.478024] on_each_cpu_cond_mask+0x46/0x150 [ 4092.482668] on_each_cpu_cond+0x1c/0x30 [ 4092.484204] invalidate_bh_lrus+0x27/0x30 [ 4092.487738] invalidate_bdev+0x5a/0x90 [ 4092.493293] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4092.498118] generic_shutdown_super+0xb1/0x1b0 [ 4092.503973] kill_block_super+0x29/0x80 [ 4092.505377] deactivate_locked_super+0x52/0xd0 [ 4092.512369] deactivate_super+0x83/0x90 [ 4092.518547] cleanup_mnt+0x5f/0xe0 [ 4092.522128] __cleanup_mnt+0x16/0x20 [ 4092.524693] task_work_run+0xc6/0x110 [ 4092.527126] exit_to_usermode_loop+0x1dd/0x1f0 [ 4092.533163] do_syscall_64+0x3d6/0x440 [ 4092.536213] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4092.540296] RIP: 0033:0x7fbb019648fb [ 4092.542702] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4092.575303] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4092.582180] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4092.591477] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4092.597427] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4092.611455] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4092.613728] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4096.056477] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4096.061655] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4096.119101] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4096.127828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4096.135422] Workqueue: events free_work [libcfs] [ 4096.138693] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4096.143452] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4096.157271] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4096.162307] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4096.165804] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4096.169502] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4096.172468] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4096.178973] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4096.182517] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4096.186353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4096.188008] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4096.199051] Call Trace: [ 4096.200481] [ 4096.201659] ? show_regs.cold.9+0x22/0x2f [ 4096.205579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4096.208332] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4096.217905] ? watchdog+0x40/0x40 [ 4096.220989] ? hrtimer_interrupt+0x128/0x2c0 [ 4096.225371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4096.228123] ? apic_timer_interrupt+0xf/0x20 [ 4096.230310] [ 4096.231402] ? on_each_cpu+0x51/0x120 [ 4096.234773] ? smp_call_function_many_cond+0x3db/0x540 [ 4096.237941] ? smp_call_function_many_cond+0x3a8/0x540 [ 4096.243422] ? load_new_mm_cr3+0x120/0x120 [ 4096.250348] ? load_new_mm_cr3+0x120/0x120 [ 4096.251862] on_each_cpu+0x51/0x120 [ 4096.256111] flush_tlb_kernel_range+0x93/0xb0 [ 4096.260597] free_unmap_vmap_area+0x63/0x70 [ 4096.264845] remove_vm_area+0xe7/0x110 [ 4096.268536] __vunmap+0x9c/0x400 [ 4096.272153] vfree+0x4d/0x100 [ 4096.274246] free_work+0x34/0x50 [libcfs] [ 4096.276277] process_one_work+0x2c8/0x700 [ 4096.278139] worker_thread+0x296/0x6e0 [ 4096.280582] ? rescuer_thread+0x570/0x570 [ 4096.282685] kthread+0x1d1/0x200 [ 4096.285569] ? set_kthread_struct+0x70/0x70 [ 4096.288461] ret_from_fork+0x1f/0x30 [ 4100.064029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1741s! [ 4100.070986] Showing busy workqueues and worker pools: [ 4100.079168] workqueue events: flags=0x0 [ 4100.084044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.090153] in-flight: 56293:free_work [libcfs] [ 4100.094433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4100.097244] in-flight: 22277:free_work [libcfs] [ 4100.107168] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4100.113356] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.117402] pending: nfs4_renew_state [nfsv4] [ 4100.121949] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4100.125323] in-flight: 47596:free_work [libcfs] [ 4100.131534] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4100.136988] workqueue events_unbound: flags=0x2 [ 4100.140581] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4100.144338] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4100.149162] workqueue events_power_efficient: flags=0x80 [ 4100.153188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4100.156597] pending: neigh_periodic_work, do_cache_clean [ 4100.159597] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.162485] pending: check_lifetime [ 4100.165113] workqueue mm_percpu_wq: flags=0x8 [ 4100.167206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.170444] pending: vmstat_update [ 4100.172489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.176289] pending: vmstat_update [ 4100.178441] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4100.187347] pending: vmstat_update [ 4100.193189] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4100.195538] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4100.198984] pending: cgroup_pidlist_destroy_work_fn [ 4100.201525] workqueue writeback: flags=0x4a [ 4100.205463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4100.208815] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4100.211239] workqueue xprtiod: flags=0xa [ 4100.213246] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4100.217089] pending: xprt_autoclose [ 4100.219498] workqueue ipv6_addrconf: flags=0x40008 [ 4100.222170] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4100.225081] pending: addrconf_verify_work [ 4100.226780] workqueue knbd0-recv: flags=0x1a [ 4100.228042] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4100.231330] in-flight: 655:recv_work [ 4100.233114] workqueue ptlrpc_pinger: flags=0x2 [ 4100.235188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4100.237887] pending: ptlrpc_pinger_main [ptlrpc] [ 4100.240605] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1742s workers=5 idle: 53684 27131 54850 46584 [ 4100.244017] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1742s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4100.248760] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1742s workers=6 idle: 53660 37674 46417 53664 46418 [ 4100.252561] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4090s workers=2 idle: 657 [ 4120.059281] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4120.062083] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4120.064179] Modules linked in: [ 4120.068129] Modules linked in: [ 4120.068132] zfs(O) spl(O) [ 4120.070189] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4120.070268] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4120.070276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4120.070279] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4120.070292] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4120.070296] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4120.070300] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4120.070303] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4120.070305] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4120.070307] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4120.070309] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4120.070311] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4120.070314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4120.070316] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4120.070324] Call Trace: [ 4120.070329] [ 4120.070334] ? show_regs.cold.9+0x22/0x2f [ 4120.070343] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4120.070349] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4120.070354] ? watchdog+0x40/0x40 [ 4120.070358] ? hrtimer_interrupt+0x128/0x2c0 [ 4120.070362] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4120.070371] ? apic_timer_interrupt+0xf/0x20 [ 4120.070375] [ 4120.070377] ? on_each_cpu_cond_mask+0x46/0x150 [ 4120.070381] ? __x64_sys_fspick+0x210/0x210 [ 4120.070389] ? smp_call_function_many_cond+0x3db/0x540 [ 4120.070393] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4120.070397] ? __x64_sys_fspick+0x210/0x210 [ 4120.070401] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4120.070405] on_each_cpu_cond_mask+0x46/0x150 [ 4120.070409] on_each_cpu_cond+0x1c/0x30 [ 4120.070425] invalidate_bh_lrus+0x27/0x30 [ 4120.070429] invalidate_bdev+0x5a/0x90 [ 4120.070436] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4120.071961] lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) [ 4120.073391] generic_shutdown_super+0xb1/0x1b0 [ 4120.103768] lfsck(O) [ 4120.111529] kill_block_super+0x29/0x80 [ 4120.111553] deactivate_locked_super+0x52/0xd0 [ 4120.111559] deactivate_super+0x83/0x90 [ 4120.111564] cleanup_mnt+0x5f/0xe0 [ 4120.111570] __cleanup_mnt+0x16/0x20 [ 4120.111575] task_work_run+0xc6/0x110 [ 4120.111583] exit_to_usermode_loop+0x1dd/0x1f0 [ 4120.111590] do_syscall_64+0x3d6/0x440 [ 4120.111595] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4120.111604] RIP: 0033:0x7fbb019648fb [ 4120.111613] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4120.111617] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4120.111623] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4120.111626] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4120.111628] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4120.111630] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4120.111632] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4120.310986] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4120.327071] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4120.332003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4120.336322] Workqueue: events free_work [libcfs] [ 4120.338550] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4120.340711] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4120.348747] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4120.351980] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4120.355962] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4120.359093] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4120.363412] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4120.368183] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4120.371647] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4120.376988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4120.380921] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4120.387623] Call Trace: [ 4120.388812] [ 4120.389880] ? show_regs.cold.9+0x22/0x2f [ 4120.392013] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4120.396649] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4120.397957] ? watchdog+0x40/0x40 [ 4120.399220] ? hrtimer_interrupt+0x128/0x2c0 [ 4120.403651] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4120.407662] ? apic_timer_interrupt+0xf/0x20 [ 4120.410574] [ 4120.411646] ? on_each_cpu+0x51/0x120 [ 4120.413746] ? smp_call_function_many_cond+0x3d9/0x540 [ 4120.419194] ? smp_call_function_many_cond+0x3a8/0x540 [ 4120.421118] ? load_new_mm_cr3+0x120/0x120 [ 4120.424024] ? load_new_mm_cr3+0x120/0x120 [ 4120.426799] on_each_cpu+0x51/0x120 [ 4120.428803] flush_tlb_kernel_range+0x93/0xb0 [ 4120.432914] free_unmap_vmap_area+0x63/0x70 [ 4120.435060] remove_vm_area+0xe7/0x110 [ 4120.437736] __vunmap+0x9c/0x400 [ 4120.439578] vfree+0x4d/0x100 [ 4120.441101] free_work+0x34/0x50 [libcfs] [ 4120.443718] process_one_work+0x2c8/0x700 [ 4120.445329] worker_thread+0x296/0x6e0 [ 4120.448774] ? rescuer_thread+0x570/0x570 [ 4120.450950] kthread+0x1d1/0x200 [ 4120.453583] ? set_kthread_struct+0x70/0x70 [ 4120.455699] ret_from_fork+0x1f/0x30 [ 4128.055032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4128.059183] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4128.094481] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4128.099975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4128.103932] Workqueue: events free_work [libcfs] [ 4128.106613] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4128.109432] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4128.119187] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4128.123309] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4128.126302] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4128.128953] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4128.131989] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4128.135169] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4128.138756] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4128.142407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4128.144525] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4128.147802] Call Trace: [ 4128.148773] [ 4128.149840] ? show_regs.cold.9+0x22/0x2f [ 4128.151681] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4128.153549] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4128.155825] ? watchdog+0x40/0x40 [ 4128.157835] ? hrtimer_interrupt+0x128/0x2c0 [ 4128.160274] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4128.162744] ? apic_timer_interrupt+0xf/0x20 [ 4128.166477] [ 4128.167756] ? on_each_cpu+0x51/0x120 [ 4128.169827] ? smp_call_function_many_cond+0x3d9/0x540 [ 4128.173388] ? smp_call_function_many_cond+0x3a8/0x540 [ 4128.175306] ? load_new_mm_cr3+0x120/0x120 [ 4128.176875] ? load_new_mm_cr3+0x120/0x120 [ 4128.179266] on_each_cpu+0x51/0x120 [ 4128.180908] flush_tlb_kernel_range+0x93/0xb0 [ 4128.183347] free_unmap_vmap_area+0x63/0x70 [ 4128.185455] remove_vm_area+0xe7/0x110 [ 4128.187389] __vunmap+0x9c/0x400 [ 4128.188907] vfree+0x4d/0x100 [ 4128.190675] free_work+0x34/0x50 [libcfs] [ 4128.192743] process_one_work+0x2c8/0x700 [ 4128.194856] worker_thread+0x296/0x6e0 [ 4128.196793] ? rescuer_thread+0x570/0x570 [ 4128.198868] kthread+0x1d1/0x200 [ 4128.200683] ? set_kthread_struct+0x70/0x70 [ 4128.202912] ret_from_fork+0x1f/0x30 [ 4130.271692] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1771s! [ 4130.283724] Showing busy workqueues and worker pools: [ 4130.287547] workqueue events: flags=0x0 [ 4130.290635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.296828] in-flight: 56293:free_work [libcfs] [ 4130.299783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4130.306698] in-flight: 22277:free_work [libcfs] [ 4130.308702] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4130.315580] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.318447] pending: nfs4_renew_state [nfsv4] [ 4130.320146] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4130.326394] in-flight: 47596:free_work [libcfs] [ 4130.329234] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4130.335113] workqueue events_unbound: flags=0x2 [ 4130.337636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4130.341283] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4130.346328] workqueue events_power_efficient: flags=0x80 [ 4130.349893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4130.353175] pending: neigh_periodic_work, do_cache_clean [ 4130.356303] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.359862] pending: check_lifetime [ 4130.362093] workqueue mm_percpu_wq: flags=0x8 [ 4130.364948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.368325] pending: vmstat_update [ 4130.370044] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.377154] pending: vmstat_update [ 4130.378611] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.382855] pending: vmstat_update [ 4130.385454] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4130.388257] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4130.391657] pending: cgroup_pidlist_destroy_work_fn [ 4130.395431] workqueue writeback: flags=0x4a [ 4130.398277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4130.401645] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4130.405494] workqueue xprtiod: flags=0xa [ 4130.407417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4130.411336] pending: xprt_autoclose [ 4130.413638] workqueue ipv6_addrconf: flags=0x40008 [ 4130.416068] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4130.419887] pending: addrconf_verify_work [ 4130.422538] workqueue knbd0-recv: flags=0x1a [ 4130.424796] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4130.427892] in-flight: 655:recv_work [ 4130.429837] workqueue ptlrpc_pinger: flags=0x2 [ 4130.432301] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4130.434557] pending: ptlrpc_pinger_main [ptlrpc] [ 4130.437419] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1772s workers=5 idle: 53684 27131 54850 46584 [ 4130.440481] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1772s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4130.444365] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1772s workers=6 idle: 53660 37674 46417 53664 46418 [ 4130.447504] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4120s workers=2 idle: 657 [ 4148.058254] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4148.062425] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4148.062532] Modules linked in: [ 4148.070931] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4148.071017] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4148.071024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4148.071031] Workqueue: events free_work [libcfs] [ 4148.071071] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4148.071081] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4148.071085] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4148.071089] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4148.071092] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4148.071094] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4148.071096] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4148.071098] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4148.071100] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4148.071103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4148.072192] zfs(O) [ 4148.111345] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4148.111362] Call Trace: [ 4148.111368] [ 4148.111373] ? show_regs.cold.9+0x22/0x2f [ 4148.111386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4148.111392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4148.111396] ? watchdog+0x40/0x40 [ 4148.111401] ? hrtimer_interrupt+0x128/0x2c0 [ 4148.111404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4148.111411] ? apic_timer_interrupt+0xf/0x20 [ 4148.111417] [ 4148.111419] ? on_each_cpu+0x51/0x120 [ 4148.111423] ? smp_call_function_many_cond+0x3db/0x540 [ 4148.111427] ? smp_call_function_many_cond+0x3a8/0x540 [ 4148.111429] ? load_new_mm_cr3+0x120/0x120 [ 4148.111447] ? load_new_mm_cr3+0x120/0x120 [ 4148.111451] on_each_cpu+0x51/0x120 [ 4148.111455] flush_tlb_kernel_range+0x93/0xb0 [ 4148.111460] free_unmap_vmap_area+0x63/0x70 [ 4148.111465] remove_vm_area+0xe7/0x110 [ 4148.111468] __vunmap+0x9c/0x400 [ 4148.111472] vfree+0x4d/0x100 [ 4148.111478] free_work+0x34/0x50 [libcfs] [ 4148.111512] process_one_work+0x2c8/0x700 [ 4148.111519] worker_thread+0x296/0x6e0 [ 4148.111523] ? rescuer_thread+0x570/0x570 [ 4148.111526] kthread+0x1d1/0x200 [ 4148.111530] ? set_kthread_struct+0x70/0x70 [ 4148.120343] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 4148.131375] ret_from_fork+0x1f/0x30 [ 4148.349780] libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4148.372867] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4148.377551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4148.383224] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4148.388864] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4148.408191] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4148.410432] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4148.425691] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4148.433670] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4148.440232] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4148.444734] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4148.450113] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4148.454829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4148.465894] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4148.474546] Call Trace: [ 4148.476813] [ 4148.477958] ? show_regs.cold.9+0x22/0x2f [ 4148.480554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4148.488739] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4148.490433] ? watchdog+0x40/0x40 [ 4148.491824] ? hrtimer_interrupt+0x128/0x2c0 [ 4148.493935] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4148.497808] ? apic_timer_interrupt+0xf/0x20 [ 4148.501671] [ 4148.502516] ? on_each_cpu_cond_mask+0x46/0x150 [ 4148.508837] ? __x64_sys_fspick+0x210/0x210 [ 4148.512062] ? smp_call_function_many_cond+0x3db/0x540 [ 4148.517338] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4148.519113] ? __x64_sys_fspick+0x210/0x210 [ 4148.528711] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4148.535882] on_each_cpu_cond_mask+0x46/0x150 [ 4148.538283] on_each_cpu_cond+0x1c/0x30 [ 4148.539521] invalidate_bh_lrus+0x27/0x30 [ 4148.543495] invalidate_bdev+0x5a/0x90 [ 4148.551908] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4148.554845] generic_shutdown_super+0xb1/0x1b0 [ 4148.558505] kill_block_super+0x29/0x80 [ 4148.561136] deactivate_locked_super+0x52/0xd0 [ 4148.563140] deactivate_super+0x83/0x90 [ 4148.566828] cleanup_mnt+0x5f/0xe0 [ 4148.568540] __cleanup_mnt+0x16/0x20 [ 4148.573449] task_work_run+0xc6/0x110 [ 4148.579941] exit_to_usermode_loop+0x1dd/0x1f0 [ 4148.582703] do_syscall_64+0x3d6/0x440 [ 4148.585486] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4148.589018] RIP: 0033:0x7fbb019648fb [ 4148.592910] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4148.604064] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4148.609992] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4148.620723] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4148.623677] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4148.629858] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4148.638379] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4156.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4156.058751] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4156.083165] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4156.091591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4156.104300] Workqueue: events free_work [libcfs] [ 4156.107656] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4156.114222] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4156.128012] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4156.136165] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4156.145145] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4156.148063] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4156.162208] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4156.167313] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4156.176306] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4156.182657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4156.189072] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4156.203238] Call Trace: [ 4156.206533] [ 4156.209314] ? show_regs.cold.9+0x22/0x2f [ 4156.211813] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4156.216754] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4156.218521] ? watchdog+0x40/0x40 [ 4156.221883] ? hrtimer_interrupt+0x128/0x2c0 [ 4156.228927] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4156.235651] ? apic_timer_interrupt+0xf/0x20 [ 4156.238913] [ 4156.240053] ? on_each_cpu+0x51/0x120 [ 4156.242965] ? smp_call_function_many_cond+0x3db/0x540 [ 4156.246375] ? smp_call_function_many_cond+0x3a8/0x540 [ 4156.249514] ? load_new_mm_cr3+0x120/0x120 [ 4156.252314] ? load_new_mm_cr3+0x120/0x120 [ 4156.254843] on_each_cpu+0x51/0x120 [ 4156.256074] flush_tlb_kernel_range+0x93/0xb0 [ 4156.257499] free_unmap_vmap_area+0x63/0x70 [ 4156.262857] remove_vm_area+0xe7/0x110 [ 4156.265654] __vunmap+0x9c/0x400 [ 4156.267545] vfree+0x4d/0x100 [ 4156.268800] free_work+0x34/0x50 [libcfs] [ 4156.271279] process_one_work+0x2c8/0x700 [ 4156.273724] worker_thread+0x296/0x6e0 [ 4156.275011] ? rescuer_thread+0x570/0x570 [ 4156.276815] kthread+0x1d1/0x200 [ 4156.280340] ? set_kthread_struct+0x70/0x70 [ 4156.285107] ret_from_fork+0x1f/0x30 [ 4160.482032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1802s! [ 4160.488774] Showing busy workqueues and worker pools: [ 4160.493030] workqueue events: flags=0x0 [ 4160.494772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4160.497766] in-flight: 56293:free_work [libcfs] [ 4160.504485] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4160.509783] in-flight: 22277:free_work [libcfs] [ 4160.512467] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4160.523552] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4160.528957] pending: nfs4_renew_state [nfsv4] [ 4160.534725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4160.538437] in-flight: 47596:free_work [libcfs] [ 4160.545081] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4160.552356] workqueue events_unbound: flags=0x2 [ 4160.555294] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4160.557932] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4160.563681] workqueue events_power_efficient: flags=0x80 [ 4160.568865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4160.578486] pending: neigh_periodic_work, do_cache_clean [ 4160.583148] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4160.586908] pending: check_lifetime [ 4160.588472] workqueue mm_percpu_wq: flags=0x8 [ 4160.591874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4160.596606] pending: vmstat_update [ 4160.598670] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4160.601758] pending: vmstat_update [ 4160.603210] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4160.609098] pending: vmstat_update [ 4160.611658] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4160.614936] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4160.618747] pending: cgroup_pidlist_destroy_work_fn [ 4160.622270] workqueue writeback: flags=0x4a [ 4160.624896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4160.627409] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4160.631938] workqueue xprtiod: flags=0xa [ 4160.634743] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4160.638667] pending: xprt_autoclose [ 4160.643225] workqueue ipv6_addrconf: flags=0x40008 [ 4160.648020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4160.656630] pending: addrconf_verify_work [ 4160.662611] workqueue knbd0-recv: flags=0x1a [ 4160.666817] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4160.672703] in-flight: 655:recv_work [ 4160.674206] workqueue ptlrpc_pinger: flags=0x2 [ 4160.680571] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4160.682935] pending: ptlrpc_pinger_main [ptlrpc] [ 4160.688942] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1802s workers=5 idle: 53684 27131 54850 46584 [ 4160.696296] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1802s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4160.709457] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1802s workers=6 idle: 53660 37674 46417 53664 46418 [ 4160.714434] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4150s workers=2 idle: 657 [ 4176.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4176.060071] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4176.061331] Modules linked in: zfs(O) [ 4176.064424] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs [ 4176.066558] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4176.066683] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4176.066691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4176.066694] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4176.066889] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4176.066894] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4176.066899] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4176.066901] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4176.066903] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4176.066905] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4176.066908] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4176.066911] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4176.066914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4176.066916] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4176.066924] Call Trace: [ 4176.066941] [ 4176.066947] ? show_regs.cold.9+0x22/0x2f [ 4176.066957] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4176.066963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4176.066967] ? watchdog+0x40/0x40 [ 4176.066972] ? hrtimer_interrupt+0x128/0x2c0 [ 4176.066976] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4176.090594] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4176.090625] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4176.090639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4176.090685] Workqueue: events free_work [libcfs] [ 4176.090729] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4176.090738] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4176.090741] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4176.090746] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4176.090748] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4176.090750] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4176.090752] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4176.090754] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4176.090757] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4176.090760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4176.090763] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4176.090771] Call Trace: [ 4176.090776] [ 4176.090781] ? show_regs.cold.9+0x22/0x2f [ 4176.090789] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4176.090795] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4176.090799] ? watchdog+0x40/0x40 [ 4176.090803] ? hrtimer_interrupt+0x128/0x2c0 [ 4176.090807] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4176.090814] ? apic_timer_interrupt+0xf/0x20 [ 4176.090819] [ 4176.090821] ? on_each_cpu+0x51/0x120 [ 4176.090825] ? smp_call_function_many_cond+0x3db/0x540 [ 4176.090829] ? smp_call_function_many_cond+0x3a8/0x540 [ 4176.090832] ? load_new_mm_cr3+0x120/0x120 [ 4176.090838] ? load_new_mm_cr3+0x120/0x120 [ 4176.090842] on_each_cpu+0x51/0x120 [ 4176.090846] flush_tlb_kernel_range+0x93/0xb0 [ 4176.090851] free_unmap_vmap_area+0x63/0x70 [ 4176.090856] remove_vm_area+0xe7/0x110 [ 4176.090859] __vunmap+0x9c/0x400 [ 4176.090862] vfree+0x4d/0x100 [ 4176.090865] free_work+0x34/0x50 [libcfs] [ 4176.090891] process_one_work+0x2c8/0x700 [ 4176.090897] worker_thread+0x296/0x6e0 [ 4176.090901] ? rescuer_thread+0x570/0x570 [ 4176.090904] kthread+0x1d1/0x200 [ 4176.090908] ? set_kthread_struct+0x70/0x70 [ 4176.090911] ret_from_fork+0x1f/0x30 [ 4176.304482] ? apic_timer_interrupt+0xf/0x20 [ 4176.308248] [ 4176.308945] ? on_each_cpu_cond_mask+0x46/0x150 [ 4176.313363] ? __x64_sys_fspick+0x210/0x210 [ 4176.314932] ? smp_call_function_many_cond+0x3db/0x540 [ 4176.318094] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4176.320334] ? __x64_sys_fspick+0x210/0x210 [ 4176.322449] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4176.325088] on_each_cpu_cond_mask+0x46/0x150 [ 4176.327206] on_each_cpu_cond+0x1c/0x30 [ 4176.329221] invalidate_bh_lrus+0x27/0x30 [ 4176.331297] invalidate_bdev+0x5a/0x90 [ 4176.333292] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4176.335853] generic_shutdown_super+0xb1/0x1b0 [ 4176.337916] kill_block_super+0x29/0x80 [ 4176.339554] deactivate_locked_super+0x52/0xd0 [ 4176.341443] deactivate_super+0x83/0x90 [ 4176.343378] cleanup_mnt+0x5f/0xe0 [ 4176.345058] __cleanup_mnt+0x16/0x20 [ 4176.346640] task_work_run+0xc6/0x110 [ 4176.348291] exit_to_usermode_loop+0x1dd/0x1f0 [ 4176.350188] do_syscall_64+0x3d6/0x440 [ 4176.354953] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4176.362465] RIP: 0033:0x7fbb019648fb [ 4176.363922] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4176.376825] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4176.380871] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4176.384942] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4176.388326] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4176.391501] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4176.394789] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4188.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4188.061655] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.106386] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.115548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4188.121217] Workqueue: events free_work [libcfs] [ 4188.124818] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4188.129491] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4188.142056] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4188.148883] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4188.154034] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4188.159015] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4188.161898] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4188.164178] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4188.167068] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4188.170445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.173526] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4188.177715] Call Trace: [ 4188.179191] [ 4188.180493] ? show_regs.cold.9+0x22/0x2f [ 4188.182923] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.185623] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.187667] ? watchdog+0x40/0x40 [ 4188.189647] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.195888] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.198702] ? apic_timer_interrupt+0xf/0x20 [ 4188.203151] [ 4188.204695] ? on_each_cpu+0x51/0x120 [ 4188.207353] ? smp_call_function_many_cond+0x3db/0x540 [ 4188.210624] ? smp_call_function_many_cond+0x3a8/0x540 [ 4188.214048] ? load_new_mm_cr3+0x120/0x120 [ 4188.217047] ? load_new_mm_cr3+0x120/0x120 [ 4188.220231] on_each_cpu+0x51/0x120 [ 4188.221627] flush_tlb_kernel_range+0x93/0xb0 [ 4188.224622] free_unmap_vmap_area+0x63/0x70 [ 4188.226996] remove_vm_area+0xe7/0x110 [ 4188.228998] __vunmap+0x9c/0x400 [ 4188.231006] vfree+0x4d/0x100 [ 4188.232491] free_work+0x34/0x50 [libcfs] [ 4188.234673] process_one_work+0x2c8/0x700 [ 4188.236964] worker_thread+0x296/0x6e0 [ 4188.238952] ? rescuer_thread+0x570/0x570 [ 4188.241530] kthread+0x1d1/0x200 [ 4188.243529] ? set_kthread_struct+0x70/0x70 [ 4188.245679] ret_from_fork+0x1f/0x30 [ 4191.200057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1832s! [ 4191.208107] Showing busy workqueues and worker pools: [ 4191.213304] workqueue events: flags=0x0 [ 4191.219496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.226418] in-flight: 56293:free_work [libcfs] [ 4191.233288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4191.242400] in-flight: 22277:free_work [libcfs] [ 4191.246012] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4191.255894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.261745] pending: nfs4_renew_state [nfsv4] [ 4191.266080] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4191.273424] in-flight: 47596:free_work [libcfs] [ 4191.278761] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4191.288483] workqueue events_unbound: flags=0x2 [ 4191.293306] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4191.299042] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4191.304358] workqueue events_power_efficient: flags=0x80 [ 4191.307757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4191.312147] pending: neigh_periodic_work, do_cache_clean [ 4191.315422] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.320979] pending: check_lifetime [ 4191.325581] workqueue mm_percpu_wq: flags=0x8 [ 4191.330830] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.336872] pending: vmstat_update [ 4191.341431] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.347813] pending: vmstat_update [ 4191.351148] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.358703] pending: vmstat_update [ 4191.360564] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4191.368385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4191.374762] pending: cgroup_pidlist_destroy_work_fn [ 4191.379973] workqueue writeback: flags=0x4a [ 4191.385625] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4191.392781] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4191.402732] workqueue xprtiod: flags=0xa [ 4191.408271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4191.414112] pending: xprt_autoclose [ 4191.415549] workqueue ipv6_addrconf: flags=0x40008 [ 4191.418393] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4191.423721] pending: addrconf_verify_work [ 4191.426458] workqueue knbd0-recv: flags=0x1a [ 4191.433537] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4191.436800] in-flight: 655:recv_work [ 4191.439323] workqueue ptlrpc_pinger: flags=0x2 [ 4191.441702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4191.447501] pending: ptlrpc_pinger_main [ptlrpc] [ 4191.451745] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1833s workers=5 idle: 53684 27131 54850 46584 [ 4191.461388] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1833s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4191.470297] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1833s workers=6 idle: 53660 37674 46417 53664 46418 [ 4191.485680] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4181s workers=2 idle: 657 [ 4204.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4204.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4204.062537] Modules linked in: zfs(O) [ 4204.066728] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4204.066909] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4204.066915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4204.066923] Workqueue: events free_work [libcfs] [ 4204.066967] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4204.066976] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4204.066980] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4204.066985] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4204.066988] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4204.066990] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4204.066993] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4204.066995] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4204.066998] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4204.067001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4204.067004] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4204.067012] Call Trace: [ 4204.067017] [ 4204.067022] ? show_regs.cold.9+0x22/0x2f [ 4204.067030] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4204.067037] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4204.067041] ? watchdog+0x40/0x40 [ 4204.067045] ? hrtimer_interrupt+0x128/0x2c0 [ 4204.067050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4204.067058] ? apic_timer_interrupt+0xf/0x20 [ 4204.067063] [ 4204.067065] ? on_each_cpu+0x51/0x120 [ 4204.067217] ? smp_call_function_many_cond+0x3db/0x540 [ 4204.067221] ? smp_call_function_many_cond+0x3a8/0x540 [ 4204.067225] ? load_new_mm_cr3+0x120/0x120 [ 4204.067231] ? load_new_mm_cr3+0x120/0x120 [ 4204.067235] on_each_cpu+0x51/0x120 [ 4204.067240] flush_tlb_kernel_range+0x93/0xb0 [ 4204.067245] free_unmap_vmap_area+0x63/0x70 [ 4204.067251] remove_vm_area+0xe7/0x110 [ 4204.067255] __vunmap+0x9c/0x400 [ 4204.067259] vfree+0x4d/0x100 [ 4204.067263] free_work+0x34/0x50 [libcfs] [ 4204.067303] process_one_work+0x2c8/0x700 [ 4204.067310] worker_thread+0x296/0x6e0 [ 4204.067315] ? rescuer_thread+0x570/0x570 [ 4204.067319] kthread+0x1d1/0x200 [ 4204.067324] ? set_kthread_struct+0x70/0x70 [ 4204.067328] ret_from_fork+0x1f/0x30 [ 4204.289463] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4204.320141] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4204.327464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4204.332333] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4204.336834] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4204.350675] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4204.356704] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4204.361730] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4204.366347] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4204.369617] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4204.373476] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4204.377280] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4204.381694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4204.389340] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4204.402763] Call Trace: [ 4204.404040] [ 4204.405275] ? show_regs.cold.9+0x22/0x2f [ 4204.408222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4204.411184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4204.414016] ? watchdog+0x40/0x40 [ 4204.415286] ? hrtimer_interrupt+0x128/0x2c0 [ 4204.423222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4204.433495] ? apic_timer_interrupt+0xf/0x20 [ 4204.443146] [ 4204.448352] ? on_each_cpu_cond_mask+0x46/0x150 [ 4204.455324] ? __x64_sys_fspick+0x210/0x210 [ 4204.459110] ? smp_call_function_many_cond+0x3db/0x540 [ 4204.463038] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4204.465100] ? __x64_sys_fspick+0x210/0x210 [ 4204.468201] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4204.469948] on_each_cpu_cond_mask+0x46/0x150 [ 4204.477314] on_each_cpu_cond+0x1c/0x30 [ 4204.481835] invalidate_bh_lrus+0x27/0x30 [ 4204.491265] invalidate_bdev+0x5a/0x90 [ 4204.498033] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4204.502153] generic_shutdown_super+0xb1/0x1b0 [ 4204.506105] kill_block_super+0x29/0x80 [ 4204.507732] deactivate_locked_super+0x52/0xd0 [ 4204.511824] deactivate_super+0x83/0x90 [ 4204.515393] cleanup_mnt+0x5f/0xe0 [ 4204.518853] __cleanup_mnt+0x16/0x20 [ 4204.520340] task_work_run+0xc6/0x110 [ 4204.529570] exit_to_usermode_loop+0x1dd/0x1f0 [ 4204.533460] do_syscall_64+0x3d6/0x440 [ 4204.536773] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4204.542807] RIP: 0033:0x7fbb019648fb [ 4204.545799] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4204.559950] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4204.573333] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4204.581279] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4204.586047] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4204.590109] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4204.597291] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4216.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4216.061838] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.115228] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.122955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4216.127729] Workqueue: events free_work [libcfs] [ 4216.129246] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4216.136113] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4216.154878] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4216.164207] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4216.179924] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4216.186057] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4216.190913] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4216.195498] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4216.201743] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4216.214053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.223578] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4216.228079] Call Trace: [ 4216.229010] [ 4216.238907] ? show_regs.cold.9+0x22/0x2f [ 4216.245065] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.250085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.256591] ? watchdog+0x40/0x40 [ 4216.261437] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.265739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.272471] ? apic_timer_interrupt+0xf/0x20 [ 4216.278153] [ 4216.281291] ? on_each_cpu+0x51/0x120 [ 4216.284379] ? smp_call_function_many_cond+0x3d9/0x540 [ 4216.288001] ? smp_call_function_many_cond+0x3a8/0x540 [ 4216.297001] ? load_new_mm_cr3+0x120/0x120 [ 4216.304112] ? load_new_mm_cr3+0x120/0x120 [ 4216.305659] on_each_cpu+0x51/0x120 [ 4216.309192] flush_tlb_kernel_range+0x93/0xb0 [ 4216.312197] free_unmap_vmap_area+0x63/0x70 [ 4216.320151] remove_vm_area+0xe7/0x110 [ 4216.322854] __vunmap+0x9c/0x400 [ 4216.326043] vfree+0x4d/0x100 [ 4216.327932] free_work+0x34/0x50 [libcfs] [ 4216.331416] process_one_work+0x2c8/0x700 [ 4216.334754] worker_thread+0x296/0x6e0 [ 4216.337112] ? rescuer_thread+0x570/0x570 [ 4216.340092] kthread+0x1d1/0x200 [ 4216.344767] ? set_kthread_struct+0x70/0x70 [ 4216.349770] ret_from_fork+0x1f/0x30 [ 4217.783013] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4217.785993] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 4217.792413] (detected by 2, t=1860053 jiffies, g=626849, q=1398) [ 4217.794608] Sending NMI from CPU 2 to CPUs 3: [ 4217.797876] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 4217.797885] rcu: rcu_sched kthread starved for 1859695 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 4217.804801] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4217.808734] rcu: RCU grace-period kthread stack dump: [ 4217.810728] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4217.815643] Call Trace: [ 4217.816636] __schedule+0x351/0xcb0 [ 4217.823008] schedule+0xc0/0x180 [ 4217.827002] schedule_timeout+0xb4/0x190 [ 4217.829109] ? do_raw_spin_unlock+0x75/0x190 [ 4217.831833] ? __next_timer_interrupt+0x160/0x160 [ 4217.835541] ? prepare_to_swait_event+0xf3/0x140 [ 4217.837695] rcu_gp_kthread+0x8dd/0x15b0 [ 4217.842886] ? force_qs_rnp+0x2c0/0x2c0 [ 4217.845776] kthread+0x1d1/0x200 [ 4217.848609] ? set_kthread_struct+0x70/0x70 [ 4217.850655] ret_from_fork+0x1f/0x30 [ 4217.852296] rcu: Stack dump where RCU GP kthread last ran: [ 4217.856368] Sending NMI from CPU 2 to CPUs 0: [ 4217.859313] NMI backtrace for cpu 0 [ 4217.859319] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4217.859321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4217.859322] Workqueue: events free_work [libcfs] [ 4217.859325] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4217.859327] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4217.859329] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000212 [ 4217.859331] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4217.859332] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4217.859333] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4217.859335] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4217.859336] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4217.859337] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4217.859338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4217.859339] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4217.859341] Call Trace: [ 4217.859341] [ 4217.859342] ? show_regs.cold.9+0x22/0x2f [ 4217.859343] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4217.859344] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4217.859345] ? nmi_handle+0xbd/0x210 [ 4217.859346] ? default_do_nmi+0x4b/0x1b0 [ 4217.859347] ? do_nmi+0x280/0x380 [ 4217.859348] ? end_repeat_nmi+0x16/0x1f [ 4217.859349] ? on_each_cpu+0x51/0x120 [ 4217.859350] ? smp_call_function_many_cond+0x3db/0x540 [ 4217.859351] ? smp_call_function_many_cond+0x3db/0x540 [ 4217.859352] ? smp_call_function_many_cond+0x3db/0x540 [ 4217.859354] [ 4217.859354] ? load_new_mm_cr3+0x120/0x120 [ 4217.859355] ? load_new_mm_cr3+0x120/0x120 [ 4217.859356] on_each_cpu+0x51/0x120 [ 4217.859357] flush_tlb_kernel_range+0x93/0xb0 [ 4217.859358] free_unmap_vmap_area+0x63/0x70 [ 4217.859359] remove_vm_area+0xe7/0x110 [ 4217.859361] __vunmap+0x9c/0x400 [ 4217.859361] vfree+0x4d/0x100 [ 4217.859362] free_work+0x34/0x50 [libcfs] [ 4217.859364] process_one_work+0x2c8/0x700 [ 4217.859365] worker_thread+0x296/0x6e0 [ 4217.859366] ? rescuer_thread+0x570/0x570 [ 4217.859367] kthread+0x1d1/0x200 [ 4217.859368] ? set_kthread_struct+0x70/0x70 [ 4217.859369] ret_from_fork+0x1f/0x30 [ 4221.919068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1863s! [ 4221.921501] Showing busy workqueues and worker pools: [ 4221.923433] workqueue events: flags=0x0 [ 4221.925242] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4221.928498] in-flight: 56293:free_work [libcfs] [ 4221.930856] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4221.934494] in-flight: 22277:free_work [libcfs] [ 4221.937493] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4221.942562] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4221.945462] pending: nfs4_renew_state [nfsv4] [ 4221.947588] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4221.950617] in-flight: 47596:free_work [libcfs] [ 4221.953567] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4221.957626] workqueue events_unbound: flags=0x2 [ 4221.960988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4221.964192] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4221.968049] workqueue events_power_efficient: flags=0x80 [ 4221.970529] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4221.975574] pending: neigh_periodic_work, do_cache_clean [ 4221.978947] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4221.984227] pending: check_lifetime [ 4221.986178] workqueue mm_percpu_wq: flags=0x8 [ 4221.988379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4221.992498] pending: vmstat_update [ 4221.995227] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.001572] pending: vmstat_update [ 4222.004928] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.011008] pending: vmstat_update [ 4222.013404] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4222.016169] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4222.018861] pending: cgroup_pidlist_destroy_work_fn [ 4222.024908] workqueue writeback: flags=0x4a [ 4222.026601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4222.037348] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4222.040971] workqueue xprtiod: flags=0xa [ 4222.044850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4222.049160] pending: xprt_autoclose [ 4222.053736] workqueue ipv6_addrconf: flags=0x40008 [ 4222.055695] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4222.062464] pending: addrconf_verify_work [ 4222.065294] workqueue knbd0-recv: flags=0x1a [ 4222.071590] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4222.080594] in-flight: 655:recv_work [ 4222.082299] workqueue ptlrpc_pinger: flags=0x2 [ 4222.084071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4222.093838] pending: ptlrpc_pinger_main [ptlrpc] [ 4222.103818] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1864s workers=5 idle: 53684 27131 54850 46584 [ 4222.113078] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1864s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4222.121461] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1864s workers=6 idle: 53660 37674 46417 53664 46418 [ 4222.128418] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4211s workers=2 idle: 657 [ 4232.058102] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4232.061886] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4232.101602] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4232.113107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4232.119519] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4232.124062] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4232.138764] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4232.145642] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4232.155862] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4232.161025] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4232.166866] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4232.170692] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4232.177119] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4232.180848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4232.186101] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4232.192600] Call Trace: [ 4232.194285] [ 4232.195718] ? show_regs.cold.9+0x22/0x2f [ 4232.197220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4232.201128] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4232.204177] ? watchdog+0x40/0x40 [ 4232.205881] ? hrtimer_interrupt+0x128/0x2c0 [ 4232.210061] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4232.212718] ? apic_timer_interrupt+0xf/0x20 [ 4232.215248] [ 4232.216551] ? on_each_cpu_cond_mask+0x46/0x150 [ 4232.219518] ? __x64_sys_fspick+0x210/0x210 [ 4232.221712] ? smp_call_function_many_cond+0x3db/0x540 [ 4232.224674] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4232.226914] ? __x64_sys_fspick+0x210/0x210 [ 4232.230526] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4232.233658] on_each_cpu_cond_mask+0x46/0x150 [ 4232.236512] on_each_cpu_cond+0x1c/0x30 [ 4232.240033] invalidate_bh_lrus+0x27/0x30 [ 4232.243259] invalidate_bdev+0x5a/0x90 [ 4232.246236] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4232.249433] generic_shutdown_super+0xb1/0x1b0 [ 4232.252693] kill_block_super+0x29/0x80 [ 4232.255142] deactivate_locked_super+0x52/0xd0 [ 4232.260304] deactivate_super+0x83/0x90 [ 4232.261663] cleanup_mnt+0x5f/0xe0 [ 4232.264821] __cleanup_mnt+0x16/0x20 [ 4232.267305] task_work_run+0xc6/0x110 [ 4232.275668] exit_to_usermode_loop+0x1dd/0x1f0 [ 4232.277511] do_syscall_64+0x3d6/0x440 [ 4232.278935] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4232.286428] RIP: 0033:0x7fbb019648fb [ 4232.289390] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4232.307293] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4232.311918] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4232.315401] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4232.320295] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4232.324908] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4232.333311] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4244.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4244.061979] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4244.092310] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4244.097503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4244.102336] Workqueue: events free_work [libcfs] [ 4244.104754] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4244.107173] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4244.116006] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4244.120392] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4244.123190] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4244.126139] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4244.128219] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4244.129822] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4244.131761] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4244.134113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4244.136443] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4244.139501] Call Trace: [ 4244.140760] [ 4244.141643] ? show_regs.cold.9+0x22/0x2f [ 4244.143457] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4244.145720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4244.147777] ? watchdog+0x40/0x40 [ 4244.149236] ? hrtimer_interrupt+0x128/0x2c0 [ 4244.151283] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4244.153211] ? apic_timer_interrupt+0xf/0x20 [ 4244.155465] [ 4244.156370] ? on_each_cpu+0x51/0x120 [ 4244.158030] ? smp_call_function_many_cond+0x3db/0x540 [ 4244.159852] ? smp_call_function_many_cond+0x3a8/0x540 [ 4244.162150] ? load_new_mm_cr3+0x120/0x120 [ 4244.163886] ? load_new_mm_cr3+0x120/0x120 [ 4244.165912] on_each_cpu+0x51/0x120 [ 4244.168145] flush_tlb_kernel_range+0x93/0xb0 [ 4244.170817] free_unmap_vmap_area+0x63/0x70 [ 4244.173202] remove_vm_area+0xe7/0x110 [ 4244.175082] __vunmap+0x9c/0x400 [ 4244.176349] vfree+0x4d/0x100 [ 4244.177378] free_work+0x34/0x50 [libcfs] [ 4244.178930] process_one_work+0x2c8/0x700 [ 4244.180387] worker_thread+0x296/0x6e0 [ 4244.181839] ? rescuer_thread+0x570/0x570 [ 4244.183433] kthread+0x1d1/0x200 [ 4244.184934] ? set_kthread_struct+0x70/0x70 [ 4244.186372] ret_from_fork+0x1f/0x30 [ 4248.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4248.059461] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4248.078256] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4248.082254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4248.085437] Workqueue: events free_work [libcfs] [ 4248.088300] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4248.090421] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4248.101065] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4248.104503] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4248.107846] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4248.110978] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4248.114422] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4248.118293] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4248.121670] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4248.126373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4248.129067] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4248.132908] Call Trace: [ 4248.134116] [ 4248.136986] ? show_regs.cold.9+0x22/0x2f [ 4248.141032] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4248.143591] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4248.146996] ? watchdog+0x40/0x40 [ 4248.148782] ? hrtimer_interrupt+0x128/0x2c0 [ 4248.151930] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4248.154322] ? apic_timer_interrupt+0xf/0x20 [ 4248.156362] [ 4248.157706] ? on_each_cpu+0x51/0x120 [ 4248.159355] ? smp_call_function_many_cond+0x3db/0x540 [ 4248.161805] ? smp_call_function_many_cond+0x3a8/0x540 [ 4248.166243] ? load_new_mm_cr3+0x120/0x120 [ 4248.168779] ? load_new_mm_cr3+0x120/0x120 [ 4248.171206] on_each_cpu+0x51/0x120 [ 4248.173372] flush_tlb_kernel_range+0x93/0xb0 [ 4248.176227] free_unmap_vmap_area+0x63/0x70 [ 4248.179321] remove_vm_area+0xe7/0x110 [ 4248.181381] __vunmap+0x9c/0x400 [ 4248.183180] vfree+0x4d/0x100 [ 4248.184778] free_work+0x34/0x50 [libcfs] [ 4248.186802] process_one_work+0x2c8/0x700 [ 4248.188683] worker_thread+0x296/0x6e0 [ 4248.191125] ? rescuer_thread+0x570/0x570 [ 4248.193948] kthread+0x1d1/0x200 [ 4248.196407] ? set_kthread_struct+0x70/0x70 [ 4248.198287] ret_from_fork+0x1f/0x30 [ 4252.639062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1894s! [ 4252.642895] Showing busy workqueues and worker pools: [ 4252.645164] workqueue events: flags=0x0 [ 4252.647182] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4252.650253] in-flight: 56293:free_work [libcfs] [ 4252.652476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4252.655221] in-flight: 22277:free_work [libcfs] [ 4252.657272] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4252.661300] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4252.664202] pending: nfs4_renew_state [nfsv4] [ 4252.669597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4252.678966] in-flight: 47596:free_work [libcfs] [ 4252.686126] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4252.691054] workqueue events_unbound: flags=0x2 [ 4252.695480] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4252.698626] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4252.706323] workqueue events_power_efficient: flags=0x80 [ 4252.709861] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4252.717498] pending: neigh_periodic_work, do_cache_clean [ 4252.721963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4252.727512] pending: check_lifetime [ 4252.733257] workqueue mm_percpu_wq: flags=0x8 [ 4252.735765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4252.739204] pending: vmstat_update [ 4252.741075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4252.745135] pending: vmstat_update [ 4252.747141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4252.750752] pending: vmstat_update [ 4252.752632] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4252.755601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4252.759099] pending: cgroup_pidlist_destroy_work_fn [ 4252.761617] workqueue writeback: flags=0x4a [ 4252.763508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4252.766812] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4252.771949] workqueue xprtiod: flags=0xa [ 4252.774543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4252.778176] pending: xprt_autoclose [ 4252.780484] workqueue ipv6_addrconf: flags=0x40008 [ 4252.784337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4252.788138] pending: addrconf_verify_work [ 4252.790607] workqueue knbd0-recv: flags=0x1a [ 4252.792748] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4252.796703] in-flight: 655:recv_work [ 4252.798669] workqueue ptlrpc_pinger: flags=0x2 [ 4252.802091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4252.805760] pending: ptlrpc_pinger_main [ptlrpc] [ 4252.808736] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1895s workers=5 idle: 53684 27131 54850 46584 [ 4252.814192] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1895s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4252.818801] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1895s workers=6 idle: 53660 37674 46417 53664 46418 [ 4252.835308] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4242s workers=2 idle: 657 [ 4260.059491] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4260.066258] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4260.101765] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4260.109067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4260.114109] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4260.118548] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4260.130606] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4260.135561] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4260.142647] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4260.147423] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4260.149754] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4260.156690] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4260.162128] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4260.170730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4260.176475] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4260.183982] Call Trace: [ 4260.185479] [ 4260.186379] ? show_regs.cold.9+0x22/0x2f [ 4260.188075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4260.190898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4260.193578] ? watchdog+0x40/0x40 [ 4260.195494] ? hrtimer_interrupt+0x128/0x2c0 [ 4260.197751] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4260.200048] ? apic_timer_interrupt+0xf/0x20 [ 4260.202453] [ 4260.203520] ? on_each_cpu_cond_mask+0x46/0x150 [ 4260.206087] ? __x64_sys_fspick+0x210/0x210 [ 4260.212446] ? smp_call_function_many_cond+0x3db/0x540 [ 4260.216508] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4260.219133] ? __x64_sys_fspick+0x210/0x210 [ 4260.225163] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4260.232024] on_each_cpu_cond_mask+0x46/0x150 [ 4260.236862] on_each_cpu_cond+0x1c/0x30 [ 4260.242110] invalidate_bh_lrus+0x27/0x30 [ 4260.243936] invalidate_bdev+0x5a/0x90 [ 4260.248082] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4260.251105] generic_shutdown_super+0xb1/0x1b0 [ 4260.256035] kill_block_super+0x29/0x80 [ 4260.260330] deactivate_locked_super+0x52/0xd0 [ 4260.262756] deactivate_super+0x83/0x90 [ 4260.265019] cleanup_mnt+0x5f/0xe0 [ 4260.267783] __cleanup_mnt+0x16/0x20 [ 4260.270115] task_work_run+0xc6/0x110 [ 4260.272359] exit_to_usermode_loop+0x1dd/0x1f0 [ 4260.274566] do_syscall_64+0x3d6/0x440 [ 4260.276034] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4260.278367] RIP: 0033:0x7fbb019648fb [ 4260.280437] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4260.290992] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4260.296304] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4260.301509] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4260.306537] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4260.311064] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4260.316071] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4272.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4272.062627] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4272.084807] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4272.090195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4272.094570] Workqueue: events free_work [libcfs] [ 4272.096612] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4272.099169] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4272.113189] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4272.116697] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4272.120810] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4272.124522] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4272.127296] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4272.129935] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4272.133045] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4272.136656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4272.140587] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4272.144129] Call Trace: [ 4272.146179] [ 4272.146969] ? show_regs.cold.9+0x22/0x2f [ 4272.149722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4272.152242] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4272.154534] ? watchdog+0x40/0x40 [ 4272.157051] ? hrtimer_interrupt+0x128/0x2c0 [ 4272.160884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4272.165851] ? apic_timer_interrupt+0xf/0x20 [ 4272.170337] [ 4272.172028] ? on_each_cpu+0x51/0x120 [ 4272.173714] ? smp_call_function_many_cond+0x3db/0x540 [ 4272.177240] ? smp_call_function_many_cond+0x3a8/0x540 [ 4272.180336] ? load_new_mm_cr3+0x120/0x120 [ 4272.183305] ? load_new_mm_cr3+0x120/0x120 [ 4272.184895] on_each_cpu+0x51/0x120 [ 4272.186595] flush_tlb_kernel_range+0x93/0xb0 [ 4272.189967] free_unmap_vmap_area+0x63/0x70 [ 4272.192322] remove_vm_area+0xe7/0x110 [ 4272.194168] __vunmap+0x9c/0x400 [ 4272.195701] vfree+0x4d/0x100 [ 4272.197294] free_work+0x34/0x50 [libcfs] [ 4272.198542] process_one_work+0x2c8/0x700 [ 4272.200564] worker_thread+0x296/0x6e0 [ 4272.204574] ? rescuer_thread+0x570/0x570 [ 4272.205874] kthread+0x1d1/0x200 [ 4272.207364] ? set_kthread_struct+0x70/0x70 [ 4272.208833] ret_from_fork+0x1f/0x30 [ 4280.055366] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4280.061713] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4280.107146] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4280.114594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4280.120978] Workqueue: events free_work [libcfs] [ 4280.123613] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4280.129098] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4280.139638] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4280.143091] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4280.146520] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4280.150238] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4280.154160] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4280.157509] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4280.161145] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4280.165133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4280.167817] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4280.171338] Call Trace: [ 4280.172519] [ 4280.173461] ? show_regs.cold.9+0x22/0x2f [ 4280.175325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4280.177462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4280.179570] ? watchdog+0x40/0x40 [ 4280.181337] ? hrtimer_interrupt+0x128/0x2c0 [ 4280.183246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4280.185519] ? apic_timer_interrupt+0xf/0x20 [ 4280.187876] [ 4280.188954] ? on_each_cpu+0x51/0x120 [ 4280.190179] ? smp_call_function_many_cond+0x3d9/0x540 [ 4280.192506] ? smp_call_function_many_cond+0x3a8/0x540 [ 4280.195132] ? load_new_mm_cr3+0x120/0x120 [ 4280.197416] ? load_new_mm_cr3+0x120/0x120 [ 4280.199178] on_each_cpu+0x51/0x120 [ 4280.201222] flush_tlb_kernel_range+0x93/0xb0 [ 4280.203480] free_unmap_vmap_area+0x63/0x70 [ 4280.205818] remove_vm_area+0xe7/0x110 [ 4280.208378] __vunmap+0x9c/0x400 [ 4280.209689] vfree+0x4d/0x100 [ 4280.210936] free_work+0x34/0x50 [libcfs] [ 4280.212652] process_one_work+0x2c8/0x700 [ 4280.214301] worker_thread+0x296/0x6e0 [ 4280.216691] ? rescuer_thread+0x570/0x570 [ 4280.218753] kthread+0x1d1/0x200 [ 4280.221118] ? set_kthread_struct+0x70/0x70 [ 4280.225671] ret_from_fork+0x1f/0x30 [ 4282.848696] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1924s! [ 4282.855475] Showing busy workqueues and worker pools: [ 4282.862291] workqueue events: flags=0x0 [ 4282.864345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4282.867284] in-flight: 56293:free_work [libcfs] [ 4282.871397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4282.877503] in-flight: 22277:free_work [libcfs] [ 4282.879856] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4282.885539] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4282.888456] pending: nfs4_renew_state [nfsv4] [ 4282.890793] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4282.893675] in-flight: 47596:free_work [libcfs] [ 4282.895761] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4282.899466] workqueue events_unbound: flags=0x2 [ 4282.901366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4282.903995] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4282.907822] workqueue events_power_efficient: flags=0x80 [ 4282.909828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4282.912287] pending: neigh_periodic_work, do_cache_clean [ 4282.914778] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4282.921077] pending: check_lifetime [ 4282.923329] workqueue mm_percpu_wq: flags=0x8 [ 4282.924783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4282.926631] pending: vmstat_update [ 4282.928144] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4282.931390] pending: vmstat_update [ 4282.933153] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4282.936021] pending: vmstat_update [ 4282.937368] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4282.939845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4282.942456] pending: cgroup_pidlist_destroy_work_fn [ 4282.945838] workqueue writeback: flags=0x4a [ 4282.949640] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4282.953998] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4282.957955] workqueue xprtiod: flags=0xa [ 4282.959386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4282.961544] pending: xprt_autoclose [ 4282.963007] workqueue ipv6_addrconf: flags=0x40008 [ 4282.964501] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4282.966553] pending: addrconf_verify_work [ 4282.968063] workqueue knbd0-recv: flags=0x1a [ 4282.969374] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4282.971617] in-flight: 655:recv_work [ 4282.972985] workqueue ptlrpc_pinger: flags=0x2 [ 4282.974370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4282.976386] pending: ptlrpc_pinger_main [ptlrpc] [ 4282.978499] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1925s workers=5 idle: 53684 27131 54850 46584 [ 4282.982164] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1925s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4282.987172] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1925s workers=6 idle: 53660 37674 46417 53664 46418 [ 4282.996461] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4272s workers=2 idle: 657 [ 4288.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4288.062731] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4288.096005] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4288.102635] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4288.112295] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4288.117496] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4288.133060] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4288.137053] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4288.148153] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4288.155652] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4288.159427] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4288.163160] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4288.177349] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4288.183221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4288.186844] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4288.191129] Call Trace: [ 4288.192894] [ 4288.194195] ? show_regs.cold.9+0x22/0x2f [ 4288.201359] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4288.206005] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4288.210503] ? watchdog+0x40/0x40 [ 4288.212496] ? hrtimer_interrupt+0x128/0x2c0 [ 4288.214840] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4288.217168] ? apic_timer_interrupt+0xf/0x20 [ 4288.219484] [ 4288.220498] ? on_each_cpu_cond_mask+0x46/0x150 [ 4288.222491] ? __x64_sys_fspick+0x210/0x210 [ 4288.224529] ? smp_call_function_many_cond+0x3d9/0x540 [ 4288.227094] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4288.229181] ? __x64_sys_fspick+0x210/0x210 [ 4288.231162] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4288.233501] on_each_cpu_cond_mask+0x46/0x150 [ 4288.235962] on_each_cpu_cond+0x1c/0x30 [ 4288.237943] invalidate_bh_lrus+0x27/0x30 [ 4288.239851] invalidate_bdev+0x5a/0x90 [ 4288.241654] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4288.244512] generic_shutdown_super+0xb1/0x1b0 [ 4288.246955] kill_block_super+0x29/0x80 [ 4288.248874] deactivate_locked_super+0x52/0xd0 [ 4288.252352] deactivate_super+0x83/0x90 [ 4288.253859] cleanup_mnt+0x5f/0xe0 [ 4288.258500] __cleanup_mnt+0x16/0x20 [ 4288.259864] task_work_run+0xc6/0x110 [ 4288.272114] exit_to_usermode_loop+0x1dd/0x1f0 [ 4288.276022] do_syscall_64+0x3d6/0x440 [ 4288.282532] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4288.291657] RIP: 0033:0x7fbb019648fb [ 4288.296688] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4288.310255] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4288.313748] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4288.317254] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4288.321484] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4288.325392] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4288.329080] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4300.059115] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4300.066137] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4300.098672] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4300.105837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4300.109140] Workqueue: events free_work [libcfs] [ 4300.111772] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4300.117767] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4300.127128] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4300.132409] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4300.137599] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4300.142475] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4300.147572] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4300.152921] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4300.157629] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4300.162651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4300.164473] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4300.168309] Call Trace: [ 4300.170484] [ 4300.171336] ? show_regs.cold.9+0x22/0x2f [ 4300.173703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4300.177678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4300.181218] ? watchdog+0x40/0x40 [ 4300.182597] ? hrtimer_interrupt+0x128/0x2c0 [ 4300.184862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4300.187747] ? apic_timer_interrupt+0xf/0x20 [ 4300.191549] [ 4300.192779] ? on_each_cpu+0x51/0x120 [ 4300.195642] ? smp_call_function_many_cond+0x3d9/0x540 [ 4300.198868] ? smp_call_function_many_cond+0x3a8/0x540 [ 4300.202982] ? load_new_mm_cr3+0x120/0x120 [ 4300.205270] ? load_new_mm_cr3+0x120/0x120 [ 4300.207131] on_each_cpu+0x51/0x120 [ 4300.208848] flush_tlb_kernel_range+0x93/0xb0 [ 4300.210906] free_unmap_vmap_area+0x63/0x70 [ 4300.212855] remove_vm_area+0xe7/0x110 [ 4300.215269] __vunmap+0x9c/0x400 [ 4300.216525] vfree+0x4d/0x100 [ 4300.218860] free_work+0x34/0x50 [libcfs] [ 4300.222492] process_one_work+0x2c8/0x700 [ 4300.224007] worker_thread+0x296/0x6e0 [ 4300.229955] ? rescuer_thread+0x570/0x570 [ 4300.232979] kthread+0x1d1/0x200 [ 4300.236911] ? set_kthread_struct+0x70/0x70 [ 4300.240048] ret_from_fork+0x1f/0x30 [ 4308.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4308.061489] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4308.089117] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4308.097233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4308.101846] Workqueue: events free_work [libcfs] [ 4308.105418] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4308.108864] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4308.121651] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4308.128996] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4308.136937] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4308.145632] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4308.148846] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4308.156049] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4308.162046] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4308.165740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4308.168952] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4308.176489] Call Trace: [ 4308.178063] [ 4308.180244] ? show_regs.cold.9+0x22/0x2f [ 4308.182507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4308.185033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4308.188409] ? watchdog+0x40/0x40 [ 4308.191464] ? hrtimer_interrupt+0x128/0x2c0 [ 4308.193717] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4308.197437] ? apic_timer_interrupt+0xf/0x20 [ 4308.200761] [ 4308.202206] ? on_each_cpu+0x51/0x120 [ 4308.205367] ? smp_call_function_many_cond+0x3db/0x540 [ 4308.210736] ? smp_call_function_many_cond+0x3a8/0x540 [ 4308.217074] ? load_new_mm_cr3+0x120/0x120 [ 4308.224534] ? load_new_mm_cr3+0x120/0x120 [ 4308.226081] on_each_cpu+0x51/0x120 [ 4308.229696] flush_tlb_kernel_range+0x93/0xb0 [ 4308.232769] free_unmap_vmap_area+0x63/0x70 [ 4308.235541] remove_vm_area+0xe7/0x110 [ 4308.236944] __vunmap+0x9c/0x400 [ 4308.242977] vfree+0x4d/0x100 [ 4308.246034] free_work+0x34/0x50 [libcfs] [ 4308.248994] process_one_work+0x2c8/0x700 [ 4308.251946] worker_thread+0x296/0x6e0 [ 4308.254637] ? rescuer_thread+0x570/0x570 [ 4308.258988] kthread+0x1d1/0x200 [ 4308.263082] ? set_kthread_struct+0x70/0x70 [ 4308.268029] ret_from_fork+0x1f/0x30 [ 4313.056715] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1954s! [ 4313.065747] Showing busy workqueues and worker pools: [ 4313.070274] workqueue events: flags=0x0 [ 4313.076776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4313.079906] in-flight: 56293:free_work [libcfs] [ 4313.083889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4313.090605] in-flight: 22277:free_work [libcfs] [ 4313.095831] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4313.108701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4313.113024] pending: nfs4_renew_state [nfsv4] [ 4313.115384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4313.124562] in-flight: 47596:free_work [libcfs] [ 4313.128103] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4313.136205] workqueue events_unbound: flags=0x2 [ 4313.138713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4313.142301] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4313.146593] workqueue events_power_efficient: flags=0x80 [ 4313.149174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4313.152892] pending: neigh_periodic_work, do_cache_clean [ 4313.155769] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4313.159124] pending: check_lifetime [ 4313.161174] workqueue mm_percpu_wq: flags=0x8 [ 4313.163128] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4313.166215] pending: vmstat_update [ 4313.167773] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4313.171639] pending: vmstat_update [ 4313.173712] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4313.179359] pending: vmstat_update [ 4313.184638] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4313.187130] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4313.192657] pending: cgroup_pidlist_destroy_work_fn [ 4313.195973] workqueue writeback: flags=0x4a [ 4313.203019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4313.209326] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4313.215787] workqueue xprtiod: flags=0xa [ 4313.221218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4313.225294] pending: xprt_autoclose [ 4313.226951] workqueue ipv6_addrconf: flags=0x40008 [ 4313.229723] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4313.236566] pending: addrconf_verify_work [ 4313.240374] workqueue knbd0-recv: flags=0x1a [ 4313.244244] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4313.248789] in-flight: 655:recv_work [ 4313.254651] workqueue ptlrpc_pinger: flags=0x2 [ 4313.261453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4313.268743] pending: ptlrpc_pinger_main [ptlrpc] [ 4313.278351] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1955s workers=5 idle: 53684 27131 54850 46584 [ 4313.290689] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1955s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4313.302542] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1955s workers=6 idle: 53660 37674 46417 53664 46418 [ 4313.311590] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4303s workers=2 idle: 657 [ 4316.058034] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4316.065621] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4316.104594] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4316.112574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4316.119649] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4316.124629] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4316.136923] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4316.142781] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4316.147776] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4316.153063] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4316.155481] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4316.163538] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4316.169067] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4316.175740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4316.178585] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4316.181894] Call Trace: [ 4316.182938] [ 4316.183816] ? show_regs.cold.9+0x22/0x2f [ 4316.185532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4316.187752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4316.189666] ? watchdog+0x40/0x40 [ 4316.191231] ? hrtimer_interrupt+0x128/0x2c0 [ 4316.192935] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4316.194900] ? apic_timer_interrupt+0xf/0x20 [ 4316.196767] [ 4316.197701] ? on_each_cpu_cond_mask+0x46/0x150 [ 4316.199698] ? __x64_sys_fspick+0x210/0x210 [ 4316.201741] ? smp_call_function_many_cond+0x3db/0x540 [ 4316.203923] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4316.205700] ? __x64_sys_fspick+0x210/0x210 [ 4316.207479] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4316.208866] on_each_cpu_cond_mask+0x46/0x150 [ 4316.210752] on_each_cpu_cond+0x1c/0x30 [ 4316.212346] invalidate_bh_lrus+0x27/0x30 [ 4316.213849] invalidate_bdev+0x5a/0x90 [ 4316.215767] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4316.217905] generic_shutdown_super+0xb1/0x1b0 [ 4316.219530] kill_block_super+0x29/0x80 [ 4316.220926] deactivate_locked_super+0x52/0xd0 [ 4316.222840] deactivate_super+0x83/0x90 [ 4316.224637] cleanup_mnt+0x5f/0xe0 [ 4316.226249] __cleanup_mnt+0x16/0x20 [ 4316.227815] task_work_run+0xc6/0x110 [ 4316.229340] exit_to_usermode_loop+0x1dd/0x1f0 [ 4316.231273] do_syscall_64+0x3d6/0x440 [ 4316.233099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4316.241376] RIP: 0033:0x7fbb019648fb [ 4316.242664] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4316.263351] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4316.275547] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4316.281204] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4316.286556] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4316.290753] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4316.294582] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4328.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4328.064173] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4328.115934] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4328.121795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4328.126196] Workqueue: events free_work [libcfs] [ 4328.128842] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4328.133821] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4328.143198] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4328.147039] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4328.150237] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4328.157376] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4328.162256] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4328.169087] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4328.175194] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4328.180198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4328.189120] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4328.193366] Call Trace: [ 4328.194483] [ 4328.195943] ? show_regs.cold.9+0x22/0x2f [ 4328.200051] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4328.205402] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4328.207821] ? watchdog+0x40/0x40 [ 4328.208916] ? hrtimer_interrupt+0x128/0x2c0 [ 4328.210859] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4328.212488] ? apic_timer_interrupt+0xf/0x20 [ 4328.216176] [ 4328.216911] ? on_each_cpu+0x51/0x120 [ 4328.218237] ? smp_call_function_many_cond+0x3db/0x540 [ 4328.221692] ? smp_call_function_many_cond+0x3a8/0x540 [ 4328.223482] ? load_new_mm_cr3+0x120/0x120 [ 4328.225306] ? load_new_mm_cr3+0x120/0x120 [ 4328.226933] on_each_cpu+0x51/0x120 [ 4328.228642] flush_tlb_kernel_range+0x93/0xb0 [ 4328.230497] free_unmap_vmap_area+0x63/0x70 [ 4328.232472] remove_vm_area+0xe7/0x110 [ 4328.234152] __vunmap+0x9c/0x400 [ 4328.235611] vfree+0x4d/0x100 [ 4328.240107] free_work+0x34/0x50 [libcfs] [ 4328.241851] process_one_work+0x2c8/0x700 [ 4328.243855] worker_thread+0x296/0x6e0 [ 4328.245527] ? rescuer_thread+0x570/0x570 [ 4328.247946] kthread+0x1d1/0x200 [ 4328.249300] ? set_kthread_struct+0x70/0x70 [ 4328.251385] ret_from_fork+0x1f/0x30 [ 4340.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4340.058763] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.088085] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.097922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4340.103962] Workqueue: events free_work [libcfs] [ 4340.106975] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4340.109332] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4340.117867] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4340.121463] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4340.125332] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4340.129102] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4340.134351] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4340.141823] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4340.145410] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4340.151940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.154208] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4340.157617] Call Trace: [ 4340.158726] [ 4340.159704] ? show_regs.cold.9+0x22/0x2f [ 4340.161330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.163381] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.165293] ? watchdog+0x40/0x40 [ 4340.166841] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.168641] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.170527] ? apic_timer_interrupt+0xf/0x20 [ 4340.172344] [ 4340.173225] ? on_each_cpu+0x51/0x120 [ 4340.175017] ? smp_call_function_many_cond+0x3d9/0x540 [ 4340.177155] ? smp_call_function_many_cond+0x3a8/0x540 [ 4340.179355] ? load_new_mm_cr3+0x120/0x120 [ 4340.181114] ? load_new_mm_cr3+0x120/0x120 [ 4340.182832] on_each_cpu+0x51/0x120 [ 4340.184578] flush_tlb_kernel_range+0x93/0xb0 [ 4340.186371] free_unmap_vmap_area+0x63/0x70 [ 4340.188167] remove_vm_area+0xe7/0x110 [ 4340.189898] __vunmap+0x9c/0x400 [ 4340.191178] vfree+0x4d/0x100 [ 4340.192414] free_work+0x34/0x50 [libcfs] [ 4340.194156] process_one_work+0x2c8/0x700 [ 4340.195642] worker_thread+0x296/0x6e0 [ 4340.197390] ? rescuer_thread+0x570/0x570 [ 4340.199060] kthread+0x1d1/0x200 [ 4340.200312] ? set_kthread_struct+0x70/0x70 [ 4340.202037] ret_from_fork+0x1f/0x30 [ 4343.775025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1985s! [ 4343.777146] Showing busy workqueues and worker pools: [ 4343.780646] workqueue events: flags=0x0 [ 4343.782328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4343.785972] in-flight: 56293:free_work [libcfs] [ 4343.787992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4343.790215] in-flight: 22277:free_work [libcfs] [ 4343.792074] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4343.795863] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4343.799075] pending: nfs4_renew_state [nfsv4] [ 4343.800944] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4343.803677] in-flight: 47596:free_work [libcfs] [ 4343.806360] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4343.810869] workqueue events_unbound: flags=0x2 [ 4343.813539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4343.815709] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4343.820789] workqueue events_power_efficient: flags=0x80 [ 4343.822539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4343.825828] pending: neigh_periodic_work, do_cache_clean [ 4343.828487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4343.830943] pending: check_lifetime [ 4343.832109] workqueue mm_percpu_wq: flags=0x8 [ 4343.834035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4343.837009] pending: vmstat_update [ 4343.838850] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4343.841853] pending: vmstat_update [ 4343.843649] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4343.847035] pending: vmstat_update [ 4343.848804] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4343.851495] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4343.854476] pending: cgroup_pidlist_destroy_work_fn [ 4343.857221] workqueue writeback: flags=0x4a [ 4343.858895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4343.861708] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4343.866703] workqueue xprtiod: flags=0xa [ 4343.869011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4343.871687] pending: xprt_autoclose [ 4343.873505] workqueue ipv6_addrconf: flags=0x40008 [ 4343.875794] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4343.878997] pending: addrconf_verify_work [ 4343.881030] workqueue knbd0-recv: flags=0x1a [ 4343.883362] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4343.886833] in-flight: 655:recv_work [ 4343.888898] workqueue ptlrpc_pinger: flags=0x2 [ 4343.890914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4343.893950] pending: ptlrpc_pinger_main [ptlrpc] [ 4343.897019] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1986s workers=5 idle: 53684 27131 54850 46584 [ 4343.902272] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=1986s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4343.909502] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=1986s workers=6 idle: 53660 37674 46417 53664 46418 [ 4343.915680] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4333s workers=2 idle: 657 [ 4344.058409] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4344.062777] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4344.084075] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4344.092495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4344.098069] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4344.101536] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4344.113602] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4344.117189] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4344.120073] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4344.123611] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4344.126575] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4344.129261] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4344.137627] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4344.141760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4344.146920] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4344.150148] Call Trace: [ 4344.151336] [ 4344.152965] ? show_regs.cold.9+0x22/0x2f [ 4344.156639] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4344.159021] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4344.161190] ? watchdog+0x40/0x40 [ 4344.163037] ? hrtimer_interrupt+0x128/0x2c0 [ 4344.165055] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4344.167089] ? apic_timer_interrupt+0xf/0x20 [ 4344.170115] [ 4344.173071] ? on_each_cpu_cond_mask+0x46/0x150 [ 4344.175860] ? __x64_sys_fspick+0x210/0x210 [ 4344.179563] ? smp_call_function_many_cond+0x3d9/0x540 [ 4344.181910] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4344.183836] ? __x64_sys_fspick+0x210/0x210 [ 4344.188566] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4344.192719] on_each_cpu_cond_mask+0x46/0x150 [ 4344.196451] on_each_cpu_cond+0x1c/0x30 [ 4344.200554] invalidate_bh_lrus+0x27/0x30 [ 4344.203119] invalidate_bdev+0x5a/0x90 [ 4344.206182] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4344.208663] generic_shutdown_super+0xb1/0x1b0 [ 4344.210643] kill_block_super+0x29/0x80 [ 4344.212108] deactivate_locked_super+0x52/0xd0 [ 4344.213884] deactivate_super+0x83/0x90 [ 4344.215601] cleanup_mnt+0x5f/0xe0 [ 4344.216965] __cleanup_mnt+0x16/0x20 [ 4344.218594] task_work_run+0xc6/0x110 [ 4344.220254] exit_to_usermode_loop+0x1dd/0x1f0 [ 4344.222341] do_syscall_64+0x3d6/0x440 [ 4344.224183] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4344.226704] RIP: 0033:0x7fbb019648fb [ 4344.228488] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4344.237134] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4344.239755] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4344.242403] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4344.246696] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4344.250887] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4344.254893] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4356.061831] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4356.067310] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4356.113609] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4356.121378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4356.130755] Workqueue: events free_work [libcfs] [ 4356.132413] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4356.139445] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4356.157404] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4356.169507] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4356.176616] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4356.180867] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4356.187315] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4356.191991] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4356.201169] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4356.204140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4356.210898] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4356.219394] Call Trace: [ 4356.223783] [ 4356.226086] ? show_regs.cold.9+0x22/0x2f [ 4356.229655] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4356.234853] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4356.236546] ? watchdog+0x40/0x40 [ 4356.239995] ? hrtimer_interrupt+0x128/0x2c0 [ 4356.243755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4356.248080] ? apic_timer_interrupt+0xf/0x20 [ 4356.252101] [ 4356.253949] ? on_each_cpu+0x51/0x120 [ 4356.255284] ? smp_call_function_many_cond+0x3db/0x540 [ 4356.264830] ? smp_call_function_many_cond+0x3a8/0x540 [ 4356.266550] ? load_new_mm_cr3+0x120/0x120 [ 4356.277836] ? load_new_mm_cr3+0x120/0x120 [ 4356.282167] on_each_cpu+0x51/0x120 [ 4356.289824] flush_tlb_kernel_range+0x93/0xb0 [ 4356.292575] free_unmap_vmap_area+0x63/0x70 [ 4356.298823] remove_vm_area+0xe7/0x110 [ 4356.301680] __vunmap+0x9c/0x400 [ 4356.303658] vfree+0x4d/0x100 [ 4356.305276] free_work+0x34/0x50 [libcfs] [ 4356.308778] process_one_work+0x2c8/0x700 [ 4356.313820] worker_thread+0x296/0x6e0 [ 4356.317540] ? rescuer_thread+0x570/0x570 [ 4356.320487] kthread+0x1d1/0x200 [ 4356.321663] ? set_kthread_struct+0x70/0x70 [ 4356.322911] ret_from_fork+0x1f/0x30 [ 4368.055129] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4368.061327] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.101612] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.110799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4368.116609] Workqueue: events free_work [libcfs] [ 4368.120821] RIP: 0010:smp_call_function_many_cond+0x3dd/0x540 [ 4368.125599] Code: 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 8b 11 <83> e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 01 eb [ 4368.140394] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4368.145123] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4368.151440] RDX: 0000000000000011 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4368.156909] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4368.162602] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4368.168222] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4368.173798] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4368.180793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.186170] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4368.191161] Call Trace: [ 4368.192259] [ 4368.192936] ? show_regs.cold.9+0x22/0x2f [ 4368.197308] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.201277] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.204574] ? watchdog+0x40/0x40 [ 4368.207756] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.211552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.215135] ? apic_timer_interrupt+0xf/0x20 [ 4368.217272] [ 4368.218173] ? on_each_cpu+0x51/0x120 [ 4368.220526] ? smp_call_function_many_cond+0x3dd/0x540 [ 4368.223976] ? smp_call_function_many_cond+0x3a8/0x540 [ 4368.227529] ? load_new_mm_cr3+0x120/0x120 [ 4368.230453] ? load_new_mm_cr3+0x120/0x120 [ 4368.233379] on_each_cpu+0x51/0x120 [ 4368.235581] flush_tlb_kernel_range+0x93/0xb0 [ 4368.238546] free_unmap_vmap_area+0x63/0x70 [ 4368.243831] remove_vm_area+0xe7/0x110 [ 4368.246829] __vunmap+0x9c/0x400 [ 4368.249472] vfree+0x4d/0x100 [ 4368.251595] free_work+0x34/0x50 [libcfs] [ 4368.254551] process_one_work+0x2c8/0x700 [ 4368.257459] worker_thread+0x296/0x6e0 [ 4368.260345] ? rescuer_thread+0x570/0x570 [ 4368.262833] kthread+0x1d1/0x200 [ 4368.265281] ? set_kthread_struct+0x70/0x70 [ 4368.268302] ret_from_fork+0x1f/0x30 [ 4372.058205] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4372.066628] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4372.101303] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4372.107723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4372.114036] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4372.115559] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4372.128137] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4372.131678] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4372.134469] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4372.139472] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4372.145611] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4372.152965] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4372.161285] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4372.167483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4372.172246] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4372.181855] Call Trace: [ 4372.183583] [ 4372.184715] ? show_regs.cold.9+0x22/0x2f [ 4372.188157] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4372.191207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4372.195031] ? watchdog+0x40/0x40 [ 4372.196749] ? hrtimer_interrupt+0x128/0x2c0 [ 4372.199150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4372.201713] ? apic_timer_interrupt+0xf/0x20 [ 4372.204344] [ 4372.205209] ? on_each_cpu_cond_mask+0x46/0x150 [ 4372.207360] ? __x64_sys_fspick+0x210/0x210 [ 4372.209011] ? smp_call_function_many_cond+0x3db/0x540 [ 4372.211529] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4372.213690] ? __x64_sys_fspick+0x210/0x210 [ 4372.216013] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4372.218161] on_each_cpu_cond_mask+0x46/0x150 [ 4372.220092] on_each_cpu_cond+0x1c/0x30 [ 4372.221744] invalidate_bh_lrus+0x27/0x30 [ 4372.223891] invalidate_bdev+0x5a/0x90 [ 4372.225600] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4372.229462] generic_shutdown_super+0xb1/0x1b0 [ 4372.231450] kill_block_super+0x29/0x80 [ 4372.233198] deactivate_locked_super+0x52/0xd0 [ 4372.235166] deactivate_super+0x83/0x90 [ 4372.237234] cleanup_mnt+0x5f/0xe0 [ 4372.238971] __cleanup_mnt+0x16/0x20 [ 4372.241479] task_work_run+0xc6/0x110 [ 4372.242880] exit_to_usermode_loop+0x1dd/0x1f0 [ 4372.245937] do_syscall_64+0x3d6/0x440 [ 4372.247554] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4372.251846] RIP: 0033:0x7fbb019648fb [ 4372.258095] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4372.273713] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4372.278091] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4372.287617] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4372.290516] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4372.294485] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4372.298106] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4373.985296] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2015s! [ 4373.994261] Showing busy workqueues and worker pools: [ 4374.005822] workqueue events: flags=0x0 [ 4374.007603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4374.021997] in-flight: 56293:free_work [libcfs] [ 4374.031078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4374.039136] in-flight: 22277:free_work [libcfs] [ 4374.043922] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4374.054092] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4374.062210] pending: nfs4_renew_state [nfsv4] [ 4374.067867] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4374.074196] in-flight: 47596:free_work [libcfs] [ 4374.080300] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4374.087377] workqueue events_unbound: flags=0x2 [ 4374.089950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4374.098068] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4374.102918] workqueue events_power_efficient: flags=0x80 [ 4374.106292] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4374.114323] pending: neigh_periodic_work, do_cache_clean [ 4374.116785] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4374.123109] pending: check_lifetime [ 4374.125840] workqueue mm_percpu_wq: flags=0x8 [ 4374.128681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4374.130801] pending: vmstat_update [ 4374.131795] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4374.145041] pending: vmstat_update [ 4374.150532] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4374.154183] pending: vmstat_update [ 4374.157123] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4374.160870] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4374.165141] pending: cgroup_pidlist_destroy_work_fn [ 4374.168097] workqueue writeback: flags=0x4a [ 4374.170627] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4374.174177] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4374.178061] workqueue xprtiod: flags=0xa [ 4374.180079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4374.184345] pending: xprt_autoclose [ 4374.187077] workqueue ipv6_addrconf: flags=0x40008 [ 4374.189368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4374.192718] pending: addrconf_verify_work [ 4374.194994] workqueue knbd0-recv: flags=0x1a [ 4374.197686] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4374.200355] in-flight: 655:recv_work [ 4374.202697] workqueue ptlrpc_pinger: flags=0x2 [ 4374.205123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4374.209229] pending: ptlrpc_pinger_main [ptlrpc] [ 4374.212659] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2016s workers=5 idle: 53684 27131 54850 46584 [ 4374.219871] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2016s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4374.231612] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2016s workers=6 idle: 53660 37674 46417 53664 46418 [ 4374.244643] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4364s workers=2 idle: 657 [ 4384.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4384.064534] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4384.090776] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4384.095068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4384.098387] Workqueue: events free_work [libcfs] [ 4384.099813] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4384.101729] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4384.108193] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4384.110489] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4384.113199] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4384.115738] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4384.120343] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4384.125407] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4384.131065] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4384.136159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4384.138519] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4384.142454] Call Trace: [ 4384.143473] [ 4384.144434] ? show_regs.cold.9+0x22/0x2f [ 4384.146385] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4384.149104] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4384.150958] ? watchdog+0x40/0x40 [ 4384.152109] ? hrtimer_interrupt+0x128/0x2c0 [ 4384.153747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4384.155542] ? apic_timer_interrupt+0xf/0x20 [ 4384.157099] [ 4384.158041] ? on_each_cpu+0x51/0x120 [ 4384.159405] ? smp_call_function_many_cond+0x3db/0x540 [ 4384.161358] ? smp_call_function_many_cond+0x3a8/0x540 [ 4384.164133] ? load_new_mm_cr3+0x120/0x120 [ 4384.165556] ? load_new_mm_cr3+0x120/0x120 [ 4384.166810] on_each_cpu+0x51/0x120 [ 4384.167992] flush_tlb_kernel_range+0x93/0xb0 [ 4384.170143] free_unmap_vmap_area+0x63/0x70 [ 4384.171626] remove_vm_area+0xe7/0x110 [ 4384.173288] __vunmap+0x9c/0x400 [ 4384.174506] vfree+0x4d/0x100 [ 4384.175694] free_work+0x34/0x50 [libcfs] [ 4384.178729] process_one_work+0x2c8/0x700 [ 4384.180967] worker_thread+0x296/0x6e0 [ 4384.184247] ? rescuer_thread+0x570/0x570 [ 4384.186016] kthread+0x1d1/0x200 [ 4384.187903] ? set_kthread_struct+0x70/0x70 [ 4384.190106] ret_from_fork+0x1f/0x30 [ 4397.790341] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4397.800723] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 4397.807816] (detected by 2, t=2040060 jiffies, g=626849, q=1398) [ 4397.812747] Sending NMI from CPU 2 to CPUs 3: [ 4397.823599] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 4397.823611] rcu: rcu_sched kthread starved for 2039702 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4397.836784] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4397.844060] rcu: RCU grace-period kthread stack dump: [ 4397.847975] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4397.853161] Call Trace: [ 4397.854131] __schedule+0x351/0xcb0 [ 4397.855522] schedule+0xc0/0x180 [ 4397.857452] schedule_timeout+0xb4/0x190 [ 4397.862976] ? do_raw_spin_unlock+0x75/0x190 [ 4397.867003] ? __next_timer_interrupt+0x160/0x160 [ 4397.873071] ? prepare_to_swait_event+0xf3/0x140 [ 4397.876230] rcu_gp_kthread+0x8dd/0x15b0 [ 4397.879308] ? force_qs_rnp+0x2c0/0x2c0 [ 4397.882098] kthread+0x1d1/0x200 [ 4397.885022] ? set_kthread_struct+0x70/0x70 [ 4397.892802] ret_from_fork+0x1f/0x30 [ 4397.901035] rcu: Stack dump where RCU GP kthread last ran: [ 4397.908879] NMI backtrace for cpu 2 [ 4397.910491] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4397.920008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4397.926662] Workqueue: events free_work [libcfs] [ 4397.931320] Call Trace: [ 4397.932208] [ 4397.934666] ? dump_stack+0xbb/0x10e [ 4397.936201] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4397.938748] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4397.940472] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4397.942900] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4397.944912] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4397.947249] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4397.949496] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 4397.951793] ? update_process_times+0x6d/0xc0 [ 4397.953141] ? tick_sched_handle.isra.18+0x27/0x90 [ 4397.955308] ? tick_sched_timer+0x53/0xb0 [ 4397.956629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4397.958044] ? tick_sched_handle.isra.18+0x90/0x90 [ 4397.961305] ? hrtimer_interrupt+0x128/0x2c0 [ 4397.964012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4397.965972] ? apic_timer_interrupt+0xf/0x20 [ 4397.970782] [ 4397.971954] ? on_each_cpu+0x51/0x120 [ 4397.974894] ? smp_call_function_many_cond+0x3db/0x540 [ 4397.979151] ? smp_call_function_many_cond+0x3a8/0x540 [ 4397.982043] ? load_new_mm_cr3+0x120/0x120 [ 4397.985951] ? load_new_mm_cr3+0x120/0x120 [ 4397.990163] ? on_each_cpu+0x51/0x120 [ 4397.991556] ? flush_tlb_kernel_range+0x93/0xb0 [ 4397.995757] ? free_unmap_vmap_area+0x63/0x70 [ 4397.999601] ? remove_vm_area+0xe7/0x110 [ 4398.005013] ? __vunmap+0x9c/0x400 [ 4398.007757] ? vfree+0x4d/0x100 [ 4398.009510] ? free_work+0x34/0x50 [libcfs] [ 4398.014599] ? process_one_work+0x2c8/0x700 [ 4398.016275] ? worker_thread+0x296/0x6e0 [ 4398.019555] ? rescuer_thread+0x570/0x570 [ 4398.023342] ? kthread+0x1d1/0x200 [ 4398.026556] ? set_kthread_struct+0x70/0x70 [ 4398.028318] ? ret_from_fork+0x1f/0x30 [ 4400.055133] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4400.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4400.058042] Modules linked in: [ 4400.060633] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4400.060720] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4400.060728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4400.060731] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4400.060744] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4400.060748] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4400.060752] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4400.060755] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4400.060756] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4400.060759] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4400.060761] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4400.060765] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4400.060768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4400.060770] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4400.060778] Call Trace: [ 4400.060783] [ 4400.060787] ? show_regs.cold.9+0x22/0x2f [ 4400.060796] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4400.061012] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4400.061017] ? watchdog+0x40/0x40 [ 4400.061022] ? hrtimer_interrupt+0x128/0x2c0 [ 4400.061026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4400.061033] ? apic_timer_interrupt+0xf/0x20 [ 4400.061038] [ 4400.061040] ? on_each_cpu_cond_mask+0x46/0x150 [ 4400.061044] ? __x64_sys_fspick+0x210/0x210 [ 4400.061051] ? smp_call_function_many_cond+0x3db/0x540 [ 4400.061055] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4400.061059] ? __x64_sys_fspick+0x210/0x210 [ 4400.061063] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4400.061067] on_each_cpu_cond_mask+0x46/0x150 [ 4400.061071] on_each_cpu_cond+0x1c/0x30 [ 4400.061074] invalidate_bh_lrus+0x27/0x30 [ 4400.061078] invalidate_bdev+0x5a/0x90 [ 4400.061084] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4400.061290] generic_shutdown_super+0xb1/0x1b0 [ 4400.061298] kill_block_super+0x29/0x80 [ 4400.061303] deactivate_locked_super+0x52/0xd0 [ 4400.061308] deactivate_super+0x83/0x90 [ 4400.061312] cleanup_mnt+0x5f/0xe0 [ 4400.061319] __cleanup_mnt+0x16/0x20 [ 4400.065404] zfs(O) [ 4400.089358] task_work_run+0xc6/0x110 [ 4400.100004] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 [ 4400.103733] exit_to_usermode_loop+0x1dd/0x1f0 [ 4400.103750] do_syscall_64+0x3d6/0x440 [ 4400.103755] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4400.103763] RIP: 0033:0x7fbb019648fb [ 4400.103770] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4400.103774] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4400.103780] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4400.103782] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4400.103784] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4400.103786] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4400.103788] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4400.303428] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4400.314955] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4400.322325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4400.326957] Workqueue: events free_work [libcfs] [ 4400.330024] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4400.332185] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4400.347798] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4400.351816] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4400.356070] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4400.359385] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4400.362929] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4400.367126] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4400.371125] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4400.374868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4400.378305] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4400.381983] Call Trace: [ 4400.383115] [ 4400.384058] ? show_regs.cold.9+0x22/0x2f [ 4400.385707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4400.387656] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4400.390071] ? watchdog+0x40/0x40 [ 4400.391654] ? hrtimer_interrupt+0x128/0x2c0 [ 4400.393651] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4400.396579] ? apic_timer_interrupt+0xf/0x20 [ 4400.398745] [ 4400.399761] ? on_each_cpu+0x51/0x120 [ 4400.401816] ? smp_call_function_many_cond+0x3db/0x540 [ 4400.404093] ? smp_call_function_many_cond+0x3a8/0x540 [ 4400.407078] ? load_new_mm_cr3+0x120/0x120 [ 4400.408846] ? load_new_mm_cr3+0x120/0x120 [ 4400.411545] on_each_cpu+0x51/0x120 [ 4400.414686] flush_tlb_kernel_range+0x93/0xb0 [ 4400.416630] free_unmap_vmap_area+0x63/0x70 [ 4400.419082] remove_vm_area+0xe7/0x110 [ 4400.426350] __vunmap+0x9c/0x400 [ 4400.428207] vfree+0x4d/0x100 [ 4400.429276] free_work+0x34/0x50 [libcfs] [ 4400.432614] process_one_work+0x2c8/0x700 [ 4400.434920] worker_thread+0x296/0x6e0 [ 4400.436551] ? rescuer_thread+0x570/0x570 [ 4400.441471] kthread+0x1d1/0x200 [ 4400.443723] ? set_kthread_struct+0x70/0x70 [ 4400.448681] ret_from_fork+0x1f/0x30 [ 4404.703028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2046s! [ 4404.707379] Showing busy workqueues and worker pools: [ 4404.709652] workqueue events: flags=0x0 [ 4404.711517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4404.714708] in-flight: 56293:free_work [libcfs] [ 4404.717202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4404.720619] in-flight: 22277:free_work [libcfs] [ 4404.723202] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4404.727645] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4404.730718] pending: nfs4_renew_state [nfsv4] [ 4404.732894] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4404.735940] in-flight: 47596:free_work [libcfs] [ 4404.738042] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4404.741996] workqueue events_unbound: flags=0x2 [ 4404.743928] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4404.746917] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4404.751637] workqueue events_power_efficient: flags=0x80 [ 4404.754020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4404.757909] pending: neigh_periodic_work, do_cache_clean [ 4404.760704] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4404.763848] pending: check_lifetime [ 4404.765817] workqueue mm_percpu_wq: flags=0x8 [ 4404.767794] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4404.770993] pending: vmstat_update [ 4404.772956] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4404.776336] pending: vmstat_update [ 4404.777935] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4404.781679] pending: vmstat_update [ 4404.783763] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4404.787575] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4404.791538] pending: cgroup_pidlist_destroy_work_fn [ 4404.794793] workqueue writeback: flags=0x4a [ 4404.797235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4404.800019] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4404.802061] workqueue xprtiod: flags=0xa [ 4404.804182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4404.808067] pending: xprt_autoclose [ 4404.810951] workqueue ipv6_addrconf: flags=0x40008 [ 4404.814770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4404.818142] pending: addrconf_verify_work [ 4404.820503] workqueue knbd0-recv: flags=0x1a [ 4404.822817] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4404.825958] in-flight: 655:recv_work [ 4404.827841] workqueue ptlrpc_pinger: flags=0x2 [ 4404.829973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4404.832818] pending: ptlrpc_pinger_main [ptlrpc] [ 4404.835594] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2047s workers=5 idle: 53684 27131 54850 46584 [ 4404.841911] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2047s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4404.851843] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2047s workers=6 idle: 53660 37674 46417 53664 46418 [ 4404.858122] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4394s workers=2 idle: 657 [ 4424.059640] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4424.062085] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4424.097341] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4424.113001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4424.123573] Workqueue: events free_work [libcfs] [ 4424.131344] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4424.138423] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4424.154968] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4424.166443] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4424.173738] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4424.179963] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4424.187361] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4424.191797] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4424.194964] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4424.199617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4424.203188] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4424.207220] Call Trace: [ 4424.208056] [ 4424.208772] ? show_regs.cold.9+0x22/0x2f [ 4424.212763] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4424.219594] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4424.225383] ? watchdog+0x40/0x40 [ 4424.226539] ? hrtimer_interrupt+0x128/0x2c0 [ 4424.229888] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4424.232320] ? apic_timer_interrupt+0xf/0x20 [ 4424.236051] [ 4424.237521] ? on_each_cpu+0x51/0x120 [ 4424.239000] ? smp_call_function_many_cond+0x3db/0x540 [ 4424.241165] ? smp_call_function_many_cond+0x3a8/0x540 [ 4424.244446] ? load_new_mm_cr3+0x120/0x120 [ 4424.249896] ? load_new_mm_cr3+0x120/0x120 [ 4424.255088] on_each_cpu+0x51/0x120 [ 4424.257372] flush_tlb_kernel_range+0x93/0xb0 [ 4424.259997] free_unmap_vmap_area+0x63/0x70 [ 4424.261878] remove_vm_area+0xe7/0x110 [ 4424.263911] __vunmap+0x9c/0x400 [ 4424.265565] vfree+0x4d/0x100 [ 4424.267440] free_work+0x34/0x50 [libcfs] [ 4424.268839] process_one_work+0x2c8/0x700 [ 4424.274238] worker_thread+0x296/0x6e0 [ 4424.275486] ? rescuer_thread+0x570/0x570 [ 4424.279866] kthread+0x1d1/0x200 [ 4424.284232] ? set_kthread_struct+0x70/0x70 [ 4424.289273] ret_from_fork+0x1f/0x30 [ 4428.058223] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4428.072708] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4428.100119] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4428.104443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4428.108105] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4428.110487] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4428.117880] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4428.120998] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4428.123928] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4428.126902] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4428.131565] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4428.135685] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4428.141997] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4428.147474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4428.152214] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4428.156779] Call Trace: [ 4428.159249] [ 4428.164989] ? show_regs.cold.9+0x22/0x2f [ 4428.166778] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4428.168830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4428.171286] ? watchdog+0x40/0x40 [ 4428.174337] ? hrtimer_interrupt+0x128/0x2c0 [ 4428.178954] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4428.183302] ? apic_timer_interrupt+0xf/0x20 [ 4428.186006] [ 4428.186634] ? on_each_cpu_cond_mask+0x46/0x150 [ 4428.188448] ? __x64_sys_fspick+0x210/0x210 [ 4428.190709] ? smp_call_function_many_cond+0x3db/0x540 [ 4428.193501] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4428.203037] ? __x64_sys_fspick+0x210/0x210 [ 4428.205078] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4428.208512] on_each_cpu_cond_mask+0x46/0x150 [ 4428.213571] on_each_cpu_cond+0x1c/0x30 [ 4428.214831] invalidate_bh_lrus+0x27/0x30 [ 4428.216156] invalidate_bdev+0x5a/0x90 [ 4428.220812] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4428.228116] generic_shutdown_super+0xb1/0x1b0 [ 4428.230950] kill_block_super+0x29/0x80 [ 4428.233905] deactivate_locked_super+0x52/0xd0 [ 4428.236280] deactivate_super+0x83/0x90 [ 4428.239502] cleanup_mnt+0x5f/0xe0 [ 4428.241348] __cleanup_mnt+0x16/0x20 [ 4428.243008] task_work_run+0xc6/0x110 [ 4428.244917] exit_to_usermode_loop+0x1dd/0x1f0 [ 4428.247203] do_syscall_64+0x3d6/0x440 [ 4428.253954] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4428.255835] RIP: 0033:0x7fbb019648fb [ 4428.259779] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4428.269058] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4428.276077] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4428.281982] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4428.288536] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4428.291103] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4428.295837] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4432.055009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4432.058421] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4432.085426] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4432.093785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4432.102940] Workqueue: events free_work [libcfs] [ 4432.106156] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4432.109109] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4432.123021] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4432.127054] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4432.130327] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4432.133366] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4432.136325] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4432.139547] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4432.143690] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4432.149226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4432.153768] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4432.158551] Call Trace: [ 4432.160437] [ 4432.161657] ? show_regs.cold.9+0x22/0x2f [ 4432.163206] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4432.166197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4432.168428] ? watchdog+0x40/0x40 [ 4432.170151] ? hrtimer_interrupt+0x128/0x2c0 [ 4432.174161] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4432.177428] ? apic_timer_interrupt+0xf/0x20 [ 4432.180470] [ 4432.181606] ? on_each_cpu+0x51/0x120 [ 4432.184552] ? smp_call_function_many_cond+0x3db/0x540 [ 4432.188469] ? smp_call_function_many_cond+0x3a8/0x540 [ 4432.192384] ? load_new_mm_cr3+0x120/0x120 [ 4432.194998] ? load_new_mm_cr3+0x120/0x120 [ 4432.197265] on_each_cpu+0x51/0x120 [ 4432.199148] flush_tlb_kernel_range+0x93/0xb0 [ 4432.201488] free_unmap_vmap_area+0x63/0x70 [ 4432.203505] remove_vm_area+0xe7/0x110 [ 4432.206694] __vunmap+0x9c/0x400 [ 4432.209072] vfree+0x4d/0x100 [ 4432.211031] free_work+0x34/0x50 [libcfs] [ 4432.212583] process_one_work+0x2c8/0x700 [ 4432.214311] worker_thread+0x296/0x6e0 [ 4432.216229] ? rescuer_thread+0x570/0x570 [ 4432.218285] kthread+0x1d1/0x200 [ 4432.219905] ? set_kthread_struct+0x70/0x70 [ 4432.222039] ret_from_fork+0x1f/0x30 [ 4434.911030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2076s! [ 4434.914379] Showing busy workqueues and worker pools: [ 4434.916618] workqueue events: flags=0x0 [ 4434.918207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4434.921010] in-flight: 56293:free_work [libcfs] [ 4434.923157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4434.926066] in-flight: 22277:free_work [libcfs] [ 4434.928136] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4434.932611] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4434.935341] pending: nfs4_renew_state [nfsv4] [ 4434.937576] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4434.940576] in-flight: 47596:free_work [libcfs] [ 4434.942291] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4434.948569] workqueue events_unbound: flags=0x2 [ 4434.951508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4434.956211] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4434.960388] workqueue events_power_efficient: flags=0x80 [ 4434.962819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4434.965865] pending: neigh_periodic_work, do_cache_clean [ 4434.968614] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4434.970887] pending: check_lifetime [ 4434.972068] workqueue mm_percpu_wq: flags=0x8 [ 4434.973870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4434.977902] pending: vmstat_update [ 4434.979988] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4434.983339] pending: vmstat_update [ 4434.986978] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4434.992362] pending: vmstat_update [ 4434.994366] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4434.998301] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4435.003951] pending: cgroup_pidlist_destroy_work_fn [ 4435.009302] workqueue writeback: flags=0x4a [ 4435.013801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4435.016386] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4435.021735] workqueue xprtiod: flags=0xa [ 4435.027690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4435.031568] pending: xprt_autoclose [ 4435.033190] workqueue ipv6_addrconf: flags=0x40008 [ 4435.035254] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4435.039419] pending: addrconf_verify_work [ 4435.041063] workqueue knbd0-recv: flags=0x1a [ 4435.043226] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4435.045889] in-flight: 655:recv_work [ 4435.047646] workqueue ptlrpc_pinger: flags=0x2 [ 4435.049564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4435.051948] pending: ptlrpc_pinger_main [ptlrpc] [ 4435.054729] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2077s workers=5 idle: 53684 27131 54850 46584 [ 4435.057966] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2077s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4435.067170] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2077s workers=6 idle: 53660 37674 46417 53664 46418 [ 4435.073378] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4424s workers=2 idle: 657 [ 4452.059020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4452.063671] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4452.105805] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4452.111390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4452.122357] Workqueue: events free_work [libcfs] [ 4452.126733] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4452.130735] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4452.143654] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4452.148426] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4452.152635] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4452.156463] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4452.169271] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4452.178473] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4452.186236] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4452.191645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4452.194869] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4452.203335] Call Trace: [ 4452.205999] [ 4452.212844] ? show_regs.cold.9+0x22/0x2f [ 4452.214749] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4452.220103] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4452.223207] ? watchdog+0x40/0x40 [ 4452.225856] ? hrtimer_interrupt+0x128/0x2c0 [ 4452.227965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4452.235871] ? apic_timer_interrupt+0xf/0x20 [ 4452.239998] [ 4452.240823] ? on_each_cpu+0x51/0x120 [ 4452.242271] ? smp_call_function_many_cond+0x3db/0x540 [ 4452.247788] ? smp_call_function_many_cond+0x3a8/0x540 [ 4452.250211] ? load_new_mm_cr3+0x120/0x120 [ 4452.252332] ? load_new_mm_cr3+0x120/0x120 [ 4452.253666] on_each_cpu+0x51/0x120 [ 4452.254732] flush_tlb_kernel_range+0x93/0xb0 [ 4452.256507] free_unmap_vmap_area+0x63/0x70 [ 4452.258439] remove_vm_area+0xe7/0x110 [ 4452.260264] __vunmap+0x9c/0x400 [ 4452.261728] vfree+0x4d/0x100 [ 4452.263211] free_work+0x34/0x50 [libcfs] [ 4452.265205] process_one_work+0x2c8/0x700 [ 4452.267346] worker_thread+0x296/0x6e0 [ 4452.268849] ? rescuer_thread+0x570/0x570 [ 4452.273457] kthread+0x1d1/0x200 [ 4452.275684] ? set_kthread_struct+0x70/0x70 [ 4452.281546] ret_from_fork+0x1f/0x30 [ 4456.058068] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4456.064547] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4456.096276] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4456.104581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4456.112877] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4456.118323] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4456.130865] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4456.136423] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4456.143048] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4456.148437] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4456.154827] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4456.159453] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4456.165354] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4456.173470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4456.178367] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4456.183254] Call Trace: [ 4456.184878] [ 4456.186295] ? show_regs.cold.9+0x22/0x2f [ 4456.189888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4456.193460] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4456.195424] ? watchdog+0x40/0x40 [ 4456.197546] ? hrtimer_interrupt+0x128/0x2c0 [ 4456.199744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4456.202061] ? apic_timer_interrupt+0xf/0x20 [ 4456.205047] [ 4456.206373] ? on_each_cpu_cond_mask+0x46/0x150 [ 4456.211062] ? __x64_sys_fspick+0x210/0x210 [ 4456.213024] ? smp_call_function_many_cond+0x3db/0x540 [ 4456.215609] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4456.217595] ? __x64_sys_fspick+0x210/0x210 [ 4456.219491] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4456.221373] on_each_cpu_cond_mask+0x46/0x150 [ 4456.223510] on_each_cpu_cond+0x1c/0x30 [ 4456.225207] invalidate_bh_lrus+0x27/0x30 [ 4456.226840] invalidate_bdev+0x5a/0x90 [ 4456.228569] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4456.230880] generic_shutdown_super+0xb1/0x1b0 [ 4456.232769] kill_block_super+0x29/0x80 [ 4456.234289] deactivate_locked_super+0x52/0xd0 [ 4456.236328] deactivate_super+0x83/0x90 [ 4456.238052] cleanup_mnt+0x5f/0xe0 [ 4456.239928] __cleanup_mnt+0x16/0x20 [ 4456.241670] task_work_run+0xc6/0x110 [ 4456.243843] exit_to_usermode_loop+0x1dd/0x1f0 [ 4456.246692] do_syscall_64+0x3d6/0x440 [ 4456.248868] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4456.252525] RIP: 0033:0x7fbb019648fb [ 4456.254678] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4456.266797] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4456.271224] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4456.275668] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4456.280518] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4456.286829] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4456.293909] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4460.055037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4460.059257] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4460.115635] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4460.128272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4460.138350] Workqueue: events free_work [libcfs] [ 4460.142503] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4460.158463] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4460.184539] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4460.188790] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4460.201177] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4460.208195] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4460.219308] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4460.227195] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4460.231539] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4460.238399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4460.243057] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4460.246124] Call Trace: [ 4460.247281] [ 4460.248087] ? show_regs.cold.9+0x22/0x2f [ 4460.251171] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4460.253783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4460.258179] ? watchdog+0x40/0x40 [ 4460.259703] ? hrtimer_interrupt+0x128/0x2c0 [ 4460.261269] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4460.262955] ? apic_timer_interrupt+0xf/0x20 [ 4460.267472] [ 4460.269191] ? on_each_cpu+0x51/0x120 [ 4460.272783] ? smp_call_function_many_cond+0x3db/0x540 [ 4460.277035] ? smp_call_function_many_cond+0x3a8/0x540 [ 4460.281625] ? load_new_mm_cr3+0x120/0x120 [ 4460.284665] ? load_new_mm_cr3+0x120/0x120 [ 4460.286184] on_each_cpu+0x51/0x120 [ 4460.294002] flush_tlb_kernel_range+0x93/0xb0 [ 4460.296770] free_unmap_vmap_area+0x63/0x70 [ 4460.299558] remove_vm_area+0xe7/0x110 [ 4460.302343] __vunmap+0x9c/0x400 [ 4460.304775] vfree+0x4d/0x100 [ 4460.305879] free_work+0x34/0x50 [libcfs] [ 4460.309332] process_one_work+0x2c8/0x700 [ 4460.312020] worker_thread+0x296/0x6e0 [ 4460.314677] ? rescuer_thread+0x570/0x570 [ 4460.316187] kthread+0x1d1/0x200 [ 4460.318664] ? set_kthread_struct+0x70/0x70 [ 4460.322043] ret_from_fork+0x1f/0x30 [ 4465.119278] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2106s! [ 4465.124101] Showing busy workqueues and worker pools: [ 4465.127909] workqueue events: flags=0x0 [ 4465.130111] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4465.133910] in-flight: 56293:free_work [libcfs] [ 4465.138442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4465.149048] in-flight: 22277:free_work [libcfs] [ 4465.153020] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4465.157048] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4465.162471] pending: nfs4_renew_state [nfsv4] [ 4465.166584] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4465.170917] in-flight: 47596:free_work [libcfs] [ 4465.174685] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4465.179478] workqueue events_unbound: flags=0x2 [ 4465.184265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4465.189157] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4465.194211] workqueue events_power_efficient: flags=0x80 [ 4465.198360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4465.204086] pending: neigh_periodic_work, do_cache_clean [ 4465.208422] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4465.213070] pending: check_lifetime [ 4465.215305] workqueue mm_percpu_wq: flags=0x8 [ 4465.217151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4465.221983] pending: vmstat_update [ 4465.224086] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4465.227260] pending: vmstat_update [ 4465.230024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4465.236181] pending: vmstat_update [ 4465.240380] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4465.243151] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4465.248070] pending: cgroup_pidlist_destroy_work_fn [ 4465.251278] workqueue writeback: flags=0x4a [ 4465.254302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4465.258196] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4465.262321] workqueue xprtiod: flags=0xa [ 4465.264044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4465.271132] pending: xprt_autoclose [ 4465.273300] workqueue ipv6_addrconf: flags=0x40008 [ 4465.278899] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4465.283563] pending: addrconf_verify_work [ 4465.288779] workqueue knbd0-recv: flags=0x1a [ 4465.290383] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4465.299720] in-flight: 655:recv_work [ 4465.302111] workqueue ptlrpc_pinger: flags=0x2 [ 4465.305890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4465.309594] pending: ptlrpc_pinger_main [ptlrpc] [ 4465.312989] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2107s workers=5 idle: 53684 27131 54850 46584 [ 4465.319586] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2107s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4465.330752] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2107s workers=6 idle: 53660 37674 46417 53664 46418 [ 4465.339778] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4455s workers=2 idle: 657 [ 4480.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4480.061958] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4480.077177] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4480.081533] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4480.083966] Workqueue: events free_work [libcfs] [ 4480.085776] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4480.087409] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4480.092845] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4480.095015] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4480.097063] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4480.099078] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4480.100819] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4480.102891] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4480.104960] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4480.107407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4480.109200] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4480.111292] Call Trace: [ 4480.112096] [ 4480.112713] ? show_regs.cold.9+0x22/0x2f [ 4480.113949] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4480.115294] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4480.116693] ? watchdog+0x40/0x40 [ 4480.117683] ? hrtimer_interrupt+0x128/0x2c0 [ 4480.118956] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4480.120735] ? apic_timer_interrupt+0xf/0x20 [ 4480.121970] [ 4480.122572] ? on_each_cpu+0x51/0x120 [ 4480.123754] ? smp_call_function_many_cond+0x3db/0x540 [ 4480.125218] ? smp_call_function_many_cond+0x3a8/0x540 [ 4480.126783] ? load_new_mm_cr3+0x120/0x120 [ 4480.128040] ? load_new_mm_cr3+0x120/0x120 [ 4480.129220] on_each_cpu+0x51/0x120 [ 4480.130281] flush_tlb_kernel_range+0x93/0xb0 [ 4480.131629] free_unmap_vmap_area+0x63/0x70 [ 4480.132835] remove_vm_area+0xe7/0x110 [ 4480.133954] __vunmap+0x9c/0x400 [ 4480.134879] vfree+0x4d/0x100 [ 4480.135797] free_work+0x34/0x50 [libcfs] [ 4480.136969] process_one_work+0x2c8/0x700 [ 4480.138150] worker_thread+0x296/0x6e0 [ 4480.139264] ? rescuer_thread+0x570/0x570 [ 4480.140401] kthread+0x1d1/0x200 [ 4480.141511] ? set_kthread_struct+0x70/0x70 [ 4480.142779] ret_from_fork+0x1f/0x30 [ 4484.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4484.060748] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4484.105630] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4484.116954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4484.137363] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4484.144783] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4484.163673] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4484.170251] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4484.174666] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4484.178835] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4484.182519] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4484.187434] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4484.191794] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4484.197595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4484.201462] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4484.207076] Call Trace: [ 4484.208911] [ 4484.210767] ? show_regs.cold.9+0x22/0x2f [ 4484.213417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4484.216572] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4484.219329] ? watchdog+0x40/0x40 [ 4484.221035] ? hrtimer_interrupt+0x128/0x2c0 [ 4484.223633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4484.226243] ? apic_timer_interrupt+0xf/0x20 [ 4484.230473] [ 4484.232050] ? on_each_cpu_cond_mask+0x46/0x150 [ 4484.235225] ? __x64_sys_fspick+0x210/0x210 [ 4484.238039] ? smp_call_function_many_cond+0x3d9/0x540 [ 4484.247295] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4484.249711] ? __x64_sys_fspick+0x210/0x210 [ 4484.252108] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4484.254387] on_each_cpu_cond_mask+0x46/0x150 [ 4484.256522] on_each_cpu_cond+0x1c/0x30 [ 4484.258231] invalidate_bh_lrus+0x27/0x30 [ 4484.260134] invalidate_bdev+0x5a/0x90 [ 4484.261921] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4484.265485] generic_shutdown_super+0xb1/0x1b0 [ 4484.268898] kill_block_super+0x29/0x80 [ 4484.272194] deactivate_locked_super+0x52/0xd0 [ 4484.277776] deactivate_super+0x83/0x90 [ 4484.279684] cleanup_mnt+0x5f/0xe0 [ 4484.281236] __cleanup_mnt+0x16/0x20 [ 4484.283233] task_work_run+0xc6/0x110 [ 4484.284773] exit_to_usermode_loop+0x1dd/0x1f0 [ 4484.287271] do_syscall_64+0x3d6/0x440 [ 4484.291080] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4484.294899] RIP: 0033:0x7fbb019648fb [ 4484.298150] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4484.311951] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4484.317633] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4484.322898] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4484.332523] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4484.337273] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4484.341332] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4492.055161] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4492.064160] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4492.119100] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4492.134379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4492.144863] Workqueue: events free_work [libcfs] [ 4492.152060] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4492.156350] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4492.168515] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4492.175564] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4492.179699] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4492.184159] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4492.189693] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4492.194356] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4492.198507] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4492.203724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4492.206493] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4492.211147] Call Trace: [ 4492.212472] [ 4492.213570] ? show_regs.cold.9+0x22/0x2f [ 4492.215271] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4492.217152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4492.220107] ? watchdog+0x40/0x40 [ 4492.221854] ? hrtimer_interrupt+0x128/0x2c0 [ 4492.224697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4492.228466] ? apic_timer_interrupt+0xf/0x20 [ 4492.232298] [ 4492.233702] ? on_each_cpu+0x51/0x120 [ 4492.237167] ? smp_call_function_many_cond+0x3d9/0x540 [ 4492.240066] ? smp_call_function_many_cond+0x3a8/0x540 [ 4492.254087] ? load_new_mm_cr3+0x120/0x120 [ 4492.255697] ? load_new_mm_cr3+0x120/0x120 [ 4492.264625] on_each_cpu+0x51/0x120 [ 4492.268254] flush_tlb_kernel_range+0x93/0xb0 [ 4492.276455] free_unmap_vmap_area+0x63/0x70 [ 4492.279335] remove_vm_area+0xe7/0x110 [ 4492.283500] __vunmap+0x9c/0x400 [ 4492.288488] vfree+0x4d/0x100 [ 4492.290669] free_work+0x34/0x50 [libcfs] [ 4492.294210] process_one_work+0x2c8/0x700 [ 4492.297473] worker_thread+0x296/0x6e0 [ 4492.300080] ? rescuer_thread+0x570/0x570 [ 4492.302715] kthread+0x1d1/0x200 [ 4492.305442] ? set_kthread_struct+0x70/0x70 [ 4492.307111] ret_from_fork+0x1f/0x30 [ 4495.839445] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2137s! [ 4495.848186] Showing busy workqueues and worker pools: [ 4495.850784] workqueue events: flags=0x0 [ 4495.854058] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4495.869599] in-flight: 56293:free_work [libcfs] [ 4495.875578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4495.881164] in-flight: 22277:free_work [libcfs] [ 4495.885098] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4495.899855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4495.914776] pending: nfs4_renew_state [nfsv4] [ 4495.927218] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4495.929847] in-flight: 47596:free_work [libcfs] [ 4495.941302] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4495.952264] workqueue events_unbound: flags=0x2 [ 4495.954012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4495.962857] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4495.969295] workqueue events_power_efficient: flags=0x80 [ 4495.974884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4495.979577] pending: neigh_periodic_work, do_cache_clean [ 4495.982790] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4495.993199] pending: check_lifetime [ 4495.998592] workqueue mm_percpu_wq: flags=0x8 [ 4496.007960] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4496.016745] pending: vmstat_update [ 4496.020618] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4496.025337] pending: vmstat_update [ 4496.029772] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4496.043421] pending: vmstat_update [ 4496.051504] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4496.060429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4496.064931] pending: cgroup_pidlist_destroy_work_fn [ 4496.070077] workqueue writeback: flags=0x4a [ 4496.075078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4496.078920] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4496.089340] workqueue xprtiod: flags=0xa [ 4496.092015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4496.102030] pending: xprt_autoclose [ 4496.107595] workqueue ipv6_addrconf: flags=0x40008 [ 4496.110242] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4496.113321] pending: addrconf_verify_work [ 4496.115585] workqueue knbd0-recv: flags=0x1a [ 4496.117530] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4496.120646] in-flight: 655:recv_work [ 4496.123574] workqueue ptlrpc_pinger: flags=0x2 [ 4496.125752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4496.128967] pending: ptlrpc_pinger_main [ptlrpc] [ 4496.131250] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2138s workers=5 idle: 53684 27131 54850 46584 [ 4496.135985] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2138s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4496.152279] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2138s workers=6 idle: 53660 37674 46417 53664 46418 [ 4496.163720] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4485s workers=2 idle: 657 [ 4508.061332] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4508.072776] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4508.150936] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4508.165592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4508.174298] Workqueue: events free_work [libcfs] [ 4508.177458] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4508.184195] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4508.205702] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4508.209726] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4508.222704] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4508.229002] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4508.235214] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4508.239721] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4508.243234] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4508.246183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4508.255346] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4508.274800] Call Trace: [ 4508.281363] [ 4508.290881] ? show_regs.cold.9+0x22/0x2f [ 4508.292638] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4508.294000] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4508.310974] ? watchdog+0x40/0x40 [ 4508.312370] ? hrtimer_interrupt+0x128/0x2c0 [ 4508.327279] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4508.331971] ? apic_timer_interrupt+0xf/0x20 [ 4508.336095] [ 4508.336978] ? on_each_cpu+0x51/0x120 [ 4508.341316] ? smp_call_function_many_cond+0x3db/0x540 [ 4508.344689] ? smp_call_function_many_cond+0x3a8/0x540 [ 4508.349312] ? load_new_mm_cr3+0x120/0x120 [ 4508.351573] ? load_new_mm_cr3+0x120/0x120 [ 4508.357244] on_each_cpu+0x51/0x120 [ 4508.359570] flush_tlb_kernel_range+0x93/0xb0 [ 4508.362461] free_unmap_vmap_area+0x63/0x70 [ 4508.365496] remove_vm_area+0xe7/0x110 [ 4508.372006] __vunmap+0x9c/0x400 [ 4508.380333] vfree+0x4d/0x100 [ 4508.382662] free_work+0x34/0x50 [libcfs] [ 4508.384235] process_one_work+0x2c8/0x700 [ 4508.389982] worker_thread+0x296/0x6e0 [ 4508.395147] ? rescuer_thread+0x570/0x570 [ 4508.400738] kthread+0x1d1/0x200 [ 4508.403271] ? set_kthread_struct+0x70/0x70 [ 4508.409615] ret_from_fork+0x1f/0x30 [ 4512.058014] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4512.066986] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4512.104624] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4512.118399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4512.123601] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4512.126351] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4512.136571] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4512.143224] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4512.156148] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4512.167533] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4512.175432] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4512.179312] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4512.184352] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4512.188626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4512.194326] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4512.200380] Call Trace: [ 4512.203108] [ 4512.205510] ? show_regs.cold.9+0x22/0x2f [ 4512.209508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4512.214039] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4512.221344] ? watchdog+0x40/0x40 [ 4512.222662] ? hrtimer_interrupt+0x128/0x2c0 [ 4512.229549] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4512.231344] ? apic_timer_interrupt+0xf/0x20 [ 4512.239444] [ 4512.240367] ? on_each_cpu_cond_mask+0x46/0x150 [ 4512.244193] ? __x64_sys_fspick+0x210/0x210 [ 4512.246201] ? smp_call_function_many_cond+0x3d9/0x540 [ 4512.249752] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4512.254593] ? __x64_sys_fspick+0x210/0x210 [ 4512.258637] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4512.261320] on_each_cpu_cond_mask+0x46/0x150 [ 4512.266372] on_each_cpu_cond+0x1c/0x30 [ 4512.267527] invalidate_bh_lrus+0x27/0x30 [ 4512.277130] invalidate_bdev+0x5a/0x90 [ 4512.280202] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4512.285275] generic_shutdown_super+0xb1/0x1b0 [ 4512.287840] kill_block_super+0x29/0x80 [ 4512.291219] deactivate_locked_super+0x52/0xd0 [ 4512.295267] deactivate_super+0x83/0x90 [ 4512.298121] cleanup_mnt+0x5f/0xe0 [ 4512.300143] __cleanup_mnt+0x16/0x20 [ 4512.302581] task_work_run+0xc6/0x110 [ 4512.304603] exit_to_usermode_loop+0x1dd/0x1f0 [ 4512.307158] do_syscall_64+0x3d6/0x440 [ 4512.309244] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4512.311838] RIP: 0033:0x7fbb019648fb [ 4512.313379] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4512.322846] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4512.326725] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4512.330260] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4512.336243] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4512.341312] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4512.348012] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4524.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4524.058660] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4524.105196] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4524.114913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4524.121999] Workqueue: events free_work [libcfs] [ 4524.124208] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4524.128032] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4524.139990] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4524.144929] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4524.151880] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4524.156175] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4524.158132] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4524.161681] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4524.166075] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4524.172194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4524.175528] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4524.179939] Call Trace: [ 4524.181458] [ 4524.182647] ? show_regs.cold.9+0x22/0x2f [ 4524.184991] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4524.188026] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4524.190945] ? watchdog+0x40/0x40 [ 4524.193256] ? hrtimer_interrupt+0x128/0x2c0 [ 4524.196047] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4524.199243] ? apic_timer_interrupt+0xf/0x20 [ 4524.205467] [ 4524.207401] ? on_each_cpu+0x51/0x120 [ 4524.209900] ? smp_call_function_many_cond+0x3db/0x540 [ 4524.213774] ? smp_call_function_many_cond+0x3a8/0x540 [ 4524.220381] ? load_new_mm_cr3+0x120/0x120 [ 4524.222754] ? load_new_mm_cr3+0x120/0x120 [ 4524.225564] on_each_cpu+0x51/0x120 [ 4524.227229] flush_tlb_kernel_range+0x93/0xb0 [ 4524.228904] free_unmap_vmap_area+0x63/0x70 [ 4524.231730] remove_vm_area+0xe7/0x110 [ 4524.233737] __vunmap+0x9c/0x400 [ 4524.235573] vfree+0x4d/0x100 [ 4524.236776] free_work+0x34/0x50 [libcfs] [ 4524.238563] process_one_work+0x2c8/0x700 [ 4524.240097] worker_thread+0x296/0x6e0 [ 4524.242083] ? rescuer_thread+0x570/0x570 [ 4524.244076] kthread+0x1d1/0x200 [ 4524.245467] ? set_kthread_struct+0x70/0x70 [ 4524.247265] ret_from_fork+0x1f/0x30 [ 4526.559119] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2168s! [ 4526.570373] Showing busy workqueues and worker pools: [ 4526.573372] workqueue events: flags=0x0 [ 4526.576256] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4526.578984] in-flight: 56293:free_work [libcfs] [ 4526.581420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4526.584789] in-flight: 22277:free_work [libcfs] [ 4526.587413] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4526.595310] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4526.603568] pending: nfs4_renew_state [nfsv4] [ 4526.606963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4526.609428] in-flight: 47596:free_work [libcfs] [ 4526.616527] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4526.627718] workqueue events_unbound: flags=0x2 [ 4526.633369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4526.639930] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4526.645222] workqueue events_power_efficient: flags=0x80 [ 4526.649209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4526.654551] pending: neigh_periodic_work, do_cache_clean [ 4526.660409] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4526.665338] pending: check_lifetime [ 4526.670149] workqueue mm_percpu_wq: flags=0x8 [ 4526.671894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4526.677389] pending: vmstat_update [ 4526.683545] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4526.687630] pending: vmstat_update [ 4526.689977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4526.692615] pending: vmstat_update [ 4526.695966] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4526.701312] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4526.709920] pending: cgroup_pidlist_destroy_work_fn [ 4526.712298] workqueue writeback: flags=0x4a [ 4526.716137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4526.722555] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4526.724981] workqueue xprtiod: flags=0xa [ 4526.728969] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4526.736419] pending: xprt_autoclose [ 4526.738791] workqueue ipv6_addrconf: flags=0x40008 [ 4526.743247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4526.753387] pending: addrconf_verify_work [ 4526.755179] workqueue knbd0-recv: flags=0x1a [ 4526.762719] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4526.775213] in-flight: 655:recv_work [ 4526.776333] workqueue ptlrpc_pinger: flags=0x2 [ 4526.777617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4526.787255] pending: ptlrpc_pinger_main [ptlrpc] [ 4526.793652] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2169s workers=5 idle: 53684 27131 54850 46584 [ 4526.805557] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2169s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4526.815660] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2169s workers=6 idle: 53660 37674 46417 53664 46418 [ 4526.822955] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4516s workers=2 idle: 657 [ 4536.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4536.061585] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4536.076787] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4536.080601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4536.083554] Workqueue: events free_work [libcfs] [ 4536.085108] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4536.087032] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4536.092729] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4536.095111] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4536.097211] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4536.099394] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4536.101501] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4536.103551] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4536.105732] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4536.108110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4536.110065] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4536.112511] Call Trace: [ 4536.113186] [ 4536.113899] ? show_regs.cold.9+0x22/0x2f [ 4536.115073] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4536.116554] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4536.117946] ? watchdog+0x40/0x40 [ 4536.119036] ? hrtimer_interrupt+0x128/0x2c0 [ 4536.120363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4536.121757] ? apic_timer_interrupt+0xf/0x20 [ 4536.123002] [ 4536.123751] ? on_each_cpu+0x51/0x120 [ 4536.124841] ? smp_call_function_many_cond+0x3d9/0x540 [ 4536.126611] ? smp_call_function_many_cond+0x3a8/0x540 [ 4536.128155] ? load_new_mm_cr3+0x120/0x120 [ 4536.129496] ? load_new_mm_cr3+0x120/0x120 [ 4536.130818] on_each_cpu+0x51/0x120 [ 4536.131881] flush_tlb_kernel_range+0x93/0xb0 [ 4536.133460] free_unmap_vmap_area+0x63/0x70 [ 4536.134744] remove_vm_area+0xe7/0x110 [ 4536.135966] __vunmap+0x9c/0x400 [ 4536.136863] vfree+0x4d/0x100 [ 4536.137786] free_work+0x34/0x50 [libcfs] [ 4536.139093] process_one_work+0x2c8/0x700 [ 4536.140500] worker_thread+0x296/0x6e0 [ 4536.142261] ? rescuer_thread+0x570/0x570 [ 4536.144490] kthread+0x1d1/0x200 [ 4536.146795] ? set_kthread_struct+0x70/0x70 [ 4536.148918] ret_from_fork+0x1f/0x30 [ 4540.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4540.060902] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4540.078443] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4540.082616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4540.085432] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4540.087268] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4540.092892] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4540.095544] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4540.097918] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4540.100337] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4540.102796] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4540.105122] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4540.107495] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4540.109916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4540.112417] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4540.114636] Call Trace: [ 4540.115425] [ 4540.115966] ? show_regs.cold.9+0x22/0x2f [ 4540.117387] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4540.118770] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4540.120093] ? watchdog+0x40/0x40 [ 4540.121270] ? hrtimer_interrupt+0x128/0x2c0 [ 4540.122562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4540.123994] ? apic_timer_interrupt+0xf/0x20 [ 4540.125458] [ 4540.125988] ? on_each_cpu_cond_mask+0x46/0x150 [ 4540.127512] ? __x64_sys_fspick+0x210/0x210 [ 4540.128748] ? smp_call_function_many_cond+0x3db/0x540 [ 4540.130633] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4540.132285] ? __x64_sys_fspick+0x210/0x210 [ 4540.133558] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4540.134909] on_each_cpu_cond_mask+0x46/0x150 [ 4540.136406] on_each_cpu_cond+0x1c/0x30 [ 4540.137563] invalidate_bh_lrus+0x27/0x30 [ 4540.138775] invalidate_bdev+0x5a/0x90 [ 4540.139895] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4540.141543] generic_shutdown_super+0xb1/0x1b0 [ 4540.142943] kill_block_super+0x29/0x80 [ 4540.144243] deactivate_locked_super+0x52/0xd0 [ 4540.145700] deactivate_super+0x83/0x90 [ 4540.147106] cleanup_mnt+0x5f/0xe0 [ 4540.148227] __cleanup_mnt+0x16/0x20 [ 4540.149330] task_work_run+0xc6/0x110 [ 4540.150456] exit_to_usermode_loop+0x1dd/0x1f0 [ 4540.151783] do_syscall_64+0x3d6/0x440 [ 4540.152927] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4540.154659] RIP: 0033:0x7fbb019648fb [ 4540.155610] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4540.164127] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4540.167247] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4540.169403] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4540.171584] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4540.174002] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4540.176272] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4552.055886] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4552.064429] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4552.121137] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4552.134694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4552.143436] Workqueue: events free_work [libcfs] [ 4552.150436] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4552.159085] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4552.169967] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4552.180274] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4552.189116] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4552.194187] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4552.198328] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4552.202422] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4552.206609] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4552.212114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4552.215032] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4552.217504] Call Trace: [ 4552.221237] [ 4552.222368] ? show_regs.cold.9+0x22/0x2f [ 4552.225329] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4552.227621] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4552.232362] ? watchdog+0x40/0x40 [ 4552.236771] ? hrtimer_interrupt+0x128/0x2c0 [ 4552.242101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4552.247865] ? apic_timer_interrupt+0xf/0x20 [ 4552.253401] [ 4552.256316] ? on_each_cpu+0x51/0x120 [ 4552.260386] ? smp_call_function_many_cond+0x3db/0x540 [ 4552.266049] ? smp_call_function_many_cond+0x3a8/0x540 [ 4552.273073] ? load_new_mm_cr3+0x120/0x120 [ 4552.274665] ? load_new_mm_cr3+0x120/0x120 [ 4552.281541] on_each_cpu+0x51/0x120 [ 4552.285202] flush_tlb_kernel_range+0x93/0xb0 [ 4552.288723] free_unmap_vmap_area+0x63/0x70 [ 4552.293825] remove_vm_area+0xe7/0x110 [ 4552.296295] __vunmap+0x9c/0x400 [ 4552.298478] vfree+0x4d/0x100 [ 4552.300481] free_work+0x34/0x50 [libcfs] [ 4552.303395] process_one_work+0x2c8/0x700 [ 4552.309169] worker_thread+0x296/0x6e0 [ 4552.311642] ? rescuer_thread+0x570/0x570 [ 4552.317350] kthread+0x1d1/0x200 [ 4552.319354] ? set_kthread_struct+0x70/0x70 [ 4552.321528] ret_from_fork+0x1f/0x30 [ 4557.279035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2199s! [ 4557.283285] Showing busy workqueues and worker pools: [ 4557.285979] workqueue events: flags=0x0 [ 4557.291191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4557.295299] in-flight: 56293:free_work [libcfs] [ 4557.297677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4557.304092] in-flight: 22277:free_work [libcfs] [ 4557.306580] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4557.313851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4557.318635] pending: nfs4_renew_state [nfsv4] [ 4557.321707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4557.324677] in-flight: 47596:free_work [libcfs] [ 4557.327637] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4557.332523] workqueue events_unbound: flags=0x2 [ 4557.335534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4557.338567] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4557.347057] workqueue events_power_efficient: flags=0x80 [ 4557.350354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4557.353961] pending: neigh_periodic_work, do_cache_clean [ 4557.357224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4557.360214] pending: check_lifetime [ 4557.363742] workqueue mm_percpu_wq: flags=0x8 [ 4557.369602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4557.378957] pending: vmstat_update [ 4557.380479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4557.385276] pending: vmstat_update [ 4557.387463] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4557.391339] pending: vmstat_update [ 4557.393612] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4557.396410] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4557.400443] pending: cgroup_pidlist_destroy_work_fn [ 4557.404627] workqueue writeback: flags=0x4a [ 4557.411668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4557.419456] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4557.430308] workqueue xprtiod: flags=0xa [ 4557.434430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4557.438830] pending: xprt_autoclose [ 4557.442435] workqueue ipv6_addrconf: flags=0x40008 [ 4557.449084] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4557.455154] pending: addrconf_verify_work [ 4557.458792] workqueue knbd0-recv: flags=0x1a [ 4557.461549] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4557.467941] in-flight: 655:recv_work [ 4557.470828] workqueue ptlrpc_pinger: flags=0x2 [ 4557.475967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4557.483177] pending: ptlrpc_pinger_main [ptlrpc] [ 4557.489000] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2199s workers=5 idle: 53684 27131 54850 46584 [ 4557.494659] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2199s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4557.504102] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2199s workers=6 idle: 53660 37674 46417 53664 46418 [ 4557.510657] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4547s workers=2 idle: 657 [ 4564.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4564.062991] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4564.088076] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4564.093281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4564.097488] Workqueue: events free_work [libcfs] [ 4564.100150] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4564.102929] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4564.111870] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4564.115495] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4564.119022] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4564.122219] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4564.125509] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4564.128868] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4564.132049] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4564.136567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4564.139021] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4564.145242] Call Trace: [ 4564.148058] [ 4564.149541] ? show_regs.cold.9+0x22/0x2f [ 4564.151944] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4564.154470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4564.156928] ? watchdog+0x40/0x40 [ 4564.158832] ? hrtimer_interrupt+0x128/0x2c0 [ 4564.161325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4564.164063] ? apic_timer_interrupt+0xf/0x20 [ 4564.166304] [ 4564.167376] ? on_each_cpu+0x51/0x120 [ 4564.169340] ? smp_call_function_many_cond+0x3db/0x540 [ 4564.172994] ? smp_call_function_many_cond+0x3a8/0x540 [ 4564.175852] ? load_new_mm_cr3+0x120/0x120 [ 4564.177907] ? load_new_mm_cr3+0x120/0x120 [ 4564.179923] on_each_cpu+0x51/0x120 [ 4564.182059] flush_tlb_kernel_range+0x93/0xb0 [ 4564.184051] free_unmap_vmap_area+0x63/0x70 [ 4564.186250] remove_vm_area+0xe7/0x110 [ 4564.188508] __vunmap+0x9c/0x400 [ 4564.190439] vfree+0x4d/0x100 [ 4564.192041] free_work+0x34/0x50 [libcfs] [ 4564.194101] process_one_work+0x2c8/0x700 [ 4564.196262] worker_thread+0x296/0x6e0 [ 4564.198265] ? rescuer_thread+0x570/0x570 [ 4564.200034] kthread+0x1d1/0x200 [ 4564.201330] ? set_kthread_struct+0x70/0x70 [ 4564.203666] ret_from_fork+0x1f/0x30 [ 4568.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4568.064188] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4568.114491] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4568.123309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4568.129684] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4568.133639] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4568.152334] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4568.163984] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4568.176271] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4568.180781] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4568.184804] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4568.192370] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4568.197509] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4568.209099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4568.216998] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4568.219303] Call Trace: [ 4568.224862] [ 4568.229916] ? show_regs.cold.9+0x22/0x2f [ 4568.235874] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4568.237690] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4568.244079] ? watchdog+0x40/0x40 [ 4568.249753] ? hrtimer_interrupt+0x128/0x2c0 [ 4568.255877] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4568.263042] ? apic_timer_interrupt+0xf/0x20 [ 4568.266397] [ 4568.267956] ? on_each_cpu_cond_mask+0x46/0x150 [ 4568.271088] ? __x64_sys_fspick+0x210/0x210 [ 4568.276197] ? smp_call_function_many_cond+0x3db/0x540 [ 4568.283105] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4568.288934] ? __x64_sys_fspick+0x210/0x210 [ 4568.292986] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4568.296966] on_each_cpu_cond_mask+0x46/0x150 [ 4568.300820] on_each_cpu_cond+0x1c/0x30 [ 4568.304508] invalidate_bh_lrus+0x27/0x30 [ 4568.307226] invalidate_bdev+0x5a/0x90 [ 4568.309809] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4568.314025] generic_shutdown_super+0xb1/0x1b0 [ 4568.318188] kill_block_super+0x29/0x80 [ 4568.322175] deactivate_locked_super+0x52/0xd0 [ 4568.328442] deactivate_super+0x83/0x90 [ 4568.331518] cleanup_mnt+0x5f/0xe0 [ 4568.335048] __cleanup_mnt+0x16/0x20 [ 4568.338040] task_work_run+0xc6/0x110 [ 4568.346616] exit_to_usermode_loop+0x1dd/0x1f0 [ 4568.350457] do_syscall_64+0x3d6/0x440 [ 4568.355275] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4568.359422] RIP: 0033:0x7fbb019648fb [ 4568.365148] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4568.393196] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4568.398114] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4568.402315] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4568.406254] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4568.413074] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4568.420222] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4577.795472] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4577.800356] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 4577.808217] (detected by 2, t=2220065 jiffies, g=626849, q=1398) [ 4577.811207] Sending NMI from CPU 2 to CPUs 3: [ 4577.816574] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 4577.816586] rcu: rcu_sched kthread starved for 2219707 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4577.833089] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4577.836969] rcu: RCU grace-period kthread stack dump: [ 4577.839052] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4577.843719] Call Trace: [ 4577.844898] __schedule+0x351/0xcb0 [ 4577.846621] schedule+0xc0/0x180 [ 4577.847899] schedule_timeout+0xb4/0x190 [ 4577.852953] ? do_raw_spin_unlock+0x75/0x190 [ 4577.855205] ? __next_timer_interrupt+0x160/0x160 [ 4577.859188] ? prepare_to_swait_event+0xf3/0x140 [ 4577.862954] rcu_gp_kthread+0x8dd/0x15b0 [ 4577.865775] ? force_qs_rnp+0x2c0/0x2c0 [ 4577.867600] kthread+0x1d1/0x200 [ 4577.869199] ? set_kthread_struct+0x70/0x70 [ 4577.871169] ret_from_fork+0x1f/0x30 [ 4577.872804] rcu: Stack dump where RCU GP kthread last ran: [ 4577.874988] NMI backtrace for cpu 2 [ 4577.876831] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4577.881656] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4577.885310] Workqueue: events free_work [libcfs] [ 4577.887714] Call Trace: [ 4577.888769] [ 4577.889607] ? dump_stack+0xbb/0x10e [ 4577.891016] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4577.893277] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4577.895106] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4577.896984] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4577.899058] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4577.900862] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4577.903052] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 4577.905248] ? update_process_times+0x6d/0xc0 [ 4577.907212] ? tick_sched_handle.isra.18+0x27/0x90 [ 4577.909122] ? tick_sched_timer+0x53/0xb0 [ 4577.910718] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4577.912292] ? tick_sched_handle.isra.18+0x90/0x90 [ 4577.914484] ? hrtimer_interrupt+0x128/0x2c0 [ 4577.916590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4577.918305] ? apic_timer_interrupt+0xf/0x20 [ 4577.920135] [ 4577.921045] ? on_each_cpu+0x51/0x120 [ 4577.922408] ? smp_call_function_many_cond+0x3d9/0x540 [ 4577.924217] ? smp_call_function_many_cond+0x3a8/0x540 [ 4577.926744] ? load_new_mm_cr3+0x120/0x120 [ 4577.928616] ? load_new_mm_cr3+0x120/0x120 [ 4577.930296] ? on_each_cpu+0x51/0x120 [ 4577.932051] ? flush_tlb_kernel_range+0x93/0xb0 [ 4577.933936] ? free_unmap_vmap_area+0x63/0x70 [ 4577.935920] ? remove_vm_area+0xe7/0x110 [ 4577.937747] ? __vunmap+0x9c/0x400 [ 4577.939194] ? vfree+0x4d/0x100 [ 4577.940733] ? free_work+0x34/0x50 [libcfs] [ 4577.942587] ? process_one_work+0x2c8/0x700 [ 4577.944567] ? worker_thread+0x296/0x6e0 [ 4577.945886] ? rescuer_thread+0x570/0x570 [ 4577.947072] ? kthread+0x1d1/0x200 [ 4577.948820] ? set_kthread_struct+0x70/0x70 [ 4577.950611] ? ret_from_fork+0x1f/0x30 [ 4584.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4584.059417] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4584.100611] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4584.110933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4584.119262] Workqueue: events free_work [libcfs] [ 4584.121502] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4584.125062] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4584.134706] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4584.138472] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4584.141702] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4584.144924] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4584.147854] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4584.150744] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4584.153652] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4584.160304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4584.170255] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4584.174904] Call Trace: [ 4584.176771] [ 4584.178494] ? show_regs.cold.9+0x22/0x2f [ 4584.180845] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4584.187663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4584.195618] ? watchdog+0x40/0x40 [ 4584.198340] ? hrtimer_interrupt+0x128/0x2c0 [ 4584.205351] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4584.208779] ? apic_timer_interrupt+0xf/0x20 [ 4584.213775] [ 4584.215698] ? on_each_cpu+0x51/0x120 [ 4584.216733] ? smp_call_function_many_cond+0x3db/0x540 [ 4584.219502] ? smp_call_function_many_cond+0x3a8/0x540 [ 4584.230473] ? load_new_mm_cr3+0x120/0x120 [ 4584.232075] ? load_new_mm_cr3+0x120/0x120 [ 4584.238574] on_each_cpu+0x51/0x120 [ 4584.243478] flush_tlb_kernel_range+0x93/0xb0 [ 4584.248526] free_unmap_vmap_area+0x63/0x70 [ 4584.253308] remove_vm_area+0xe7/0x110 [ 4584.258919] __vunmap+0x9c/0x400 [ 4584.262622] vfree+0x4d/0x100 [ 4584.263765] free_work+0x34/0x50 [libcfs] [ 4584.266348] process_one_work+0x2c8/0x700 [ 4584.267822] worker_thread+0x296/0x6e0 [ 4584.278084] ? rescuer_thread+0x570/0x570 [ 4584.279667] kthread+0x1d1/0x200 [ 4584.284518] ? set_kthread_struct+0x70/0x70 [ 4584.288271] ret_from_fork+0x1f/0x30 [ 4588.000664] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2229s! [ 4588.005196] Showing busy workqueues and worker pools: [ 4588.009372] workqueue events: flags=0x0 [ 4588.010897] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4588.016303] in-flight: 56293:free_work [libcfs] [ 4588.019507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4588.027807] in-flight: 22277:free_work [libcfs] [ 4588.031641] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4588.035310] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4588.046915] pending: nfs4_renew_state [nfsv4] [ 4588.049687] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4588.055990] in-flight: 47596:free_work [libcfs] [ 4588.057961] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4588.070184] workqueue events_unbound: flags=0x2 [ 4588.071901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4588.078111] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4588.083049] workqueue events_power_efficient: flags=0x80 [ 4588.088533] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4588.090990] pending: neigh_periodic_work, do_cache_clean [ 4588.096041] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4588.101146] pending: check_lifetime [ 4588.102904] workqueue mm_percpu_wq: flags=0x8 [ 4588.105646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4588.112938] pending: vmstat_update [ 4588.117443] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4588.121672] pending: vmstat_update [ 4588.122724] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4588.129882] pending: vmstat_update [ 4588.133412] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4588.135341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4588.146339] pending: cgroup_pidlist_destroy_work_fn [ 4588.148300] workqueue writeback: flags=0x4a [ 4588.157502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4588.162023] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4588.176537] workqueue xprtiod: flags=0xa [ 4588.181866] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4588.189939] pending: xprt_autoclose [ 4588.192143] workqueue ipv6_addrconf: flags=0x40008 [ 4588.197494] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4588.201765] pending: addrconf_verify_work [ 4588.210093] workqueue knbd0-recv: flags=0x1a [ 4588.212560] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4588.220038] in-flight: 655:recv_work [ 4588.221896] workqueue ptlrpc_pinger: flags=0x2 [ 4588.225920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4588.235491] pending: ptlrpc_pinger_main [ptlrpc] [ 4588.237641] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2230s workers=5 idle: 53684 27131 54850 46584 [ 4588.249606] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2230s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4588.255452] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2230s workers=6 idle: 53660 37674 46417 53664 46418 [ 4588.263927] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4578s workers=2 idle: 657 [ 4596.058008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4596.061381] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4596.089709] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4596.096901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4596.101325] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4596.107492] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4596.115758] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4596.120762] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4596.124490] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4596.127516] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4596.131469] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4596.137767] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4596.152178] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4596.156742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4596.158861] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4596.163669] Call Trace: [ 4596.165318] [ 4596.165845] ? show_regs.cold.9+0x22/0x2f [ 4596.167726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4596.169797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4596.173035] ? watchdog+0x40/0x40 [ 4596.175351] ? hrtimer_interrupt+0x128/0x2c0 [ 4596.177100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4596.181441] ? apic_timer_interrupt+0xf/0x20 [ 4596.182987] [ 4596.183745] ? on_each_cpu_cond_mask+0x46/0x150 [ 4596.188587] ? __x64_sys_fspick+0x210/0x210 [ 4596.190613] ? smp_call_function_many_cond+0x3db/0x540 [ 4596.192908] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4596.194361] ? __x64_sys_fspick+0x210/0x210 [ 4596.198496] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4596.201086] on_each_cpu_cond_mask+0x46/0x150 [ 4596.203095] on_each_cpu_cond+0x1c/0x30 [ 4596.204476] invalidate_bh_lrus+0x27/0x30 [ 4596.208640] invalidate_bdev+0x5a/0x90 [ 4596.210902] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4596.214372] generic_shutdown_super+0xb1/0x1b0 [ 4596.216759] kill_block_super+0x29/0x80 [ 4596.218702] deactivate_locked_super+0x52/0xd0 [ 4596.220475] deactivate_super+0x83/0x90 [ 4596.221824] cleanup_mnt+0x5f/0xe0 [ 4596.223546] __cleanup_mnt+0x16/0x20 [ 4596.224747] task_work_run+0xc6/0x110 [ 4596.228670] exit_to_usermode_loop+0x1dd/0x1f0 [ 4596.230765] do_syscall_64+0x3d6/0x440 [ 4596.233092] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4596.235093] RIP: 0033:0x7fbb019648fb [ 4596.236501] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4596.253591] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4596.259856] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4596.263623] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4596.267608] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4596.274284] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4596.279585] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4604.059487] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4604.080314] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4604.120213] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4604.130176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4604.137546] Workqueue: events free_work [libcfs] [ 4604.143367] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4604.145647] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4604.164616] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4604.172417] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4604.176984] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4604.180960] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4604.185953] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4604.189524] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4604.195593] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4604.201496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4604.204394] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4604.210322] Call Trace: [ 4604.211373] [ 4604.212256] ? show_regs.cold.9+0x22/0x2f [ 4604.213699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4604.215651] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4604.217506] ? watchdog+0x40/0x40 [ 4604.218958] ? hrtimer_interrupt+0x128/0x2c0 [ 4604.220662] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4604.222595] ? apic_timer_interrupt+0xf/0x20 [ 4604.224344] [ 4604.225212] ? on_each_cpu+0x51/0x120 [ 4604.226702] ? smp_call_function_many_cond+0x3d9/0x540 [ 4604.228966] ? smp_call_function_many_cond+0x3a8/0x540 [ 4604.231095] ? load_new_mm_cr3+0x120/0x120 [ 4604.232857] ? load_new_mm_cr3+0x120/0x120 [ 4604.234432] on_each_cpu+0x51/0x120 [ 4604.235981] flush_tlb_kernel_range+0x93/0xb0 [ 4604.237746] free_unmap_vmap_area+0x63/0x70 [ 4604.239647] remove_vm_area+0xe7/0x110 [ 4604.241236] __vunmap+0x9c/0x400 [ 4604.242575] vfree+0x4d/0x100 [ 4604.243983] free_work+0x34/0x50 [libcfs] [ 4604.245625] process_one_work+0x2c8/0x700 [ 4604.247357] worker_thread+0x296/0x6e0 [ 4604.248953] ? rescuer_thread+0x570/0x570 [ 4604.250518] kthread+0x1d1/0x200 [ 4604.251882] ? set_kthread_struct+0x70/0x70 [ 4604.253578] ret_from_fork+0x1f/0x30 [ 4616.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4616.059407] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4616.085326] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4616.090083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4616.095196] Workqueue: events free_work [libcfs] [ 4616.098917] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4616.107925] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4616.121193] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4616.128114] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4616.131474] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4616.135429] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4616.141640] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4616.147120] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4616.152229] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4616.160540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4616.164519] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4616.167679] Call Trace: [ 4616.168484] [ 4616.171010] ? show_regs.cold.9+0x22/0x2f [ 4616.173088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4616.175047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4616.176941] ? watchdog+0x40/0x40 [ 4616.178733] ? hrtimer_interrupt+0x128/0x2c0 [ 4616.180424] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4616.182845] ? apic_timer_interrupt+0xf/0x20 [ 4616.184150] [ 4616.186457] ? on_each_cpu+0x51/0x120 [ 4616.188410] ? smp_call_function_many_cond+0x3db/0x540 [ 4616.191865] ? smp_call_function_many_cond+0x3a8/0x540 [ 4616.195564] ? load_new_mm_cr3+0x120/0x120 [ 4616.197467] ? load_new_mm_cr3+0x120/0x120 [ 4616.200605] on_each_cpu+0x51/0x120 [ 4616.203200] flush_tlb_kernel_range+0x93/0xb0 [ 4616.204873] free_unmap_vmap_area+0x63/0x70 [ 4616.207831] remove_vm_area+0xe7/0x110 [ 4616.210013] __vunmap+0x9c/0x400 [ 4616.210991] vfree+0x4d/0x100 [ 4616.213086] free_work+0x34/0x50 [libcfs] [ 4616.216149] process_one_work+0x2c8/0x700 [ 4616.220048] worker_thread+0x296/0x6e0 [ 4616.228995] ? rescuer_thread+0x570/0x570 [ 4616.232635] kthread+0x1d1/0x200 [ 4616.235647] ? set_kthread_struct+0x70/0x70 [ 4616.238040] ret_from_fork+0x1f/0x30 [ 4618.722033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2260s! [ 4618.730131] Showing busy workqueues and worker pools: [ 4618.732082] workqueue events: flags=0x0 [ 4618.734615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4618.738167] in-flight: 56293:free_work [libcfs] [ 4618.741033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4618.745917] in-flight: 22277:free_work [libcfs] [ 4618.750892] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4618.756071] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4618.760045] pending: nfs4_renew_state [nfsv4] [ 4618.764846] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4618.773553] in-flight: 47596:free_work [libcfs] [ 4618.776971] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4618.786619] workqueue events_unbound: flags=0x2 [ 4618.789946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4618.799257] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4618.803689] workqueue events_power_efficient: flags=0x80 [ 4618.806089] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4618.809807] pending: neigh_periodic_work, do_cache_clean [ 4618.818240] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4618.823290] pending: check_lifetime [ 4618.827097] workqueue mm_percpu_wq: flags=0x8 [ 4618.830747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4618.835856] pending: vmstat_update [ 4618.839899] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4618.845544] pending: vmstat_update [ 4618.850396] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4618.854837] pending: vmstat_update [ 4618.857886] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4618.860696] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4618.866747] pending: cgroup_pidlist_destroy_work_fn [ 4618.870063] workqueue writeback: flags=0x4a [ 4618.872496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4618.875720] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4618.878651] workqueue xprtiod: flags=0xa [ 4618.882973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4618.891025] pending: xprt_autoclose [ 4618.895635] workqueue ipv6_addrconf: flags=0x40008 [ 4618.898647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4618.901186] pending: addrconf_verify_work [ 4618.909875] workqueue knbd0-recv: flags=0x1a [ 4618.913698] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4618.916778] in-flight: 655:recv_work [ 4618.921146] workqueue ptlrpc_pinger: flags=0x2 [ 4618.923535] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4618.927563] pending: ptlrpc_pinger_main [ptlrpc] [ 4618.931922] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2261s workers=5 idle: 53684 27131 54850 46584 [ 4618.940225] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2261s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4618.949126] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2261s workers=6 idle: 53660 37674 46417 53664 46418 [ 4618.960854] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4608s workers=2 idle: 657 [ 4624.058016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4624.060871] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4624.078569] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4624.082477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4624.085642] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4624.087681] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4624.094236] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4624.097001] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4624.099569] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4624.102075] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4624.104710] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4624.107244] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4624.109884] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4624.112770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4624.114837] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4624.117302] Call Trace: [ 4624.118214] [ 4624.118994] ? show_regs.cold.9+0x22/0x2f [ 4624.120472] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4624.122164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4624.123863] ? watchdog+0x40/0x40 [ 4624.125079] ? hrtimer_interrupt+0x128/0x2c0 [ 4624.126670] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4624.128838] ? apic_timer_interrupt+0xf/0x20 [ 4624.131414] [ 4624.132169] ? on_each_cpu_cond_mask+0x46/0x150 [ 4624.135524] ? __x64_sys_fspick+0x210/0x210 [ 4624.138294] ? smp_call_function_many_cond+0x3d9/0x540 [ 4624.140156] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4624.142772] ? __x64_sys_fspick+0x210/0x210 [ 4624.144174] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4624.146757] on_each_cpu_cond_mask+0x46/0x150 [ 4624.148319] on_each_cpu_cond+0x1c/0x30 [ 4624.151385] invalidate_bh_lrus+0x27/0x30 [ 4624.153174] invalidate_bdev+0x5a/0x90 [ 4624.154587] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4624.156502] generic_shutdown_super+0xb1/0x1b0 [ 4624.159986] kill_block_super+0x29/0x80 [ 4624.161451] deactivate_locked_super+0x52/0xd0 [ 4624.163300] deactivate_super+0x83/0x90 [ 4624.164835] cleanup_mnt+0x5f/0xe0 [ 4624.166796] __cleanup_mnt+0x16/0x20 [ 4624.168272] task_work_run+0xc6/0x110 [ 4624.169664] exit_to_usermode_loop+0x1dd/0x1f0 [ 4624.171274] do_syscall_64+0x3d6/0x440 [ 4624.174234] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4624.176022] RIP: 0033:0x7fbb019648fb [ 4624.177817] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4624.187865] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4624.191790] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4624.194506] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4624.198251] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4624.201027] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4624.204566] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4632.064419] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4632.071921] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4632.104810] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4632.108641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4632.112456] Workqueue: events free_work [libcfs] [ 4632.115223] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4632.121077] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4632.130052] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4632.135026] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4632.139417] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4632.143782] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4632.148427] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4632.152143] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4632.156026] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4632.160819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4632.163324] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4632.166502] Call Trace: [ 4632.167596] [ 4632.168793] ? show_regs.cold.9+0x22/0x2f [ 4632.170631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4632.172756] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4632.174667] ? watchdog+0x40/0x40 [ 4632.176106] ? hrtimer_interrupt+0x128/0x2c0 [ 4632.178124] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4632.180398] ? apic_timer_interrupt+0xf/0x20 [ 4632.182399] [ 4632.183383] ? on_each_cpu+0x51/0x120 [ 4632.185042] ? smp_call_function_many_cond+0x3db/0x540 [ 4632.187493] ? smp_call_function_many_cond+0x3a8/0x540 [ 4632.193859] ? load_new_mm_cr3+0x120/0x120 [ 4632.195451] ? load_new_mm_cr3+0x120/0x120 [ 4632.198978] on_each_cpu+0x51/0x120 [ 4632.201109] flush_tlb_kernel_range+0x93/0xb0 [ 4632.203095] free_unmap_vmap_area+0x63/0x70 [ 4632.205324] remove_vm_area+0xe7/0x110 [ 4632.208048] __vunmap+0x9c/0x400 [ 4632.211707] vfree+0x4d/0x100 [ 4632.214000] free_work+0x34/0x50 [libcfs] [ 4632.219441] process_one_work+0x2c8/0x700 [ 4632.221687] worker_thread+0x296/0x6e0 [ 4632.223744] ? rescuer_thread+0x570/0x570 [ 4632.226110] kthread+0x1d1/0x200 [ 4632.227407] ? set_kthread_struct+0x70/0x70 [ 4632.229039] ret_from_fork+0x1f/0x30 [ 4644.055269] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4644.060470] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4644.089043] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4644.096631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4644.101278] Workqueue: events free_work [libcfs] [ 4644.103273] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4644.107361] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4644.117094] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4644.126119] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4644.129860] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4644.135296] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4644.140312] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4644.143569] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4644.146563] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4644.150087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4644.152979] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4644.156238] Call Trace: [ 4644.157248] [ 4644.158353] ? show_regs.cold.9+0x22/0x2f [ 4644.160531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4644.162416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4644.165417] ? watchdog+0x40/0x40 [ 4644.166984] ? hrtimer_interrupt+0x128/0x2c0 [ 4644.169346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4644.171501] ? apic_timer_interrupt+0xf/0x20 [ 4644.173177] [ 4644.174074] ? on_each_cpu+0x51/0x120 [ 4644.175527] ? smp_call_function_many_cond+0x3db/0x540 [ 4644.177491] ? smp_call_function_many_cond+0x3a8/0x540 [ 4644.179624] ? load_new_mm_cr3+0x120/0x120 [ 4644.181312] ? load_new_mm_cr3+0x120/0x120 [ 4644.183018] on_each_cpu+0x51/0x120 [ 4644.184756] flush_tlb_kernel_range+0x93/0xb0 [ 4644.186498] free_unmap_vmap_area+0x63/0x70 [ 4644.188222] remove_vm_area+0xe7/0x110 [ 4644.189867] __vunmap+0x9c/0x400 [ 4644.191384] vfree+0x4d/0x100 [ 4644.193005] free_work+0x34/0x50 [libcfs] [ 4644.196243] process_one_work+0x2c8/0x700 [ 4644.198982] worker_thread+0x296/0x6e0 [ 4644.200796] ? rescuer_thread+0x570/0x570 [ 4644.203088] kthread+0x1d1/0x200 [ 4644.204398] ? set_kthread_struct+0x70/0x70 [ 4644.205988] ret_from_fork+0x1f/0x30 [ 4649.439764] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2291s! [ 4649.444638] Showing busy workqueues and worker pools: [ 4649.448353] workqueue events: flags=0x0 [ 4649.449771] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.453468] in-flight: 56293:free_work [libcfs] [ 4649.457859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4649.470135] in-flight: 22277:free_work [libcfs] [ 4649.476552] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4649.479867] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.484301] pending: nfs4_renew_state [nfsv4] [ 4649.488102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4649.491322] in-flight: 47596:free_work [libcfs] [ 4649.497194] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4649.507135] workqueue events_unbound: flags=0x2 [ 4649.509026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4649.511682] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4649.518296] workqueue events_power_efficient: flags=0x80 [ 4649.522632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4649.531527] pending: neigh_periodic_work, do_cache_clean [ 4649.538294] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.544654] pending: check_lifetime [ 4649.550240] workqueue mm_percpu_wq: flags=0x8 [ 4649.551478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.559078] pending: vmstat_update [ 4649.560510] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.568506] pending: vmstat_update [ 4649.572254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.577906] pending: vmstat_update [ 4649.578857] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4649.592383] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4649.594084] pending: cgroup_pidlist_destroy_work_fn [ 4649.595423] workqueue writeback: flags=0x4a [ 4649.602862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4649.609079] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4649.612368] workqueue xprtiod: flags=0xa [ 4649.614976] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4649.618287] pending: xprt_autoclose [ 4649.620321] workqueue ipv6_addrconf: flags=0x40008 [ 4649.622361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4649.625882] pending: addrconf_verify_work [ 4649.628257] workqueue knbd0-recv: flags=0x1a [ 4649.630065] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4649.634110] in-flight: 655:recv_work [ 4649.636762] workqueue ptlrpc_pinger: flags=0x2 [ 4649.639141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4649.642742] pending: ptlrpc_pinger_main [ptlrpc] [ 4649.646343] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2291s workers=5 idle: 53684 27131 54850 46584 [ 4649.651614] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2291s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4649.657396] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2291s workers=6 idle: 53660 37674 46417 53664 46418 [ 4649.661862] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4639s workers=2 idle: 657 [ 4652.059090] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4652.061235] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4652.097326] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4652.105576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4652.110619] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4652.113575] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4652.130030] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4652.138246] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4652.142820] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4652.147164] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4652.151127] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4652.156397] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4652.158875] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4652.162126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4652.164435] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4652.167496] Call Trace: [ 4652.168577] [ 4652.169410] ? show_regs.cold.9+0x22/0x2f [ 4652.171119] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4652.173217] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4652.175229] ? watchdog+0x40/0x40 [ 4652.176646] ? hrtimer_interrupt+0x128/0x2c0 [ 4652.178532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4652.180443] ? apic_timer_interrupt+0xf/0x20 [ 4652.182591] [ 4652.183333] ? on_each_cpu_cond_mask+0x46/0x150 [ 4652.184815] ? __x64_sys_fspick+0x210/0x210 [ 4652.186968] ? smp_call_function_many_cond+0x3d9/0x540 [ 4652.191641] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4652.193883] ? __x64_sys_fspick+0x210/0x210 [ 4652.196017] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4652.198613] on_each_cpu_cond_mask+0x46/0x150 [ 4652.200752] on_each_cpu_cond+0x1c/0x30 [ 4652.202676] invalidate_bh_lrus+0x27/0x30 [ 4652.204670] invalidate_bdev+0x5a/0x90 [ 4652.206641] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4652.209125] generic_shutdown_super+0xb1/0x1b0 [ 4652.210886] kill_block_super+0x29/0x80 [ 4652.213577] deactivate_locked_super+0x52/0xd0 [ 4652.216026] deactivate_super+0x83/0x90 [ 4652.218482] cleanup_mnt+0x5f/0xe0 [ 4652.220360] __cleanup_mnt+0x16/0x20 [ 4652.222478] task_work_run+0xc6/0x110 [ 4652.225944] exit_to_usermode_loop+0x1dd/0x1f0 [ 4652.229193] do_syscall_64+0x3d6/0x440 [ 4652.232651] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4652.234591] RIP: 0033:0x7fbb019648fb [ 4652.237916] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4652.251896] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4652.256894] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4652.261467] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4652.265965] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4652.270544] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4652.275567] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4660.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4660.066903] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4660.105099] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4660.110908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4660.115019] Workqueue: events free_work [libcfs] [ 4660.119035] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4660.120792] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4660.132067] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4660.136082] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4660.140488] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4660.144404] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4660.147711] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4660.150744] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4660.153972] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4660.157749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4660.160450] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4660.164107] Call Trace: [ 4660.164993] [ 4660.166017] ? show_regs.cold.9+0x22/0x2f [ 4660.167861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4660.171310] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4660.173288] ? watchdog+0x40/0x40 [ 4660.174835] ? hrtimer_interrupt+0x128/0x2c0 [ 4660.176860] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4660.179055] ? apic_timer_interrupt+0xf/0x20 [ 4660.180807] [ 4660.181582] ? on_each_cpu+0x51/0x120 [ 4660.183145] ? smp_call_function_many_cond+0x3db/0x540 [ 4660.185467] ? smp_call_function_many_cond+0x3a8/0x540 [ 4660.187825] ? load_new_mm_cr3+0x120/0x120 [ 4660.189510] ? load_new_mm_cr3+0x120/0x120 [ 4660.192213] on_each_cpu+0x51/0x120 [ 4660.194459] flush_tlb_kernel_range+0x93/0xb0 [ 4660.197041] free_unmap_vmap_area+0x63/0x70 [ 4660.199287] remove_vm_area+0xe7/0x110 [ 4660.201485] __vunmap+0x9c/0x400 [ 4660.203345] vfree+0x4d/0x100 [ 4660.206071] free_work+0x34/0x50 [libcfs] [ 4660.208420] process_one_work+0x2c8/0x700 [ 4660.212362] worker_thread+0x296/0x6e0 [ 4660.214431] ? rescuer_thread+0x570/0x570 [ 4660.219209] kthread+0x1d1/0x200 [ 4660.228365] ? set_kthread_struct+0x70/0x70 [ 4660.231295] ret_from_fork+0x1f/0x30 [ 4676.056720] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4676.068042] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4676.108310] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4676.115135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4676.120235] Workqueue: events free_work [libcfs] [ 4676.123064] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4676.126208] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4676.146627] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4676.150515] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4676.154231] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4676.160875] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4676.165668] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4676.168863] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4676.171978] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4676.178133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4676.181982] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4676.185720] Call Trace: [ 4676.187768] [ 4676.188927] ? show_regs.cold.9+0x22/0x2f [ 4676.191189] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4676.197191] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4676.204624] ? watchdog+0x40/0x40 [ 4676.208649] ? hrtimer_interrupt+0x128/0x2c0 [ 4676.210325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4676.219538] ? apic_timer_interrupt+0xf/0x20 [ 4676.224245] [ 4676.228367] ? on_each_cpu+0x51/0x120 [ 4676.230472] ? smp_call_function_many_cond+0x3db/0x540 [ 4676.234004] ? smp_call_function_many_cond+0x3a8/0x540 [ 4676.237171] ? load_new_mm_cr3+0x120/0x120 [ 4676.243872] ? load_new_mm_cr3+0x120/0x120 [ 4676.245480] on_each_cpu+0x51/0x120 [ 4676.247182] flush_tlb_kernel_range+0x93/0xb0 [ 4676.249451] free_unmap_vmap_area+0x63/0x70 [ 4676.254593] remove_vm_area+0xe7/0x110 [ 4676.256487] __vunmap+0x9c/0x400 [ 4676.258549] vfree+0x4d/0x100 [ 4676.260172] free_work+0x34/0x50 [libcfs] [ 4676.262083] process_one_work+0x2c8/0x700 [ 4676.264019] worker_thread+0x296/0x6e0 [ 4676.265893] ? rescuer_thread+0x570/0x570 [ 4676.267894] kthread+0x1d1/0x200 [ 4676.269379] ? set_kthread_struct+0x70/0x70 [ 4676.271084] ret_from_fork+0x1f/0x30 [ 4680.058243] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4680.063048] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4680.093902] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4680.098941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4680.103697] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4680.105928] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4680.113624] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4680.117519] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4680.120997] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4680.124696] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4680.128674] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4680.132249] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4680.135244] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4680.138670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4680.141018] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4680.145633] Call Trace: [ 4680.146827] [ 4680.147851] ? show_regs.cold.9+0x22/0x2f [ 4680.150388] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4680.152792] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4680.159032] ? watchdog+0x40/0x40 [ 4680.161027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 [ 4680.161767] ? hrtimer_interrupt+0x128/0x2c0 [ 4680.161774] stuck for 2321s! [ 4680.161782] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4680.167342] Showing busy workqueues and worker pools: [ 4680.167348] workqueue events: flags=0x0 [ 4680.167354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4680.167367] in-flight: 56293:free_work [libcfs] [ 4680.167414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4680.167423] in-flight: 22277:free_work [libcfs] [ 4680.167448] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4680.170276] ? apic_timer_interrupt+0xf/0x20 [ 4680.172722] [ 4680.172752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4680.172810] pending: [ 4680.176584] [ 4680.181474] nfs4_renew_state [nfsv4] [ 4680.183222] ? on_each_cpu_cond_mask+0x46/0x150 [ 4680.189769] [ 4680.193521] ? __x64_sys_fspick+0x210/0x210 [ 4680.193541] ? smp_call_function_many_cond+0x3d9/0x540 [ 4680.193548] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4680.193552] ? __x64_sys_fspick+0x210/0x210 [ 4680.193557] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4680.193560] on_each_cpu_cond_mask+0x46/0x150 [ 4680.193566] on_each_cpu_cond+0x1c/0x30 [ 4680.193570] invalidate_bh_lrus+0x27/0x30 [ 4680.193573] invalidate_bdev+0x5a/0x90 [ 4680.193581] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4680.193713] generic_shutdown_super+0xb1/0x1b0 [ 4680.193723] kill_block_super+0x29/0x80 [ 4680.193728] deactivate_locked_super+0x52/0xd0 [ 4680.193733] deactivate_super+0x83/0x90 [ 4680.193737] cleanup_mnt+0x5f/0xe0 [ 4680.193743] __cleanup_mnt+0x16/0x20 [ 4680.193747] task_work_run+0xc6/0x110 [ 4680.193755] exit_to_usermode_loop+0x1dd/0x1f0 [ 4680.193763] do_syscall_64+0x3d6/0x440 [ 4680.193767] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4680.193775] RIP: 0033:0x7fbb019648fb [ 4680.193783] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4680.193788] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4680.193793] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4680.193795] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4680.193797] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4680.193799] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4680.193801] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4680.363165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4680.364686] in-flight: 47596:free_work [libcfs] [ 4680.368960] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4680.373285] workqueue events_unbound: flags=0x2 [ 4680.377065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4680.380395] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4680.386663] workqueue events_power_efficient: flags=0x80 [ 4680.391138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4680.396677] pending: neigh_periodic_work, do_cache_clean [ 4680.402830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4680.411267] pending: check_lifetime [ 4680.414128] workqueue mm_percpu_wq: flags=0x8 [ 4680.416535] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4680.420044] pending: vmstat_update [ 4680.424794] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4680.438266] pending: vmstat_update [ 4680.439717] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4680.451663] pending: vmstat_update [ 4680.454426] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4680.457994] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4680.463242] pending: cgroup_pidlist_destroy_work_fn [ 4680.470389] workqueue writeback: flags=0x4a [ 4680.476751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4680.482191] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4680.486995] workqueue xprtiod: flags=0xa [ 4680.491076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4680.500151] pending: xprt_autoclose [ 4680.501762] workqueue ipv6_addrconf: flags=0x40008 [ 4680.505682] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4680.512013] pending: addrconf_verify_work [ 4680.516929] workqueue knbd0-recv: flags=0x1a [ 4680.519461] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4680.522682] in-flight: 655:recv_work [ 4680.525955] workqueue ptlrpc_pinger: flags=0x2 [ 4680.530705] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4680.535686] pending: ptlrpc_pinger_main [ptlrpc] [ 4680.541032] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2322s workers=5 idle: 53684 27131 54850 46584 [ 4680.546307] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2322s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4680.552214] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2322s workers=6 idle: 53660 37674 46417 53664 46418 [ 4680.556868] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4670s workers=2 idle: 657 [ 4688.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4688.062523] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4688.098972] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4688.112189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4688.118378] Workqueue: events free_work [libcfs] [ 4688.125126] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4688.134068] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4688.147379] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4688.152106] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4688.154508] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4688.158113] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4688.163325] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4688.167495] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4688.171543] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4688.176575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4688.178653] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4688.185249] Call Trace: [ 4688.186681] [ 4688.188251] ? show_regs.cold.9+0x22/0x2f [ 4688.192263] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4688.195076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4688.198552] ? watchdog+0x40/0x40 [ 4688.200884] ? hrtimer_interrupt+0x128/0x2c0 [ 4688.204474] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4688.206657] ? apic_timer_interrupt+0xf/0x20 [ 4688.209054] [ 4688.210472] ? on_each_cpu+0x51/0x120 [ 4688.213472] ? smp_call_function_many_cond+0x3db/0x540 [ 4688.219304] ? smp_call_function_many_cond+0x3a8/0x540 [ 4688.224200] ? load_new_mm_cr3+0x120/0x120 [ 4688.227680] ? load_new_mm_cr3+0x120/0x120 [ 4688.229802] on_each_cpu+0x51/0x120 [ 4688.232991] flush_tlb_kernel_range+0x93/0xb0 [ 4688.235561] free_unmap_vmap_area+0x63/0x70 [ 4688.237820] remove_vm_area+0xe7/0x110 [ 4688.240595] __vunmap+0x9c/0x400 [ 4688.243249] vfree+0x4d/0x100 [ 4688.245630] free_work+0x34/0x50 [libcfs] [ 4688.248567] process_one_work+0x2c8/0x700 [ 4688.251805] worker_thread+0x296/0x6e0 [ 4688.253058] ? rescuer_thread+0x570/0x570 [ 4688.255499] kthread+0x1d1/0x200 [ 4688.257840] ? set_kthread_struct+0x70/0x70 [ 4688.261515] ret_from_fork+0x1f/0x30 [ 4708.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4708.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4708.064110] Modules linked in: [ 4708.067969] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4708.068052] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4708.068059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4708.068063] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4708.068075] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4708.068079] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4708.068082] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4708.068085] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4708.068088] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4708.068090] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4708.068093] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4708.068096] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4708.068099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4708.068101] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4708.068109] Call Trace: [ 4708.068114] [ 4708.068119] ? show_regs.cold.9+0x22/0x2f [ 4708.068130] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4708.068136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4708.068140] ? watchdog+0x40/0x40 [ 4708.068144] ? hrtimer_interrupt+0x128/0x2c0 [ 4708.068148] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4708.068156] ? apic_timer_interrupt+0xf/0x20 [ 4708.068160] [ 4708.068162] ? on_each_cpu_cond_mask+0x46/0x150 [ 4708.068165] ? __x64_sys_fspick+0x210/0x210 [ 4708.068173] ? smp_call_function_many_cond+0x3db/0x540 [ 4708.068176] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4708.068180] ? __x64_sys_fspick+0x210/0x210 [ 4708.068184] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4708.068188] on_each_cpu_cond_mask+0x46/0x150 [ 4708.068192] on_each_cpu_cond+0x1c/0x30 [ 4708.068196] invalidate_bh_lrus+0x27/0x30 [ 4708.068199] invalidate_bdev+0x5a/0x90 [ 4708.068207] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4708.068575] generic_shutdown_super+0xb1/0x1b0 [ 4708.068681] kill_block_super+0x29/0x80 [ 4708.068689] deactivate_locked_super+0x52/0xd0 [ 4708.068694] deactivate_super+0x83/0x90 [ 4708.068698] cleanup_mnt+0x5f/0xe0 [ 4708.068703] __cleanup_mnt+0x16/0x20 [ 4708.068707] task_work_run+0xc6/0x110 [ 4708.068714] exit_to_usermode_loop+0x1dd/0x1f0 [ 4708.068720] do_syscall_64+0x3d6/0x440 [ 4708.068724] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4708.068738] RIP: 0033:0x7fbb019648fb [ 4708.068745] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4708.068749] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4708.068753] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4708.068756] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4708.068758] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4708.068760] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4708.068762] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4708.415439] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4708.459017] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4708.467138] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4708.472347] Workqueue: events free_work [libcfs] [ 4708.474522] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4708.479431] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4708.493845] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4708.498472] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4708.503114] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4708.511400] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4708.518263] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4708.526852] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4708.537682] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4708.551897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4708.555108] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4708.559028] Call Trace: [ 4708.560299] [ 4708.561161] ? show_regs.cold.9+0x22/0x2f [ 4708.566182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4708.568619] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4708.570611] ? watchdog+0x40/0x40 [ 4708.572300] ? hrtimer_interrupt+0x128/0x2c0 [ 4708.574586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4708.576499] ? apic_timer_interrupt+0xf/0x20 [ 4708.578960] [ 4708.579987] ? on_each_cpu+0x51/0x120 [ 4708.581901] ? smp_call_function_many_cond+0x3db/0x540 [ 4708.584475] ? smp_call_function_many_cond+0x3a8/0x540 [ 4708.587114] ? load_new_mm_cr3+0x120/0x120 [ 4708.589136] ? load_new_mm_cr3+0x120/0x120 [ 4708.591264] on_each_cpu+0x51/0x120 [ 4708.592958] flush_tlb_kernel_range+0x93/0xb0 [ 4708.594620] free_unmap_vmap_area+0x63/0x70 [ 4708.596710] remove_vm_area+0xe7/0x110 [ 4708.598576] __vunmap+0x9c/0x400 [ 4708.600078] vfree+0x4d/0x100 [ 4708.601814] free_work+0x34/0x50 [libcfs] [ 4708.603820] process_one_work+0x2c8/0x700 [ 4708.605695] worker_thread+0x296/0x6e0 [ 4708.607824] ? rescuer_thread+0x570/0x570 [ 4708.609764] kthread+0x1d1/0x200 [ 4708.611332] ? set_kthread_struct+0x70/0x70 [ 4708.613165] ret_from_fork+0x1f/0x30 [ 4710.881662] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2352s! [ 4710.895276] Showing busy workqueues and worker pools: [ 4710.899745] workqueue events: flags=0x0 [ 4710.904432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4710.908230] in-flight: 56293:free_work [libcfs] [ 4710.914890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4710.920162] in-flight: 22277:free_work [libcfs] [ 4710.924422] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4710.929236] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4710.938008] pending: nfs4_renew_state [nfsv4] [ 4710.943976] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4710.948361] in-flight: 47596:free_work [libcfs] [ 4710.954754] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4710.966362] workqueue events_unbound: flags=0x2 [ 4710.971335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4710.979185] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4710.987570] workqueue events_power_efficient: flags=0x80 [ 4710.993496] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4711.002020] pending: neigh_periodic_work, do_cache_clean [ 4711.013222] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4711.018233] pending: check_lifetime [ 4711.021288] workqueue mm_percpu_wq: flags=0x8 [ 4711.023017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4711.031343] pending: vmstat_update [ 4711.034194] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4711.041889] pending: vmstat_update [ 4711.045739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4711.051246] pending: vmstat_update [ 4711.055646] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4711.060313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4711.063996] pending: cgroup_pidlist_destroy_work_fn [ 4711.069257] workqueue writeback: flags=0x4a [ 4711.072451] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4711.076843] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4711.082568] workqueue xprtiod: flags=0xa [ 4711.085230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4711.088997] pending: xprt_autoclose [ 4711.092964] workqueue ipv6_addrconf: flags=0x40008 [ 4711.096019] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4711.099181] pending: addrconf_verify_work [ 4711.100849] workqueue knbd0-recv: flags=0x1a [ 4711.104833] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4711.111914] in-flight: 655:recv_work [ 4711.113985] workqueue ptlrpc_pinger: flags=0x2 [ 4711.116302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4711.121035] pending: ptlrpc_pinger_main [ptlrpc] [ 4711.127026] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2353s workers=5 idle: 53684 27131 54850 46584 [ 4711.131473] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2353s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4711.147958] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2353s workers=6 idle: 53660 37674 46417 53664 46418 [ 4711.157660] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4700s workers=2 idle: 657 [ 4716.061271] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4716.065848] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4716.106805] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4716.117321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4716.128795] Workqueue: events free_work [libcfs] [ 4716.135213] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4716.141975] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4716.154750] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4716.161793] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4716.164217] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4716.169815] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4716.174819] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4716.181691] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4716.185072] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4716.192010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4716.197230] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4716.204372] Call Trace: [ 4716.209924] [ 4716.210828] ? show_regs.cold.9+0x22/0x2f [ 4716.219928] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4716.221661] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4716.224312] ? watchdog+0x40/0x40 [ 4716.225960] ? hrtimer_interrupt+0x128/0x2c0 [ 4716.230125] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4716.231803] ? apic_timer_interrupt+0xf/0x20 [ 4716.239735] [ 4716.242033] ? on_each_cpu+0x51/0x120 [ 4716.245925] ? smp_call_function_many_cond+0x3db/0x540 [ 4716.249551] ? smp_call_function_many_cond+0x3a8/0x540 [ 4716.255763] ? load_new_mm_cr3+0x120/0x120 [ 4716.257795] ? load_new_mm_cr3+0x120/0x120 [ 4716.260659] on_each_cpu+0x51/0x120 [ 4716.262506] flush_tlb_kernel_range+0x93/0xb0 [ 4716.268487] free_unmap_vmap_area+0x63/0x70 [ 4716.271426] remove_vm_area+0xe7/0x110 [ 4716.275035] __vunmap+0x9c/0x400 [ 4716.280863] vfree+0x4d/0x100 [ 4716.282569] free_work+0x34/0x50 [libcfs] [ 4716.285648] process_one_work+0x2c8/0x700 [ 4716.288900] worker_thread+0x296/0x6e0 [ 4716.292726] ? rescuer_thread+0x570/0x570 [ 4716.296602] kthread+0x1d1/0x200 [ 4716.300114] ? set_kthread_struct+0x70/0x70 [ 4716.304104] ret_from_fork+0x1f/0x30 [ 4736.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4736.063456] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4736.065661] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4736.065751] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4736.065759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4736.065765] Workqueue: events free_work [libcfs] [ 4736.065809] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4736.065817] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4736.065821] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4736.065825] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4736.065828] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4736.065830] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4736.065832] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4736.065835] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4736.065838] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4736.065840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4736.065843] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4736.065851] Call Trace: [ 4736.065855] [ 4736.065860] ? show_regs.cold.9+0x22/0x2f [ 4736.065869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4736.065874] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4736.072109] Modules linked in: zfs(O) [ 4736.140138] ? watchdog+0x40/0x40 [ 4736.152286] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4736.152377] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4736.152384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4736.152387] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4736.152400] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4736.152404] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4736.161448] ? hrtimer_interrupt+0x128/0x2c0 [ 4736.161471] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4736.161479] ? apic_timer_interrupt+0xf/0x20 [ 4736.161488] [ 4736.161491] ? on_each_cpu+0x51/0x120 [ 4736.161500] ? smp_call_function_many_cond+0x3db/0x540 [ 4736.161505] ? smp_call_function_many_cond+0x3a8/0x540 [ 4736.161508] ? load_new_mm_cr3+0x120/0x120 [ 4736.161516] ? load_new_mm_cr3+0x120/0x120 [ 4736.161520] on_each_cpu+0x51/0x120 [ 4736.161528] flush_tlb_kernel_range+0x93/0xb0 [ 4736.161534] free_unmap_vmap_area+0x63/0x70 [ 4736.161635] remove_vm_area+0xe7/0x110 [ 4736.161641] __vunmap+0x9c/0x400 [ 4736.161645] vfree+0x4d/0x100 [ 4736.161651] free_work+0x34/0x50 [libcfs] [ 4736.161690] process_one_work+0x2c8/0x700 [ 4736.161697] worker_thread+0x296/0x6e0 [ 4736.161705] ? rescuer_thread+0x570/0x570 [ 4736.161709] kthread+0x1d1/0x200 [ 4736.161716] ? set_kthread_struct+0x70/0x70 [ 4736.161720] ret_from_fork+0x1f/0x30 [ 4736.420942] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4736.424792] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4736.428834] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4736.431911] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4736.435187] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4736.439053] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4736.443422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4736.446414] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4736.450319] Call Trace: [ 4736.451524] [ 4736.455451] ? show_regs.cold.9+0x22/0x2f [ 4736.458479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4736.466016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4736.468420] ? watchdog+0x40/0x40 [ 4736.470743] ? hrtimer_interrupt+0x128/0x2c0 [ 4736.473834] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4736.477209] ? apic_timer_interrupt+0xf/0x20 [ 4736.479359] [ 4736.480206] ? on_each_cpu_cond_mask+0x46/0x150 [ 4736.486566] ? __x64_sys_fspick+0x210/0x210 [ 4736.488139] ? smp_call_function_many_cond+0x3db/0x540 [ 4736.496675] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4736.498137] ? __x64_sys_fspick+0x210/0x210 [ 4736.503063] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4736.505748] on_each_cpu_cond_mask+0x46/0x150 [ 4736.511628] on_each_cpu_cond+0x1c/0x30 [ 4736.515548] invalidate_bh_lrus+0x27/0x30 [ 4736.519791] invalidate_bdev+0x5a/0x90 [ 4736.522500] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4736.525488] generic_shutdown_super+0xb1/0x1b0 [ 4736.527578] kill_block_super+0x29/0x80 [ 4736.530052] deactivate_locked_super+0x52/0xd0 [ 4736.535348] deactivate_super+0x83/0x90 [ 4736.539470] cleanup_mnt+0x5f/0xe0 [ 4736.543700] __cleanup_mnt+0x16/0x20 [ 4736.548484] task_work_run+0xc6/0x110 [ 4736.554453] exit_to_usermode_loop+0x1dd/0x1f0 [ 4736.557831] do_syscall_64+0x3d6/0x440 [ 4736.560500] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4736.563802] RIP: 0033:0x7fbb019648fb [ 4736.568419] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4736.588788] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4736.593313] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4736.598359] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4736.602596] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4736.606603] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4736.610159] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4741.600672] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2383s! [ 4741.607272] Showing busy workqueues and worker pools: [ 4741.613361] workqueue events: flags=0x0 [ 4741.615970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4741.625589] in-flight: 56293:free_work [libcfs] [ 4741.627725] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4741.632601] in-flight: 22277:free_work [libcfs] [ 4741.635501] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4741.639700] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4741.643167] pending: nfs4_renew_state [nfsv4] [ 4741.645225] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4741.648482] in-flight: 47596:free_work [libcfs] [ 4741.650873] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4741.655185] workqueue events_unbound: flags=0x2 [ 4741.657378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4741.659924] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4741.665040] workqueue events_power_efficient: flags=0x80 [ 4741.668159] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4741.673219] pending: neigh_periodic_work, do_cache_clean [ 4741.678629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4741.683142] pending: check_lifetime [ 4741.686163] workqueue mm_percpu_wq: flags=0x8 [ 4741.688328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4741.692190] pending: vmstat_update [ 4741.694262] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4741.698022] pending: vmstat_update [ 4741.699639] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4741.703027] pending: vmstat_update [ 4741.704771] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4741.707496] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4741.711177] pending: cgroup_pidlist_destroy_work_fn [ 4741.714843] workqueue writeback: flags=0x4a [ 4741.718215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4741.721924] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4741.725481] workqueue xprtiod: flags=0xa [ 4741.733375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4741.738949] pending: xprt_autoclose [ 4741.740746] workqueue ipv6_addrconf: flags=0x40008 [ 4741.745968] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4741.753402] pending: addrconf_verify_work [ 4741.754543] workqueue knbd0-recv: flags=0x1a [ 4741.755776] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4741.757883] in-flight: 655:recv_work [ 4741.759556] workqueue ptlrpc_pinger: flags=0x2 [ 4741.761564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4741.767291] pending: ptlrpc_pinger_main [ptlrpc] [ 4741.771721] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2384s workers=5 idle: 53684 27131 54850 46584 [ 4741.774868] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2384s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4741.784639] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2384s workers=6 idle: 53660 37674 46417 53664 46418 [ 4741.787135] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4731s workers=2 idle: 657 [ 4744.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4744.066700] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4744.100014] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4744.110409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4744.120456] Workqueue: events free_work [libcfs] [ 4744.124423] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4744.127668] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4744.137231] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4744.140696] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4744.144466] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4744.148113] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4744.151911] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4744.159057] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4744.166246] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4744.175416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4744.185067] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4744.187907] Call Trace: [ 4744.193158] [ 4744.193972] ? show_regs.cold.9+0x22/0x2f [ 4744.195614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4744.203588] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4744.205937] ? watchdog+0x40/0x40 [ 4744.207703] ? hrtimer_interrupt+0x128/0x2c0 [ 4744.210084] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4744.212411] ? apic_timer_interrupt+0xf/0x20 [ 4744.214673] [ 4744.215905] ? on_each_cpu+0x51/0x120 [ 4744.223487] ? smp_call_function_many_cond+0x3db/0x540 [ 4744.230236] ? smp_call_function_many_cond+0x3a8/0x540 [ 4744.232509] ? load_new_mm_cr3+0x120/0x120 [ 4744.237232] ? load_new_mm_cr3+0x120/0x120 [ 4744.242402] on_each_cpu+0x51/0x120 [ 4744.245183] flush_tlb_kernel_range+0x93/0xb0 [ 4744.249932] free_unmap_vmap_area+0x63/0x70 [ 4744.251632] remove_vm_area+0xe7/0x110 [ 4744.260359] __vunmap+0x9c/0x400 [ 4744.265209] vfree+0x4d/0x100 [ 4744.267489] free_work+0x34/0x50 [libcfs] [ 4744.274713] process_one_work+0x2c8/0x700 [ 4744.281071] worker_thread+0x296/0x6e0 [ 4744.282651] ? rescuer_thread+0x570/0x570 [ 4744.286658] kthread+0x1d1/0x200 [ 4744.293263] ? set_kthread_struct+0x70/0x70 [ 4744.295118] ret_from_fork+0x1f/0x30 [ 4757.800008] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4757.807905] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 4757.813709] (detected by 0, t=2400083 jiffies, g=626849, q=1398) [ 4757.817592] Sending NMI from CPU 0 to CPUs 3: [ 4757.820881] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 4757.820891] rcu: rcu_sched kthread starved for 2399732 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4757.830253] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4757.833686] rcu: RCU grace-period kthread stack dump: [ 4757.836637] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4757.845261] Call Trace: [ 4757.848202] __schedule+0x351/0xcb0 [ 4757.851768] schedule+0xc0/0x180 [ 4757.853068] schedule_timeout+0xb4/0x190 [ 4757.854607] ? do_raw_spin_unlock+0x75/0x190 [ 4757.856799] ? __next_timer_interrupt+0x160/0x160 [ 4757.859945] ? prepare_to_swait_event+0xf3/0x140 [ 4757.862759] rcu_gp_kthread+0x8dd/0x15b0 [ 4757.864913] ? force_qs_rnp+0x2c0/0x2c0 [ 4757.866808] kthread+0x1d1/0x200 [ 4757.871166] ? set_kthread_struct+0x70/0x70 [ 4757.873144] ret_from_fork+0x1f/0x30 [ 4757.875552] rcu: Stack dump where RCU GP kthread last ran: [ 4757.889048] Sending NMI from CPU 0 to CPUs 1: [ 4757.893205] NMI backtrace for cpu 1 [ 4757.893211] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4757.893213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4757.893215] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4757.893217] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4757.893218] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 [ 4757.893221] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4757.893222] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4757.893223] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4757.893224] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4757.893225] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4757.893227] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4757.893228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4757.893229] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4757.893230] Call Trace: [ 4757.893231] [ 4757.893232] ? show_regs.cold.9+0x22/0x2f [ 4757.893233] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4757.893235] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4757.893236] ? nmi_handle+0xbd/0x210 [ 4757.893237] ? default_do_nmi+0x4b/0x1b0 [ 4757.893238] ? do_nmi+0x280/0x380 [ 4757.893238] ? end_repeat_nmi+0x16/0x1f [ 4757.893239] ? on_each_cpu_cond_mask+0x46/0x150 [ 4757.893240] ? __x64_sys_fspick+0x210/0x210 [ 4757.893241] ? smp_call_function_many_cond+0x3db/0x540 [ 4757.893242] ? smp_call_function_many_cond+0x3db/0x540 [ 4757.893243] ? smp_call_function_many_cond+0x3db/0x540 [ 4757.893244] [ 4757.893245] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4757.893246] ? __x64_sys_fspick+0x210/0x210 [ 4757.893247] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4757.893248] on_each_cpu_cond_mask+0x46/0x150 [ 4757.893249] on_each_cpu_cond+0x1c/0x30 [ 4757.893250] invalidate_bh_lrus+0x27/0x30 [ 4757.893251] invalidate_bdev+0x5a/0x90 [ 4757.893252] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4757.893254] generic_shutdown_super+0xb1/0x1b0 [ 4757.893255] kill_block_super+0x29/0x80 [ 4757.893256] deactivate_locked_super+0x52/0xd0 [ 4757.893257] deactivate_super+0x83/0x90 [ 4757.893258] cleanup_mnt+0x5f/0xe0 [ 4757.893259] __cleanup_mnt+0x16/0x20 [ 4757.893260] task_work_run+0xc6/0x110 [ 4757.893261] exit_to_usermode_loop+0x1dd/0x1f0 [ 4757.893262] do_syscall_64+0x3d6/0x440 [ 4757.893263] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4757.893264] RIP: 0033:0x7fbb019648fb [ 4757.893266] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4757.893267] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4757.893270] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4757.893271] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4757.893273] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4757.893274] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4757.893276] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4771.807025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2413s! [ 4771.809231] Showing busy workqueues and worker pools: [ 4771.810551] workqueue events: flags=0x0 [ 4771.811638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.813902] in-flight: 56293:free_work [libcfs] [ 4771.815130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4771.816592] in-flight: 22277:free_work [libcfs] [ 4771.821584] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4771.826552] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.830846] pending: nfs4_renew_state [nfsv4] [ 4771.832059] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4771.835635] in-flight: 47596:free_work [libcfs] [ 4771.839702] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4771.842677] workqueue events_unbound: flags=0x2 [ 4771.844463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4771.847768] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4771.850599] workqueue events_power_efficient: flags=0x80 [ 4771.852548] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4771.854638] pending: neigh_periodic_work, do_cache_clean [ 4771.856633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.858670] pending: check_lifetime [ 4771.859759] workqueue mm_percpu_wq: flags=0x8 [ 4771.860970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.862944] pending: vmstat_update [ 4771.863988] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.865728] pending: vmstat_update [ 4771.866725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.868529] pending: vmstat_update [ 4771.869543] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4771.870845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4771.872625] pending: cgroup_pidlist_destroy_work_fn [ 4771.874076] workqueue writeback: flags=0x4a [ 4771.875242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4771.876809] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4771.878775] workqueue xprtiod: flags=0xa [ 4771.879930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4771.881598] pending: xprt_autoclose [ 4771.882631] workqueue ipv6_addrconf: flags=0x40008 [ 4771.884016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4771.885723] pending: addrconf_verify_work [ 4771.886964] workqueue knbd0-recv: flags=0x1a [ 4771.888089] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4771.889794] in-flight: 655:recv_work [ 4771.890944] workqueue ptlrpc_pinger: flags=0x2 [ 4771.892124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4771.893692] pending: ptlrpc_pinger_main [ptlrpc] [ 4771.895415] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2414s workers=5 idle: 53684 27131 54850 46584 [ 4771.897757] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2414s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4771.900670] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2414s workers=6 idle: 53660 37674 46417 53664 46418 [ 4771.903255] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4761s workers=2 idle: 657 [ 4772.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4772.062622] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4772.087671] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4772.095248] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4772.099790] Workqueue: events free_work [libcfs] [ 4772.102046] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4772.104819] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4772.119119] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4772.123045] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4772.126318] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4772.129511] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4772.131675] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4772.133633] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4772.137939] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4772.142036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4772.144879] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4772.147723] Call Trace: [ 4772.148962] [ 4772.149928] ? show_regs.cold.9+0x22/0x2f [ 4772.151857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4772.154245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4772.157081] ? watchdog+0x40/0x40 [ 4772.159938] ? hrtimer_interrupt+0x128/0x2c0 [ 4772.162716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4772.166703] ? apic_timer_interrupt+0xf/0x20 [ 4772.170585] [ 4772.171985] ? on_each_cpu+0x51/0x120 [ 4772.175002] ? smp_call_function_many_cond+0x3db/0x540 [ 4772.177195] ? smp_call_function_many_cond+0x3a8/0x540 [ 4772.180218] ? load_new_mm_cr3+0x120/0x120 [ 4772.183827] ? load_new_mm_cr3+0x120/0x120 [ 4772.186547] on_each_cpu+0x51/0x120 [ 4772.188844] flush_tlb_kernel_range+0x93/0xb0 [ 4772.191922] free_unmap_vmap_area+0x63/0x70 [ 4772.194802] remove_vm_area+0xe7/0x110 [ 4772.199064] __vunmap+0x9c/0x400 [ 4772.200318] vfree+0x4d/0x100 [ 4772.201327] free_work+0x34/0x50 [libcfs] [ 4772.202986] process_one_work+0x2c8/0x700 [ 4772.205759] worker_thread+0x296/0x6e0 [ 4772.207274] ? rescuer_thread+0x570/0x570 [ 4772.208956] kthread+0x1d1/0x200 [ 4772.210859] ? set_kthread_struct+0x70/0x70 [ 4772.212627] ret_from_fork+0x1f/0x30 [ 4784.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4784.060384] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4784.077134] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4784.080809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4784.084100] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4784.086059] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4784.092178] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4784.095084] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4784.096790] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4784.099535] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4784.101732] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4784.104201] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4784.106581] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4784.109190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4784.111097] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4784.113323] Call Trace: [ 4784.114199] [ 4784.114831] ? show_regs.cold.9+0x22/0x2f [ 4784.116321] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4784.117764] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4784.119369] ? watchdog+0x40/0x40 [ 4784.120290] ? hrtimer_interrupt+0x128/0x2c0 [ 4784.121765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4784.123444] ? apic_timer_interrupt+0xf/0x20 [ 4784.124705] [ 4784.125571] ? on_each_cpu_cond_mask+0x46/0x150 [ 4784.127160] ? __x64_sys_fspick+0x210/0x210 [ 4784.128629] ? smp_call_function_many_cond+0x3db/0x540 [ 4784.130469] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4784.131766] ? __x64_sys_fspick+0x210/0x210 [ 4784.133405] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4784.135041] on_each_cpu_cond_mask+0x46/0x150 [ 4784.136483] on_each_cpu_cond+0x1c/0x30 [ 4784.137720] invalidate_bh_lrus+0x27/0x30 [ 4784.139186] invalidate_bdev+0x5a/0x90 [ 4784.140375] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4784.141658] generic_shutdown_super+0xb1/0x1b0 [ 4784.143315] kill_block_super+0x29/0x80 [ 4784.144753] deactivate_locked_super+0x52/0xd0 [ 4784.146446] deactivate_super+0x83/0x90 [ 4784.147642] cleanup_mnt+0x5f/0xe0 [ 4784.148796] __cleanup_mnt+0x16/0x20 [ 4784.150169] task_work_run+0xc6/0x110 [ 4784.151484] exit_to_usermode_loop+0x1dd/0x1f0 [ 4784.152821] do_syscall_64+0x3d6/0x440 [ 4784.154195] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4784.156118] RIP: 0033:0x7fbb019648fb [ 4784.157358] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4784.163555] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4784.165992] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4784.168188] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4784.170672] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4784.173184] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4784.175624] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4796.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4796.058191] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.090127] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.098242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4796.104252] Workqueue: events free_work [libcfs] [ 4796.107378] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4796.110729] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4796.123285] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4796.130590] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4796.137243] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4796.140425] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4796.145085] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4796.150082] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4796.155079] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4796.160296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.164466] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4796.170274] Call Trace: [ 4796.171884] [ 4796.173086] ? show_regs.cold.9+0x22/0x2f [ 4796.176288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.179373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.182622] ? watchdog+0x40/0x40 [ 4796.183936] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.186435] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.189933] ? apic_timer_interrupt+0xf/0x20 [ 4796.193099] [ 4796.194438] ? on_each_cpu+0x51/0x120 [ 4796.197238] ? smp_call_function_many_cond+0x3db/0x540 [ 4796.200413] ? smp_call_function_many_cond+0x3a8/0x540 [ 4796.203739] ? load_new_mm_cr3+0x120/0x120 [ 4796.206162] ? load_new_mm_cr3+0x120/0x120 [ 4796.207746] on_each_cpu+0x51/0x120 [ 4796.215210] flush_tlb_kernel_range+0x93/0xb0 [ 4796.218604] free_unmap_vmap_area+0x63/0x70 [ 4796.222765] remove_vm_area+0xe7/0x110 [ 4796.225272] __vunmap+0x9c/0x400 [ 4796.228224] vfree+0x4d/0x100 [ 4796.230239] free_work+0x34/0x50 [libcfs] [ 4796.232907] process_one_work+0x2c8/0x700 [ 4796.236164] worker_thread+0x296/0x6e0 [ 4796.241511] ? rescuer_thread+0x570/0x570 [ 4796.245322] kthread+0x1d1/0x200 [ 4796.247474] ? set_kthread_struct+0x70/0x70 [ 4796.250936] ret_from_fork+0x1f/0x30 [ 4800.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4800.061739] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4800.082291] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4800.086936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4800.090802] Workqueue: events free_work [libcfs] [ 4800.092840] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4800.095985] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4800.107198] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4800.109879] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4800.112601] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4800.114864] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4800.117098] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4800.119175] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4800.121243] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4800.123730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4800.125364] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4800.127460] Call Trace: [ 4800.128207] [ 4800.128814] ? show_regs.cold.9+0x22/0x2f [ 4800.130126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4800.131884] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4800.133978] ? watchdog+0x40/0x40 [ 4800.135472] ? hrtimer_interrupt+0x128/0x2c0 [ 4800.138514] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4800.139939] ? apic_timer_interrupt+0xf/0x20 [ 4800.141219] [ 4800.141933] ? on_each_cpu+0x51/0x120 [ 4800.142990] ? smp_call_function_many_cond+0x3db/0x540 [ 4800.145233] ? smp_call_function_many_cond+0x3a8/0x540 [ 4800.146876] ? load_new_mm_cr3+0x120/0x120 [ 4800.148657] ? load_new_mm_cr3+0x120/0x120 [ 4800.150015] on_each_cpu+0x51/0x120 [ 4800.151773] flush_tlb_kernel_range+0x93/0xb0 [ 4800.155284] free_unmap_vmap_area+0x63/0x70 [ 4800.158922] remove_vm_area+0xe7/0x110 [ 4800.163154] __vunmap+0x9c/0x400 [ 4800.165997] vfree+0x4d/0x100 [ 4800.168684] free_work+0x34/0x50 [libcfs] [ 4800.172409] process_one_work+0x2c8/0x700 [ 4800.176078] worker_thread+0x296/0x6e0 [ 4800.177483] ? rescuer_thread+0x570/0x570 [ 4800.181575] kthread+0x1d1/0x200 [ 4800.183517] ? set_kthread_struct+0x70/0x70 [ 4800.185770] ret_from_fork+0x1f/0x30 [ 4802.016091] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2443s! [ 4802.023084] Showing busy workqueues and worker pools: [ 4802.026811] workqueue events: flags=0x0 [ 4802.028835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.032955] in-flight: 56293:free_work [libcfs] [ 4802.035441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4802.039990] in-flight: 22277:free_work [libcfs] [ 4802.044168] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4802.050051] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.054573] pending: nfs4_renew_state [nfsv4] [ 4802.057553] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4802.060447] in-flight: 47596:free_work [libcfs] [ 4802.063767] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4802.073150] workqueue events_unbound: flags=0x2 [ 4802.077664] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4802.083713] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4802.089219] workqueue events_power_efficient: flags=0x80 [ 4802.092996] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4802.099995] pending: neigh_periodic_work, do_cache_clean [ 4802.102515] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.107031] pending: check_lifetime [ 4802.108646] workqueue mm_percpu_wq: flags=0x8 [ 4802.114179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.120297] pending: vmstat_update [ 4802.121814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.123658] pending: vmstat_update [ 4802.124651] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.126347] pending: vmstat_update [ 4802.128448] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4802.132858] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4802.136682] pending: cgroup_pidlist_destroy_work_fn [ 4802.142051] workqueue writeback: flags=0x4a [ 4802.143666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4802.153201] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4802.159399] workqueue xprtiod: flags=0xa [ 4802.161934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4802.166667] pending: xprt_autoclose [ 4802.170444] workqueue ipv6_addrconf: flags=0x40008 [ 4802.173789] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4802.181827] pending: addrconf_verify_work [ 4802.184605] workqueue knbd0-recv: flags=0x1a [ 4802.192604] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4802.198029] in-flight: 655:recv_work [ 4802.201480] workqueue ptlrpc_pinger: flags=0x2 [ 4802.203922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4802.208975] pending: ptlrpc_pinger_main [ptlrpc] [ 4802.211557] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2444s workers=5 idle: 53684 27131 54850 46584 [ 4802.216588] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2444s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4802.222150] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2444s workers=6 idle: 53660 37674 46417 53664 46418 [ 4802.227011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4792s workers=2 idle: 657 [ 4812.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4812.060688] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4812.082731] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4812.086256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4812.089146] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4812.091124] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4812.097517] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4812.099953] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4812.102467] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4812.104854] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4812.108460] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4812.110781] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4812.113154] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4812.115710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4812.117564] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4812.119819] Call Trace: [ 4812.120678] [ 4812.121380] ? show_regs.cold.9+0x22/0x2f [ 4812.122717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4812.124336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4812.125737] ? watchdog+0x40/0x40 [ 4812.126830] ? hrtimer_interrupt+0x128/0x2c0 [ 4812.128146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4812.129795] ? apic_timer_interrupt+0xf/0x20 [ 4812.131209] [ 4812.132012] ? on_each_cpu_cond_mask+0x46/0x150 [ 4812.133712] ? __x64_sys_fspick+0x210/0x210 [ 4812.136025] ? smp_call_function_many_cond+0x3d9/0x540 [ 4812.137749] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4812.139173] ? __x64_sys_fspick+0x210/0x210 [ 4812.140659] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4812.142046] on_each_cpu_cond_mask+0x46/0x150 [ 4812.143493] on_each_cpu_cond+0x1c/0x30 [ 4812.144734] invalidate_bh_lrus+0x27/0x30 [ 4812.146031] invalidate_bdev+0x5a/0x90 [ 4812.147215] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4812.149069] generic_shutdown_super+0xb1/0x1b0 [ 4812.150588] kill_block_super+0x29/0x80 [ 4812.151845] deactivate_locked_super+0x52/0xd0 [ 4812.153206] deactivate_super+0x83/0x90 [ 4812.154676] cleanup_mnt+0x5f/0xe0 [ 4812.156234] __cleanup_mnt+0x16/0x20 [ 4812.157372] task_work_run+0xc6/0x110 [ 4812.158757] exit_to_usermode_loop+0x1dd/0x1f0 [ 4812.160364] do_syscall_64+0x3d6/0x440 [ 4812.161634] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4812.163243] RIP: 0033:0x7fbb019648fb [ 4812.165492] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4812.173785] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4812.177214] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4812.180080] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4812.183015] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4812.184882] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4812.186524] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4828.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4828.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4828.061001] Modules linked in: [ 4828.065605] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) [ 4828.067248] zfs(O) spl(O) [ 4828.072867] ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4828.072951] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4828.072958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4828.072964] Workqueue: events free_work [libcfs] [ 4828.074341] lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4828.074406] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4828.074412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4828.074418] Workqueue: events free_work [libcfs] [ 4828.074455] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4828.074463] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4828.074468] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4828.074472] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4828.074474] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4828.074476] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4828.094816] [ 4828.103461] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4828.103470] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4828.103473] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4828.103477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4828.103479] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4828.103487] Call Trace: [ 4828.103493] [ 4828.103498] ? show_regs.cold.9+0x22/0x2f [ 4828.103510] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4828.103516] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4828.103521] ? watchdog+0x40/0x40 [ 4828.103526] ? hrtimer_interrupt+0x128/0x2c0 [ 4828.103530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4828.103538] ? apic_timer_interrupt+0xf/0x20 [ 4828.103543] [ 4828.103545] ? on_each_cpu+0x51/0x120 [ 4828.103550] ? smp_call_function_many_cond+0x3db/0x540 [ 4828.103555] ? smp_call_function_many_cond+0x3a8/0x540 [ 4828.103558] ? load_new_mm_cr3+0x120/0x120 [ 4828.103563] ? load_new_mm_cr3+0x120/0x120 [ 4828.103567] on_each_cpu+0x51/0x120 [ 4828.103571] flush_tlb_kernel_range+0x93/0xb0 [ 4828.103577] free_unmap_vmap_area+0x63/0x70 [ 4828.103581] remove_vm_area+0xe7/0x110 [ 4828.103585] __vunmap+0x9c/0x400 [ 4828.103588] vfree+0x4d/0x100 [ 4828.103594] free_work+0x34/0x50 [libcfs] [ 4828.103631] process_one_work+0x2c8/0x700 [ 4828.103638] worker_thread+0x296/0x6e0 [ 4828.103643] ? rescuer_thread+0x570/0x570 [ 4828.103647] kthread+0x1d1/0x200 [ 4828.103652] ? set_kthread_struct+0x70/0x70 [ 4828.103656] ret_from_fork+0x1f/0x30 [ 4828.225288] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4828.227577] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4828.235230] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4828.239062] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4828.244487] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4828.248060] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4828.250505] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4828.253828] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4828.258079] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4828.265882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4828.269692] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4828.276719] Call Trace: [ 4828.282792] [ 4828.285046] ? show_regs.cold.9+0x22/0x2f [ 4828.288653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4828.293433] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4828.299462] ? watchdog+0x40/0x40 [ 4828.300692] ? hrtimer_interrupt+0x128/0x2c0 [ 4828.308238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4828.309850] ? apic_timer_interrupt+0xf/0x20 [ 4828.312224] [ 4828.313009] ? on_each_cpu+0x51/0x120 [ 4828.316468] ? smp_call_function_many_cond+0x3db/0x540 [ 4828.321716] ? smp_call_function_many_cond+0x3a8/0x540 [ 4828.327329] ? load_new_mm_cr3+0x120/0x120 [ 4828.330800] ? load_new_mm_cr3+0x120/0x120 [ 4828.332105] on_each_cpu+0x51/0x120 [ 4828.333648] flush_tlb_kernel_range+0x93/0xb0 [ 4828.341715] free_unmap_vmap_area+0x63/0x70 [ 4828.347430] remove_vm_area+0xe7/0x110 [ 4828.352444] __vunmap+0x9c/0x400 [ 4828.357308] vfree+0x4d/0x100 [ 4828.358578] free_work+0x34/0x50 [libcfs] [ 4828.362430] process_one_work+0x2c8/0x700 [ 4828.366563] worker_thread+0x296/0x6e0 [ 4828.370613] ? rescuer_thread+0x570/0x570 [ 4828.373789] kthread+0x1d1/0x200 [ 4828.375471] ? set_kthread_struct+0x70/0x70 [ 4828.377438] ret_from_fork+0x1f/0x30 [ 4832.735029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2474s! [ 4832.739595] Showing busy workqueues and worker pools: [ 4832.743406] workqueue events: flags=0x0 [ 4832.745469] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.751447] in-flight: 56293:free_work [libcfs] [ 4832.754122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4832.758183] in-flight: 22277:free_work [libcfs] [ 4832.760978] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4832.767097] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.770776] pending: nfs4_renew_state [nfsv4] [ 4832.775015] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4832.778423] in-flight: 47596:free_work [libcfs] [ 4832.781203] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4832.786217] workqueue events_unbound: flags=0x2 [ 4832.788566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4832.792113] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4832.797279] workqueue events_power_efficient: flags=0x80 [ 4832.800109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4832.803532] pending: neigh_periodic_work, do_cache_clean [ 4832.806277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.809581] pending: check_lifetime [ 4832.812028] workqueue mm_percpu_wq: flags=0x8 [ 4832.814280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.817952] pending: vmstat_update [ 4832.820397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.824512] pending: vmstat_update [ 4832.826652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.831272] pending: vmstat_update [ 4832.833502] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4832.835856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4832.840711] pending: cgroup_pidlist_destroy_work_fn [ 4832.843800] workqueue writeback: flags=0x4a [ 4832.845424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4832.849080] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4832.852982] workqueue xprtiod: flags=0xa [ 4832.854982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4832.858603] pending: xprt_autoclose [ 4832.861053] workqueue ipv6_addrconf: flags=0x40008 [ 4832.865174] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4832.870052] pending: addrconf_verify_work [ 4832.872454] workqueue knbd0-recv: flags=0x1a [ 4832.875798] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4832.881658] in-flight: 655:recv_work [ 4832.885081] workqueue ptlrpc_pinger: flags=0x2 [ 4832.887812] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4832.892364] pending: ptlrpc_pinger_main [ptlrpc] [ 4832.897524] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2475s workers=5 idle: 53684 27131 54850 46584 [ 4832.902989] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2475s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4832.912026] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2475s workers=6 idle: 53660 37674 46417 53664 46418 [ 4832.918110] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4822s workers=2 idle: 657 [ 4840.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4840.060053] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4840.084600] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4840.088309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4840.091197] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4840.093513] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4840.100332] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4840.102871] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4840.105388] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4840.107838] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4840.112107] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4840.113793] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4840.119370] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4840.126036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4840.129999] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4840.136012] Call Trace: [ 4840.137348] [ 4840.139151] ? show_regs.cold.9+0x22/0x2f [ 4840.141055] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4840.142183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4840.143292] ? watchdog+0x40/0x40 [ 4840.146303] ? hrtimer_interrupt+0x128/0x2c0 [ 4840.151192] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4840.152684] ? apic_timer_interrupt+0xf/0x20 [ 4840.154222] [ 4840.155125] ? on_each_cpu_cond_mask+0x46/0x150 [ 4840.157129] ? __x64_sys_fspick+0x210/0x210 [ 4840.158837] ? smp_call_function_many_cond+0x3db/0x540 [ 4840.161184] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4840.162784] ? __x64_sys_fspick+0x210/0x210 [ 4840.164205] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4840.167292] on_each_cpu_cond_mask+0x46/0x150 [ 4840.173709] on_each_cpu_cond+0x1c/0x30 [ 4840.174961] invalidate_bh_lrus+0x27/0x30 [ 4840.176169] invalidate_bdev+0x5a/0x90 [ 4840.180543] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4840.181848] generic_shutdown_super+0xb1/0x1b0 [ 4840.188578] kill_block_super+0x29/0x80 [ 4840.189933] deactivate_locked_super+0x52/0xd0 [ 4840.192359] deactivate_super+0x83/0x90 [ 4840.196066] cleanup_mnt+0x5f/0xe0 [ 4840.199938] __cleanup_mnt+0x16/0x20 [ 4840.201146] task_work_run+0xc6/0x110 [ 4840.203704] exit_to_usermode_loop+0x1dd/0x1f0 [ 4840.204789] do_syscall_64+0x3d6/0x440 [ 4840.205809] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4840.210481] RIP: 0033:0x7fbb019648fb [ 4840.213197] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4840.227111] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4840.229532] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4840.231529] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4840.235277] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4840.237635] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4840.240109] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4856.059176] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4856.063898] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4856.097459] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4856.104814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4856.110207] Workqueue: events free_work [libcfs] [ 4856.115586] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4856.119635] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4856.134259] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4856.139766] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4856.143931] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4856.148255] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4856.156572] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4856.161134] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4856.165434] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4856.172966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4856.176169] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4856.180111] Call Trace: [ 4856.181987] [ 4856.183546] ? show_regs.cold.9+0x22/0x2f [ 4856.186482] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4856.190104] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4856.192954] ? watchdog+0x40/0x40 [ 4856.195486] ? hrtimer_interrupt+0x128/0x2c0 [ 4856.198256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4856.202060] ? apic_timer_interrupt+0xf/0x20 [ 4856.205007] [ 4856.206282] ? on_each_cpu+0x51/0x120 [ 4856.209148] ? smp_call_function_many_cond+0x3db/0x540 [ 4856.212294] ? smp_call_function_many_cond+0x3a8/0x540 [ 4856.216151] ? load_new_mm_cr3+0x120/0x120 [ 4856.220149] ? load_new_mm_cr3+0x120/0x120 [ 4856.222568] on_each_cpu+0x51/0x120 [ 4856.225143] flush_tlb_kernel_range+0x93/0xb0 [ 4856.227994] free_unmap_vmap_area+0x63/0x70 [ 4856.229830] remove_vm_area+0xe7/0x110 [ 4856.232249] __vunmap+0x9c/0x400 [ 4856.234272] vfree+0x4d/0x100 [ 4856.235217] free_work+0x34/0x50 [libcfs] [ 4856.238142] process_one_work+0x2c8/0x700 [ 4856.241052] worker_thread+0x296/0x6e0 [ 4856.242657] ? rescuer_thread+0x570/0x570 [ 4856.245136] kthread+0x1d1/0x200 [ 4856.247142] ? set_kthread_struct+0x70/0x70 [ 4856.250106] ret_from_fork+0x1f/0x30 [ 4860.056523] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4860.060149] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4860.088027] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4860.091987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4860.095109] Workqueue: events free_work [libcfs] [ 4860.096927] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4860.099787] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4860.115387] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4860.121190] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4860.126377] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4860.128970] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4860.131543] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4860.136512] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4860.139071] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4860.147372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4860.151644] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4860.153610] Call Trace: [ 4860.154597] [ 4860.155110] ? show_regs.cold.9+0x22/0x2f [ 4860.157293] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4860.158408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4860.161564] ? watchdog+0x40/0x40 [ 4860.162725] ? hrtimer_interrupt+0x128/0x2c0 [ 4860.164129] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4860.167496] ? apic_timer_interrupt+0xf/0x20 [ 4860.168558] [ 4860.169043] ? on_each_cpu+0x51/0x120 [ 4860.169931] ? smp_call_function_many_cond+0x3db/0x540 [ 4860.172461] ? smp_call_function_many_cond+0x3a8/0x540 [ 4860.174091] ? load_new_mm_cr3+0x120/0x120 [ 4860.177609] ? load_new_mm_cr3+0x120/0x120 [ 4860.181126] on_each_cpu+0x51/0x120 [ 4860.182434] flush_tlb_kernel_range+0x93/0xb0 [ 4860.183793] free_unmap_vmap_area+0x63/0x70 [ 4860.184960] remove_vm_area+0xe7/0x110 [ 4860.186002] __vunmap+0x9c/0x400 [ 4860.190024] vfree+0x4d/0x100 [ 4860.191113] free_work+0x34/0x50 [libcfs] [ 4860.192005] process_one_work+0x2c8/0x700 [ 4860.192917] worker_thread+0x296/0x6e0 [ 4860.196728] ? rescuer_thread+0x570/0x570 [ 4860.198255] kthread+0x1d1/0x200 [ 4860.199032] ? set_kthread_struct+0x70/0x70 [ 4860.199997] ret_from_fork+0x1f/0x30 [ 4862.943028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2504s! [ 4862.947583] Showing busy workqueues and worker pools: [ 4862.949642] workqueue events: flags=0x0 [ 4862.951045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4862.953776] in-flight: 56293:free_work [libcfs] [ 4862.955814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4862.958686] in-flight: 22277:free_work [libcfs] [ 4862.960738] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4862.964324] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4862.967069] pending: nfs4_renew_state [nfsv4] [ 4862.969287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4862.972090] in-flight: 47596:free_work [libcfs] [ 4862.974737] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4862.980133] workqueue events_unbound: flags=0x2 [ 4862.981802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4862.983947] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4862.987252] workqueue events_power_efficient: flags=0x80 [ 4862.989453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4862.992159] pending: neigh_periodic_work, do_cache_clean [ 4862.994653] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4862.997492] pending: check_lifetime [ 4862.999031] workqueue mm_percpu_wq: flags=0x8 [ 4863.000886] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4863.005820] pending: vmstat_update [ 4863.008039] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4863.012909] pending: vmstat_update [ 4863.015022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4863.018924] pending: vmstat_update [ 4863.021823] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4863.026016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4863.030845] pending: cgroup_pidlist_destroy_work_fn [ 4863.034250] workqueue writeback: flags=0x4a [ 4863.036871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4863.040334] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4863.044013] workqueue xprtiod: flags=0xa [ 4863.046201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4863.047842] pending: xprt_autoclose [ 4863.048900] workqueue ipv6_addrconf: flags=0x40008 [ 4863.051830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4863.055329] pending: addrconf_verify_work [ 4863.057626] workqueue knbd0-recv: flags=0x1a [ 4863.059673] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4863.066224] in-flight: 655:recv_work [ 4863.070588] workqueue ptlrpc_pinger: flags=0x2 [ 4863.076173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4863.081457] pending: ptlrpc_pinger_main [ptlrpc] [ 4863.083021] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2505s workers=5 idle: 53684 27131 54850 46584 [ 4863.087548] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2505s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4863.100084] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2505s workers=6 idle: 53660 37674 46417 53664 46418 [ 4863.109335] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4852s workers=2 idle: 657 [ 4868.058399] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4868.065727] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4868.111226] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4868.119791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4868.129426] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4868.133547] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4868.149031] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4868.153767] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4868.160756] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4868.164672] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4868.167902] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4868.173756] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4868.176657] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4868.185608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4868.187999] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4868.199543] Call Trace: [ 4868.202780] [ 4868.204128] ? show_regs.cold.9+0x22/0x2f [ 4868.206063] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4868.211811] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4868.213230] ? watchdog+0x40/0x40 [ 4868.220194] ? hrtimer_interrupt+0x128/0x2c0 [ 4868.221922] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4868.225359] ? apic_timer_interrupt+0xf/0x20 [ 4868.227946] [ 4868.229302] ? on_each_cpu_cond_mask+0x46/0x150 [ 4868.231484] ? __x64_sys_fspick+0x210/0x210 [ 4868.238409] ? smp_call_function_many_cond+0x3db/0x540 [ 4868.241162] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4868.243559] ? __x64_sys_fspick+0x210/0x210 [ 4868.247561] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4868.250421] on_each_cpu_cond_mask+0x46/0x150 [ 4868.254783] on_each_cpu_cond+0x1c/0x30 [ 4868.257931] invalidate_bh_lrus+0x27/0x30 [ 4868.260967] invalidate_bdev+0x5a/0x90 [ 4868.262492] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4868.267944] generic_shutdown_super+0xb1/0x1b0 [ 4868.272151] kill_block_super+0x29/0x80 [ 4868.273578] deactivate_locked_super+0x52/0xd0 [ 4868.275204] deactivate_super+0x83/0x90 [ 4868.284591] cleanup_mnt+0x5f/0xe0 [ 4868.291661] __cleanup_mnt+0x16/0x20 [ 4868.294824] task_work_run+0xc6/0x110 [ 4868.300528] exit_to_usermode_loop+0x1dd/0x1f0 [ 4868.303126] do_syscall_64+0x3d6/0x440 [ 4868.305967] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4868.310020] RIP: 0033:0x7fbb019648fb [ 4868.314681] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4868.328582] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4868.332035] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4868.346363] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4868.348671] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4868.356969] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4868.360786] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4884.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4884.061941] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4884.078583] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4884.082537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4884.085400] Workqueue: events free_work [libcfs] [ 4884.087191] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4884.089219] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4884.095214] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4884.098429] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4884.101923] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4884.104991] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4884.107920] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4884.110578] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4884.113306] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4884.117011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4884.119336] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4884.122186] Call Trace: [ 4884.123169] [ 4884.124080] ? show_regs.cold.9+0x22/0x2f [ 4884.125647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4884.127117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4884.128387] ? watchdog+0x40/0x40 [ 4884.129513] ? hrtimer_interrupt+0x128/0x2c0 [ 4884.131129] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4884.132593] ? apic_timer_interrupt+0xf/0x20 [ 4884.134040] [ 4884.134600] ? on_each_cpu+0x51/0x120 [ 4884.135695] ? smp_call_function_many_cond+0x3db/0x540 [ 4884.137081] ? smp_call_function_many_cond+0x3a8/0x540 [ 4884.138176] ? load_new_mm_cr3+0x120/0x120 [ 4884.139091] ? load_new_mm_cr3+0x120/0x120 [ 4884.139995] on_each_cpu+0x51/0x120 [ 4884.140628] flush_tlb_kernel_range+0x93/0xb0 [ 4884.142101] free_unmap_vmap_area+0x63/0x70 [ 4884.143565] remove_vm_area+0xe7/0x110 [ 4884.145496] __vunmap+0x9c/0x400 [ 4884.147518] vfree+0x4d/0x100 [ 4884.149600] free_work+0x34/0x50 [libcfs] [ 4884.152566] process_one_work+0x2c8/0x700 [ 4884.155507] worker_thread+0x296/0x6e0 [ 4884.158135] ? rescuer_thread+0x570/0x570 [ 4884.160612] kthread+0x1d1/0x200 [ 4884.163170] ? set_kthread_struct+0x70/0x70 [ 4884.165581] ret_from_fork+0x1f/0x30 [ 4888.055701] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 4888.059331] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4888.089707] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4888.096208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4888.100539] Workqueue: events free_work [libcfs] [ 4888.105145] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4888.109017] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4888.120858] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4888.129601] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4888.133593] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4888.140285] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4888.146140] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4888.150700] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4888.155990] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4888.158046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4888.164180] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4888.168957] Call Trace: [ 4888.170590] [ 4888.171813] ? show_regs.cold.9+0x22/0x2f [ 4888.173798] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4888.176491] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4888.179362] ? watchdog+0x40/0x40 [ 4888.180291] ? hrtimer_interrupt+0x128/0x2c0 [ 4888.181527] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4888.183529] ? apic_timer_interrupt+0xf/0x20 [ 4888.184602] [ 4888.185367] ? on_each_cpu+0x51/0x120 [ 4888.186330] ? smp_call_function_many_cond+0x3db/0x540 [ 4888.193109] ? smp_call_function_many_cond+0x3a8/0x540 [ 4888.194548] ? load_new_mm_cr3+0x120/0x120 [ 4888.196052] ? load_new_mm_cr3+0x120/0x120 [ 4888.197314] on_each_cpu+0x51/0x120 [ 4888.198292] flush_tlb_kernel_range+0x93/0xb0 [ 4888.199519] free_unmap_vmap_area+0x63/0x70 [ 4888.200887] remove_vm_area+0xe7/0x110 [ 4888.201981] __vunmap+0x9c/0x400 [ 4888.202876] vfree+0x4d/0x100 [ 4888.203611] free_work+0x34/0x50 [libcfs] [ 4888.204983] process_one_work+0x2c8/0x700 [ 4888.206081] worker_thread+0x296/0x6e0 [ 4888.208613] ? rescuer_thread+0x570/0x570 [ 4888.210267] kthread+0x1d1/0x200 [ 4888.211483] ? set_kthread_struct+0x70/0x70 [ 4888.214181] ret_from_fork+0x1f/0x30 [ 4893.151332] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2534s! [ 4893.158505] Showing busy workqueues and worker pools: [ 4893.162939] workqueue events: flags=0x0 [ 4893.164215] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4893.167381] in-flight: 56293:free_work [libcfs] [ 4893.169383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4893.172173] in-flight: 22277:free_work [libcfs] [ 4893.173599] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4893.179122] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4893.188616] pending: nfs4_renew_state [nfsv4] [ 4893.190133] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4893.196848] in-flight: 47596:free_work [libcfs] [ 4893.199171] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4893.204157] workqueue events_unbound: flags=0x2 [ 4893.207209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4893.209867] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4893.216460] workqueue events_power_efficient: flags=0x80 [ 4893.218374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4893.222013] pending: neigh_periodic_work, do_cache_clean [ 4893.226866] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4893.229698] pending: check_lifetime [ 4893.231189] workqueue mm_percpu_wq: flags=0x8 [ 4893.233061] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4893.235094] pending: vmstat_update [ 4893.238161] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4893.240625] pending: vmstat_update [ 4893.244197] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4893.248400] pending: vmstat_update [ 4893.252092] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4893.254372] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4893.257251] pending: cgroup_pidlist_destroy_work_fn [ 4893.260472] workqueue writeback: flags=0x4a [ 4893.263208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4893.272886] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4893.279330] workqueue xprtiod: flags=0xa [ 4893.286234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4893.289195] pending: xprt_autoclose [ 4893.292055] workqueue ipv6_addrconf: flags=0x40008 [ 4893.294719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4893.299026] pending: addrconf_verify_work [ 4893.301898] workqueue knbd0-recv: flags=0x1a [ 4893.304674] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4893.308416] in-flight: 655:recv_work [ 4893.310086] workqueue ptlrpc_pinger: flags=0x2 [ 4893.311697] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4893.315635] pending: ptlrpc_pinger_main [ptlrpc] [ 4893.317965] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2535s workers=5 idle: 53684 27131 54850 46584 [ 4893.323055] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2535s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4893.328480] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2535s workers=6 idle: 53660 37674 46417 53664 46418 [ 4893.334120] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4883s workers=2 idle: 657 [ 4896.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4896.060986] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4896.088490] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4896.093745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4896.098957] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4896.102139] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4896.110347] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4896.114993] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4896.118276] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4896.122293] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4896.126315] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4896.133062] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4896.136501] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4896.140421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4896.143726] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4896.147327] Call Trace: [ 4896.148570] [ 4896.149630] ? show_regs.cold.9+0x22/0x2f [ 4896.151617] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4896.153422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4896.156180] ? watchdog+0x40/0x40 [ 4896.157732] ? hrtimer_interrupt+0x128/0x2c0 [ 4896.159538] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4896.161484] ? apic_timer_interrupt+0xf/0x20 [ 4896.163793] [ 4896.164409] ? on_each_cpu_cond_mask+0x46/0x150 [ 4896.165652] ? __x64_sys_fspick+0x210/0x210 [ 4896.166701] ? smp_call_function_many_cond+0x3db/0x540 [ 4896.168210] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4896.169951] ? __x64_sys_fspick+0x210/0x210 [ 4896.171355] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4896.172889] on_each_cpu_cond_mask+0x46/0x150 [ 4896.174751] on_each_cpu_cond+0x1c/0x30 [ 4896.176313] invalidate_bh_lrus+0x27/0x30 [ 4896.178268] invalidate_bdev+0x5a/0x90 [ 4896.179859] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4896.181799] generic_shutdown_super+0xb1/0x1b0 [ 4896.183432] kill_block_super+0x29/0x80 [ 4896.185058] deactivate_locked_super+0x52/0xd0 [ 4896.186646] deactivate_super+0x83/0x90 [ 4896.187893] cleanup_mnt+0x5f/0xe0 [ 4896.189044] __cleanup_mnt+0x16/0x20 [ 4896.190538] task_work_run+0xc6/0x110 [ 4896.191991] exit_to_usermode_loop+0x1dd/0x1f0 [ 4896.193331] do_syscall_64+0x3d6/0x440 [ 4896.194804] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4896.196812] RIP: 0033:0x7fbb019648fb [ 4896.198285] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4896.205567] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4896.208311] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4896.210971] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4896.213375] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4896.216277] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4896.219027] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4912.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 4912.063352] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4912.092759] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4912.099437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4912.107868] Workqueue: events free_work [libcfs] [ 4912.110364] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4912.112022] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4912.124696] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4912.127522] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4912.130713] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4912.134518] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4912.138071] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4912.141510] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4912.144878] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4912.150381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4912.152925] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4912.158517] Call Trace: [ 4912.160403] [ 4912.162890] ? show_regs.cold.9+0x22/0x2f [ 4912.165306] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4912.168782] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4912.171889] ? watchdog+0x40/0x40 [ 4912.173794] ? hrtimer_interrupt+0x128/0x2c0 [ 4912.177035] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4912.180965] ? apic_timer_interrupt+0xf/0x20 [ 4912.184487] [ 4912.185897] ? on_each_cpu+0x51/0x120 [ 4912.187911] ? smp_call_function_many_cond+0x3db/0x540 [ 4912.191078] ? smp_call_function_many_cond+0x3a8/0x540 [ 4912.194491] ? load_new_mm_cr3+0x120/0x120 [ 4912.198818] ? load_new_mm_cr3+0x120/0x120 [ 4912.200819] on_each_cpu+0x51/0x120 [ 4912.202185] flush_tlb_kernel_range+0x93/0xb0 [ 4912.206427] free_unmap_vmap_area+0x63/0x70 [ 4912.210182] remove_vm_area+0xe7/0x110 [ 4912.213674] __vunmap+0x9c/0x400 [ 4912.215497] vfree+0x4d/0x100 [ 4912.217002] free_work+0x34/0x50 [libcfs] [ 4912.220707] process_one_work+0x2c8/0x700 [ 4912.223021] worker_thread+0x296/0x6e0 [ 4912.224961] ? rescuer_thread+0x570/0x570 [ 4912.229517] kthread+0x1d1/0x200 [ 4912.231472] ? set_kthread_struct+0x70/0x70 [ 4912.233603] ret_from_fork+0x1f/0x30 [ 4920.057469] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4920.060304] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4920.092283] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4920.097490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4920.102061] Workqueue: events free_work [libcfs] [ 4920.103918] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4920.106380] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4920.115704] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4920.121317] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4920.123605] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4920.128746] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4920.132972] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4920.137821] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4920.143138] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4920.148910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4920.156354] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4920.162295] Call Trace: [ 4920.163252] [ 4920.163991] ? show_regs.cold.9+0x22/0x2f [ 4920.168915] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4920.171738] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4920.175633] ? watchdog+0x40/0x40 [ 4920.177274] ? hrtimer_interrupt+0x128/0x2c0 [ 4920.182206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4920.184519] ? apic_timer_interrupt+0xf/0x20 [ 4920.188520] [ 4920.191596] ? on_each_cpu+0x51/0x120 [ 4920.195176] ? smp_call_function_many_cond+0x3db/0x540 [ 4920.198553] ? smp_call_function_many_cond+0x3a8/0x540 [ 4920.201740] ? load_new_mm_cr3+0x120/0x120 [ 4920.203916] ? load_new_mm_cr3+0x120/0x120 [ 4920.210768] on_each_cpu+0x51/0x120 [ 4920.212368] flush_tlb_kernel_range+0x93/0xb0 [ 4920.214435] free_unmap_vmap_area+0x63/0x70 [ 4920.218677] remove_vm_area+0xe7/0x110 [ 4920.220060] __vunmap+0x9c/0x400 [ 4920.221550] vfree+0x4d/0x100 [ 4920.222878] free_work+0x34/0x50 [libcfs] [ 4920.224767] process_one_work+0x2c8/0x700 [ 4920.227392] worker_thread+0x296/0x6e0 [ 4920.228863] ? rescuer_thread+0x570/0x570 [ 4920.230970] kthread+0x1d1/0x200 [ 4920.232162] ? set_kthread_struct+0x70/0x70 [ 4920.235352] ret_from_fork+0x1f/0x30 [ 4923.359025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2565s! [ 4923.361465] Showing busy workqueues and worker pools: [ 4923.363145] workqueue events: flags=0x0 [ 4923.364470] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4923.366554] in-flight: 56293:free_work [libcfs] [ 4923.368388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4923.370448] in-flight: 22277:free_work [libcfs] [ 4923.372238] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4923.375262] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4923.377267] pending: nfs4_renew_state [nfsv4] [ 4923.378747] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4923.381249] in-flight: 47596:free_work [libcfs] [ 4923.382669] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4923.385491] workqueue events_unbound: flags=0x2 [ 4923.386665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4923.388599] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4923.391268] workqueue events_power_efficient: flags=0x80 [ 4923.392491] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4923.394589] pending: neigh_periodic_work, do_cache_clean [ 4923.396266] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4923.398337] pending: check_lifetime [ 4923.399470] workqueue mm_percpu_wq: flags=0x8 [ 4923.401190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4923.403175] pending: vmstat_update [ 4923.404315] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4923.406347] pending: vmstat_update [ 4923.407494] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4923.409485] pending: vmstat_update [ 4923.410626] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4923.412361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4923.414321] pending: cgroup_pidlist_destroy_work_fn [ 4923.415763] workqueue writeback: flags=0x4a [ 4923.417224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4923.419199] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4923.421514] workqueue xprtiod: flags=0xa [ 4923.422739] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4923.424776] pending: xprt_autoclose [ 4923.426314] workqueue ipv6_addrconf: flags=0x40008 [ 4923.427729] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4923.429758] pending: addrconf_verify_work [ 4923.431323] workqueue knbd0-recv: flags=0x1a [ 4923.432622] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4923.434646] in-flight: 655:recv_work [ 4923.436025] workqueue ptlrpc_pinger: flags=0x2 [ 4923.437327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4923.439210] pending: ptlrpc_pinger_main [ptlrpc] [ 4923.441116] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2565s workers=5 idle: 53684 27131 54850 46584 [ 4923.443643] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2565s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4923.447061] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2565s workers=6 idle: 53660 37674 46417 53664 46418 [ 4923.450772] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4913s workers=2 idle: 657 [ 4924.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4924.060847] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4924.086775] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4924.092789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4924.098302] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4924.100432] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4924.116580] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4924.120530] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4924.125100] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4924.128444] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4924.131556] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4924.134651] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4924.137611] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4924.141240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4924.143700] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4924.146820] Call Trace: [ 4924.148109] [ 4924.148832] ? show_regs.cold.9+0x22/0x2f [ 4924.150458] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4924.151986] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4924.153640] ? watchdog+0x40/0x40 [ 4924.154753] ? hrtimer_interrupt+0x128/0x2c0 [ 4924.156628] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4924.158318] ? apic_timer_interrupt+0xf/0x20 [ 4924.160145] [ 4924.161170] ? on_each_cpu_cond_mask+0x46/0x150 [ 4924.163242] ? __x64_sys_fspick+0x210/0x210 [ 4924.165246] ? smp_call_function_many_cond+0x3db/0x540 [ 4924.167578] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4924.169458] ? __x64_sys_fspick+0x210/0x210 [ 4924.171356] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4924.173432] on_each_cpu_cond_mask+0x46/0x150 [ 4924.175207] on_each_cpu_cond+0x1c/0x30 [ 4924.177055] invalidate_bh_lrus+0x27/0x30 [ 4924.178645] invalidate_bdev+0x5a/0x90 [ 4924.180381] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4924.182766] generic_shutdown_super+0xb1/0x1b0 [ 4924.185243] kill_block_super+0x29/0x80 [ 4924.187246] deactivate_locked_super+0x52/0xd0 [ 4924.189414] deactivate_super+0x83/0x90 [ 4924.191533] cleanup_mnt+0x5f/0xe0 [ 4924.193347] __cleanup_mnt+0x16/0x20 [ 4924.195415] task_work_run+0xc6/0x110 [ 4924.197446] exit_to_usermode_loop+0x1dd/0x1f0 [ 4924.200106] do_syscall_64+0x3d6/0x440 [ 4924.202366] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4924.205365] RIP: 0033:0x7fbb019648fb [ 4924.206673] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4924.218335] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4924.222574] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4924.226632] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4924.230143] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4924.234296] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4924.238234] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4937.807113] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4937.815251] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 4937.826056] (detected by 2, t=2580077 jiffies, g=626849, q=1398) [ 4937.828301] Sending NMI from CPU 2 to CPUs 3: [ 4937.838648] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 4937.838657] rcu: rcu_sched kthread starved for 2579719 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4937.848084] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4937.852701] rcu: RCU grace-period kthread stack dump: [ 4937.856190] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4937.861235] Call Trace: [ 4937.862636] __schedule+0x351/0xcb0 [ 4937.864620] schedule+0xc0/0x180 [ 4937.866354] schedule_timeout+0xb4/0x190 [ 4937.868459] ? do_raw_spin_unlock+0x75/0x190 [ 4937.870669] ? __next_timer_interrupt+0x160/0x160 [ 4937.873804] ? prepare_to_swait_event+0xf3/0x140 [ 4937.877401] rcu_gp_kthread+0x8dd/0x15b0 [ 4937.880426] ? force_qs_rnp+0x2c0/0x2c0 [ 4937.883986] kthread+0x1d1/0x200 [ 4937.887227] ? set_kthread_struct+0x70/0x70 [ 4937.890976] ret_from_fork+0x1f/0x30 [ 4937.894174] rcu: Stack dump where RCU GP kthread last ran: [ 4937.899478] NMI backtrace for cpu 2 [ 4937.901206] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4937.908481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4937.913536] Workqueue: events free_work [libcfs] [ 4937.915701] Call Trace: [ 4937.917013] [ 4937.918102] ? dump_stack+0xbb/0x10e [ 4937.919918] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4937.925342] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4937.927087] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4937.930248] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4937.932325] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4937.937232] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4937.940225] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 4937.943222] ? update_process_times+0x6d/0xc0 [ 4937.945431] ? tick_sched_handle.isra.18+0x27/0x90 [ 4937.948101] ? tick_sched_timer+0x53/0xb0 [ 4937.951325] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4937.955410] ? tick_sched_handle.isra.18+0x90/0x90 [ 4937.958357] ? hrtimer_interrupt+0x128/0x2c0 [ 4937.960265] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4937.963076] ? apic_timer_interrupt+0xf/0x20 [ 4937.965109] [ 4937.965908] ? on_each_cpu+0x51/0x120 [ 4937.967372] ? smp_call_function_many_cond+0x3db/0x540 [ 4937.970456] ? smp_call_function_many_cond+0x3a8/0x540 [ 4937.973541] ? load_new_mm_cr3+0x120/0x120 [ 4937.975305] ? load_new_mm_cr3+0x120/0x120 [ 4937.981152] ? on_each_cpu+0x51/0x120 [ 4937.983330] ? flush_tlb_kernel_range+0x93/0xb0 [ 4937.987124] ? free_unmap_vmap_area+0x63/0x70 [ 4937.988336] ? remove_vm_area+0xe7/0x110 [ 4937.993511] ? __vunmap+0x9c/0x400 [ 4937.996720] ? vfree+0x4d/0x100 [ 4937.999054] ? free_work+0x34/0x50 [libcfs] [ 4938.003721] ? process_one_work+0x2c8/0x700 [ 4938.007972] ? worker_thread+0x296/0x6e0 [ 4938.012329] ? rescuer_thread+0x570/0x570 [ 4938.016031] ? kthread+0x1d1/0x200 [ 4938.017970] ? set_kthread_struct+0x70/0x70 [ 4938.021315] ? ret_from_fork+0x1f/0x30 [ 4948.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4948.057709] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4948.077885] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4948.082770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4948.086674] Workqueue: events free_work [libcfs] [ 4948.088592] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4948.090979] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4948.098932] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4948.101679] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4948.103898] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4948.106974] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4948.109997] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4948.113215] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4948.116260] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4948.119468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4948.121253] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4948.124626] Call Trace: [ 4948.125712] [ 4948.126312] ? show_regs.cold.9+0x22/0x2f [ 4948.128249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4948.130298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4948.132313] ? watchdog+0x40/0x40 [ 4948.133854] ? hrtimer_interrupt+0x128/0x2c0 [ 4948.135681] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4948.137890] ? apic_timer_interrupt+0xf/0x20 [ 4948.139700] [ 4948.140679] ? on_each_cpu+0x51/0x120 [ 4948.142111] ? smp_call_function_many_cond+0x3db/0x540 [ 4948.143832] ? smp_call_function_many_cond+0x3a8/0x540 [ 4948.145969] ? load_new_mm_cr3+0x120/0x120 [ 4948.147828] ? load_new_mm_cr3+0x120/0x120 [ 4948.149671] on_each_cpu+0x51/0x120 [ 4948.150988] flush_tlb_kernel_range+0x93/0xb0 [ 4948.152966] free_unmap_vmap_area+0x63/0x70 [ 4948.154885] remove_vm_area+0xe7/0x110 [ 4948.156611] __vunmap+0x9c/0x400 [ 4948.157975] vfree+0x4d/0x100 [ 4948.159210] free_work+0x34/0x50 [libcfs] [ 4948.161024] process_one_work+0x2c8/0x700 [ 4948.162880] worker_thread+0x296/0x6e0 [ 4948.164517] ? rescuer_thread+0x570/0x570 [ 4948.166195] kthread+0x1d1/0x200 [ 4948.167868] ? set_kthread_struct+0x70/0x70 [ 4948.169730] ret_from_fork+0x1f/0x30 [ 4952.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 4952.061036] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4952.081562] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4952.086180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4952.090941] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4952.092968] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4952.102204] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4952.105723] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4952.109438] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4952.112483] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4952.115025] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4952.119079] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4952.122645] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4952.126572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4952.129143] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4952.132315] Call Trace: [ 4952.133472] [ 4952.134363] ? show_regs.cold.9+0x22/0x2f [ 4952.140830] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4952.144214] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4952.147225] ? watchdog+0x40/0x40 [ 4952.149566] ? hrtimer_interrupt+0x128/0x2c0 [ 4952.152546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4952.155532] ? apic_timer_interrupt+0xf/0x20 [ 4952.157739] [ 4952.158893] ? on_each_cpu_cond_mask+0x46/0x150 [ 4952.161475] ? __x64_sys_fspick+0x210/0x210 [ 4952.164453] ? smp_call_function_many_cond+0x3db/0x540 [ 4952.167547] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4952.169408] ? __x64_sys_fspick+0x210/0x210 [ 4952.171502] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4952.173523] on_each_cpu_cond_mask+0x46/0x150 [ 4952.176132] on_each_cpu_cond+0x1c/0x30 [ 4952.178523] invalidate_bh_lrus+0x27/0x30 [ 4952.181588] invalidate_bdev+0x5a/0x90 [ 4952.184524] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4952.187738] generic_shutdown_super+0xb1/0x1b0 [ 4952.191033] kill_block_super+0x29/0x80 [ 4952.193679] deactivate_locked_super+0x52/0xd0 [ 4952.196133] deactivate_super+0x83/0x90 [ 4952.197453] cleanup_mnt+0x5f/0xe0 [ 4952.199739] __cleanup_mnt+0x16/0x20 [ 4952.201692] task_work_run+0xc6/0x110 [ 4952.203213] exit_to_usermode_loop+0x1dd/0x1f0 [ 4952.205425] do_syscall_64+0x3d6/0x440 [ 4952.207537] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4952.209771] RIP: 0033:0x7fbb019648fb [ 4952.211449] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4952.219760] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4952.222522] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4952.225646] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4952.229059] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4952.232593] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4952.235731] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4953.570296] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2595s! [ 4953.578133] Showing busy workqueues and worker pools: [ 4953.580176] workqueue events: flags=0x0 [ 4953.584916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4953.590677] in-flight: 56293:free_work [libcfs] [ 4953.592613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4953.597526] in-flight: 22277:free_work [libcfs] [ 4953.604153] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4953.612347] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4953.615535] pending: nfs4_renew_state [nfsv4] [ 4953.618811] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4953.621701] in-flight: 47596:free_work [libcfs] [ 4953.624883] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4953.634295] workqueue events_unbound: flags=0x2 [ 4953.636739] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4953.640478] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4953.648064] workqueue events_power_efficient: flags=0x80 [ 4953.650711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4953.654809] pending: neigh_periodic_work, do_cache_clean [ 4953.657740] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4953.661310] pending: check_lifetime [ 4953.664464] workqueue mm_percpu_wq: flags=0x8 [ 4953.667966] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4953.671512] pending: vmstat_update [ 4953.673380] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4953.676304] pending: vmstat_update [ 4953.678191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4953.682387] pending: vmstat_update [ 4953.684515] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4953.686891] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4953.690152] pending: cgroup_pidlist_destroy_work_fn [ 4953.692885] workqueue writeback: flags=0x4a [ 4953.695965] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4953.699216] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4953.703814] workqueue xprtiod: flags=0xa [ 4953.706805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4953.709911] pending: xprt_autoclose [ 4953.712319] workqueue ipv6_addrconf: flags=0x40008 [ 4953.715848] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4953.719284] pending: addrconf_verify_work [ 4953.722142] workqueue knbd0-recv: flags=0x1a [ 4953.725264] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4953.728811] in-flight: 655:recv_work [ 4953.731372] workqueue ptlrpc_pinger: flags=0x2 [ 4953.733268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4953.738796] pending: ptlrpc_pinger_main [ptlrpc] [ 4953.743149] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2595s workers=5 idle: 53684 27131 54850 46584 [ 4953.748953] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2596s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4953.755196] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2596s workers=6 idle: 53660 37674 46417 53664 46418 [ 4953.762061] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4943s workers=2 idle: 657 [ 4964.063140] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4964.072355] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4964.095624] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4964.105147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4964.112483] Workqueue: events free_work [libcfs] [ 4964.115083] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4964.117514] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4964.125552] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4964.128851] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4964.132883] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4964.135915] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4964.139322] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4964.142952] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4964.149133] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4964.156174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4964.158445] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4964.161296] Call Trace: [ 4964.162244] [ 4964.162820] ? show_regs.cold.9+0x22/0x2f [ 4964.164657] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4964.166373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4964.169251] ? watchdog+0x40/0x40 [ 4964.171004] ? hrtimer_interrupt+0x128/0x2c0 [ 4964.173651] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4964.176469] ? apic_timer_interrupt+0xf/0x20 [ 4964.178704] [ 4964.179592] ? on_each_cpu+0x51/0x120 [ 4964.181119] ? smp_call_function_many_cond+0x3db/0x540 [ 4964.183740] ? smp_call_function_many_cond+0x3a8/0x540 [ 4964.187622] ? load_new_mm_cr3+0x120/0x120 [ 4964.189448] ? load_new_mm_cr3+0x120/0x120 [ 4964.191449] on_each_cpu+0x51/0x120 [ 4964.193116] flush_tlb_kernel_range+0x93/0xb0 [ 4964.194992] free_unmap_vmap_area+0x63/0x70 [ 4964.196598] remove_vm_area+0xe7/0x110 [ 4964.199616] __vunmap+0x9c/0x400 [ 4964.201350] vfree+0x4d/0x100 [ 4964.203583] free_work+0x34/0x50 [libcfs] [ 4964.206125] process_one_work+0x2c8/0x700 [ 4964.209260] worker_thread+0x296/0x6e0 [ 4964.212146] ? rescuer_thread+0x570/0x570 [ 4964.215855] kthread+0x1d1/0x200 [ 4964.218226] ? set_kthread_struct+0x70/0x70 [ 4964.220091] ret_from_fork+0x1f/0x30 [ 4980.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 4980.058068] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 4980.059296] Modules linked in: zfs(O) [ 4980.063490] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4980.063587] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4980.063595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4980.063599] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 4980.065983] spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4980.066070] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4980.066077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4980.066083] Workqueue: events free_work [libcfs] [ 4980.115619] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 4980.115633] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4980.115641] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 4980.115643] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 4980.115645] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 4980.115647] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 4980.115649] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 4980.115654] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 4980.115656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4980.115659] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 4980.115666] Call Trace: [ 4980.115672] [ 4980.115678] ? show_regs.cold.9+0x22/0x2f [ 4980.115690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4980.123573] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4980.123592] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4980.123596] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4980.123601] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 4980.123603] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 4980.123605] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 4980.123607] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4980.123609] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 4980.123612] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 4980.123615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4980.123617] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 4980.123626] Call Trace: [ 4980.123631] [ 4980.123716] ? show_regs.cold.9+0x22/0x2f [ 4980.133062] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4980.136346] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4980.169705] ? watchdog+0x40/0x40 [ 4980.169723] ? hrtimer_interrupt+0x128/0x2c0 [ 4980.169728] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4980.169736] ? apic_timer_interrupt+0xf/0x20 [ 4980.169741] [ 4980.169745] ? on_each_cpu_cond_mask+0x46/0x150 [ 4980.169749] ? __x64_sys_fspick+0x210/0x210 [ 4980.169757] ? smp_call_function_many_cond+0x3db/0x540 [ 4980.169761] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4980.169765] ? __x64_sys_fspick+0x210/0x210 [ 4980.169769] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 4980.169772] on_each_cpu_cond_mask+0x46/0x150 [ 4980.169777] on_each_cpu_cond+0x1c/0x30 [ 4980.169781] invalidate_bh_lrus+0x27/0x30 [ 4980.169784] invalidate_bdev+0x5a/0x90 [ 4980.169791] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 4980.169993] generic_shutdown_super+0xb1/0x1b0 [ 4980.170001] kill_block_super+0x29/0x80 [ 4980.170006] deactivate_locked_super+0x52/0xd0 [ 4980.170011] deactivate_super+0x83/0x90 [ 4980.170016] cleanup_mnt+0x5f/0xe0 [ 4980.170022] __cleanup_mnt+0x16/0x20 [ 4980.170026] task_work_run+0xc6/0x110 [ 4980.170034] exit_to_usermode_loop+0x1dd/0x1f0 [ 4980.170041] do_syscall_64+0x3d6/0x440 [ 4980.170046] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4980.170052] RIP: 0033:0x7fbb019648fb [ 4980.170059] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 4980.170062] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 4980.170068] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 4980.170071] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 4980.170073] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 4980.170075] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 4980.170077] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 4980.353712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4980.358059] ? watchdog+0x40/0x40 [ 4980.362161] ? hrtimer_interrupt+0x128/0x2c0 [ 4980.365093] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4980.368249] ? apic_timer_interrupt+0xf/0x20 [ 4980.370921] [ 4980.372149] ? on_each_cpu+0x51/0x120 [ 4980.374012] ? smp_call_function_many_cond+0x3d9/0x540 [ 4980.376272] ? smp_call_function_many_cond+0x3a8/0x540 [ 4980.379522] ? load_new_mm_cr3+0x120/0x120 [ 4980.381754] ? load_new_mm_cr3+0x120/0x120 [ 4980.383932] on_each_cpu+0x51/0x120 [ 4980.387776] flush_tlb_kernel_range+0x93/0xb0 [ 4980.391077] free_unmap_vmap_area+0x63/0x70 [ 4980.394080] remove_vm_area+0xe7/0x110 [ 4980.396287] __vunmap+0x9c/0x400 [ 4980.397573] vfree+0x4d/0x100 [ 4980.399397] free_work+0x34/0x50 [libcfs] [ 4980.401834] process_one_work+0x2c8/0x700 [ 4980.403138] worker_thread+0x296/0x6e0 [ 4980.404635] ? rescuer_thread+0x570/0x570 [ 4980.407557] kthread+0x1d1/0x200 [ 4980.410187] ? set_kthread_struct+0x70/0x70 [ 4980.414964] ret_from_fork+0x1f/0x30 [ 4983.781025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2625s! [ 4983.786536] Showing busy workqueues and worker pools: [ 4983.794086] workqueue events: flags=0x0 [ 4983.803226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4983.812907] in-flight: 56293:free_work [libcfs] [ 4983.819874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4983.828010] in-flight: 22277:free_work [libcfs] [ 4983.832931] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 4983.850087] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4983.858536] pending: nfs4_renew_state [nfsv4] [ 4983.864287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4983.874869] in-flight: 47596:free_work [libcfs] [ 4983.878993] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4983.887546] workqueue events_unbound: flags=0x2 [ 4983.890551] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 4983.893727] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4983.898058] workqueue events_power_efficient: flags=0x80 [ 4983.900719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4983.903451] pending: neigh_periodic_work, do_cache_clean [ 4983.906037] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4983.910839] pending: check_lifetime [ 4983.913992] workqueue mm_percpu_wq: flags=0x8 [ 4983.916406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4983.920517] pending: vmstat_update [ 4983.922857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4983.927248] pending: vmstat_update [ 4983.929399] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4983.933016] pending: vmstat_update [ 4983.934688] workqueue cgroup_pidlist_destroy: flags=0x0 [ 4983.937906] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4983.941999] pending: cgroup_pidlist_destroy_work_fn [ 4983.946125] workqueue writeback: flags=0x4a [ 4983.951017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4983.955976] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 4983.962531] workqueue xprtiod: flags=0xa [ 4983.966445] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4983.970952] pending: xprt_autoclose [ 4983.973929] workqueue ipv6_addrconf: flags=0x40008 [ 4983.976727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4983.984680] pending: addrconf_verify_work [ 4983.989013] workqueue knbd0-recv: flags=0x1a [ 4983.993794] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4983.996274] in-flight: 655:recv_work [ 4984.001844] workqueue ptlrpc_pinger: flags=0x2 [ 4984.006388] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4984.014300] pending: ptlrpc_pinger_main [ptlrpc] [ 4984.020854] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2626s workers=5 idle: 53684 27131 54850 46584 [ 4984.027818] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2626s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 4984.043638] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2626s workers=6 idle: 53660 37674 46417 53664 46418 [ 4984.052124] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4973s workers=2 idle: 657 [ 4992.062625] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 4992.065328] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4992.097953] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4992.101272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4992.104305] Workqueue: events free_work [libcfs] [ 4992.105807] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 4992.107502] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 4992.112974] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4992.115216] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 4992.118616] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 4992.120514] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 4992.128127] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 4992.130326] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 4992.132992] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 4992.138024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4992.141726] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 4992.144054] Call Trace: [ 4992.145524] [ 4992.146132] ? show_regs.cold.9+0x22/0x2f [ 4992.148719] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4992.151063] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4992.152779] ? watchdog+0x40/0x40 [ 4992.155204] ? hrtimer_interrupt+0x128/0x2c0 [ 4992.156558] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4992.159104] ? apic_timer_interrupt+0xf/0x20 [ 4992.161873] [ 4992.163255] ? on_each_cpu+0x51/0x120 [ 4992.164986] ? smp_call_function_many_cond+0x3d9/0x540 [ 4992.167920] ? smp_call_function_many_cond+0x3a8/0x540 [ 4992.170784] ? load_new_mm_cr3+0x120/0x120 [ 4992.174235] ? load_new_mm_cr3+0x120/0x120 [ 4992.176834] on_each_cpu+0x51/0x120 [ 4992.178697] flush_tlb_kernel_range+0x93/0xb0 [ 4992.180307] free_unmap_vmap_area+0x63/0x70 [ 4992.182840] remove_vm_area+0xe7/0x110 [ 4992.184871] __vunmap+0x9c/0x400 [ 4992.186804] vfree+0x4d/0x100 [ 4992.188021] free_work+0x34/0x50 [libcfs] [ 4992.190791] process_one_work+0x2c8/0x700 [ 4992.192372] worker_thread+0x296/0x6e0 [ 4992.194276] ? rescuer_thread+0x570/0x570 [ 4992.196716] kthread+0x1d1/0x200 [ 4992.198672] ? set_kthread_struct+0x70/0x70 [ 4992.201096] ret_from_fork+0x1f/0x30 [ 5008.058834] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5008.063908] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5008.103816] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5008.108814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5008.113019] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5008.115789] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5008.126353] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5008.130531] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5008.134689] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5008.138860] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5008.143723] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5008.149386] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5008.156129] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5008.164856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5008.167460] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5008.170814] Call Trace: [ 5008.173355] [ 5008.174159] ? show_regs.cold.9+0x22/0x2f [ 5008.176504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5008.178438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5008.179829] ? watchdog+0x40/0x40 [ 5008.181697] ? hrtimer_interrupt+0x128/0x2c0 [ 5008.187363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5008.190085] ? apic_timer_interrupt+0xf/0x20 [ 5008.192028] [ 5008.193157] ? on_each_cpu_cond_mask+0x46/0x150 [ 5008.194814] ? __x64_sys_fspick+0x210/0x210 [ 5008.196761] ? smp_call_function_many_cond+0x3db/0x540 [ 5008.199133] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5008.201151] ? __x64_sys_fspick+0x210/0x210 [ 5008.203124] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5008.205152] on_each_cpu_cond_mask+0x46/0x150 [ 5008.207168] on_each_cpu_cond+0x1c/0x30 [ 5008.208806] invalidate_bh_lrus+0x27/0x30 [ 5008.211292] invalidate_bdev+0x5a/0x90 [ 5008.212887] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5008.215427] generic_shutdown_super+0xb1/0x1b0 [ 5008.217397] kill_block_super+0x29/0x80 [ 5008.219159] deactivate_locked_super+0x52/0xd0 [ 5008.221164] deactivate_super+0x83/0x90 [ 5008.222910] cleanup_mnt+0x5f/0xe0 [ 5008.224720] __cleanup_mnt+0x16/0x20 [ 5008.226479] task_work_run+0xc6/0x110 [ 5008.228300] exit_to_usermode_loop+0x1dd/0x1f0 [ 5008.230454] do_syscall_64+0x3d6/0x440 [ 5008.232201] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5008.234653] RIP: 0033:0x7fbb019648fb [ 5008.237327] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5008.247509] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5008.255598] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5008.260882] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5008.267816] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5008.273676] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5008.280682] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5012.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5012.057481] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5012.079349] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5012.083682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5012.091236] Workqueue: events free_work [libcfs] [ 5012.093429] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5012.096344] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5012.108671] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5012.111633] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5012.116506] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5012.123656] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5012.129437] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5012.134310] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5012.140107] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5012.147791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5012.154495] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5012.160100] Call Trace: [ 5012.161111] [ 5012.168154] ? show_regs.cold.9+0x22/0x2f [ 5012.171394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5012.177440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5012.180825] ? watchdog+0x40/0x40 [ 5012.183229] ? hrtimer_interrupt+0x128/0x2c0 [ 5012.189244] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5012.194794] ? apic_timer_interrupt+0xf/0x20 [ 5012.198443] [ 5012.199134] ? on_each_cpu+0x51/0x120 [ 5012.201553] ? smp_call_function_many_cond+0x3db/0x540 [ 5012.204360] ? smp_call_function_many_cond+0x3a8/0x540 [ 5012.212619] ? load_new_mm_cr3+0x120/0x120 [ 5012.214980] ? load_new_mm_cr3+0x120/0x120 [ 5012.217081] on_each_cpu+0x51/0x120 [ 5012.219756] flush_tlb_kernel_range+0x93/0xb0 [ 5012.223718] free_unmap_vmap_area+0x63/0x70 [ 5012.227544] remove_vm_area+0xe7/0x110 [ 5012.232397] __vunmap+0x9c/0x400 [ 5012.236321] vfree+0x4d/0x100 [ 5012.239539] free_work+0x34/0x50 [libcfs] [ 5012.241076] process_one_work+0x2c8/0x700 [ 5012.242641] worker_thread+0x296/0x6e0 [ 5012.245584] ? rescuer_thread+0x570/0x570 [ 5012.247653] kthread+0x1d1/0x200 [ 5012.249713] ? set_kthread_struct+0x70/0x70 [ 5012.253423] ret_from_fork+0x1f/0x30 [ 5014.496803] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2656s! [ 5014.506399] Showing busy workqueues and worker pools: [ 5014.513271] workqueue events: flags=0x0 [ 5014.519876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5014.528821] in-flight: 56293:free_work [libcfs] [ 5014.531886] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5014.542394] in-flight: 22277:free_work [libcfs] [ 5014.545794] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5014.559062] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5014.561826] pending: nfs4_renew_state [nfsv4] [ 5014.566580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5014.570991] in-flight: 47596:free_work [libcfs] [ 5014.574905] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5014.580204] workqueue events_unbound: flags=0x2 [ 5014.582825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5014.586477] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5014.590538] workqueue events_power_efficient: flags=0x80 [ 5014.593463] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5014.597081] pending: neigh_periodic_work, do_cache_clean [ 5014.600069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5014.603720] pending: check_lifetime [ 5014.608666] workqueue mm_percpu_wq: flags=0x8 [ 5014.612689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5014.620171] pending: vmstat_update [ 5014.622982] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5014.625750] pending: vmstat_update [ 5014.627606] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5014.630780] pending: vmstat_update [ 5014.632392] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5014.634795] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5014.637912] pending: cgroup_pidlist_destroy_work_fn [ 5014.640165] workqueue writeback: flags=0x4a [ 5014.641756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5014.644536] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5014.647347] workqueue xprtiod: flags=0xa [ 5014.648794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5014.651500] pending: xprt_autoclose [ 5014.653282] workqueue ipv6_addrconf: flags=0x40008 [ 5014.655249] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5014.657996] pending: addrconf_verify_work [ 5014.659881] workqueue knbd0-recv: flags=0x1a [ 5014.661681] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5014.666338] in-flight: 655:recv_work [ 5014.669266] workqueue ptlrpc_pinger: flags=0x2 [ 5014.671482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5014.674488] pending: ptlrpc_pinger_main [ptlrpc] [ 5014.677568] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2656s workers=5 idle: 53684 27131 54850 46584 [ 5014.682091] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2656s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5014.687474] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2656s workers=6 idle: 53660 37674 46417 53664 46418 [ 5014.692319] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5004s workers=2 idle: 657 [ 5020.059014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 5020.064024] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5020.101912] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5020.109342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5020.114672] Workqueue: events free_work [libcfs] [ 5020.117429] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5020.120736] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5020.131986] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5020.140396] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5020.146456] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5020.154453] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5020.160935] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5020.165564] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5020.170474] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5020.175457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5020.178746] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5020.184072] Call Trace: [ 5020.185923] [ 5020.186982] ? show_regs.cold.9+0x22/0x2f [ 5020.189334] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5020.192394] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5020.195257] ? watchdog+0x40/0x40 [ 5020.197889] ? hrtimer_interrupt+0x128/0x2c0 [ 5020.202607] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5020.206470] ? apic_timer_interrupt+0xf/0x20 [ 5020.209415] [ 5020.210977] ? on_each_cpu+0x51/0x120 [ 5020.213341] ? smp_call_function_many_cond+0x3d9/0x540 [ 5020.216457] ? smp_call_function_many_cond+0x3a8/0x540 [ 5020.221307] ? load_new_mm_cr3+0x120/0x120 [ 5020.224264] ? load_new_mm_cr3+0x120/0x120 [ 5020.227365] on_each_cpu+0x51/0x120 [ 5020.229312] flush_tlb_kernel_range+0x93/0xb0 [ 5020.232645] free_unmap_vmap_area+0x63/0x70 [ 5020.236376] remove_vm_area+0xe7/0x110 [ 5020.239633] __vunmap+0x9c/0x400 [ 5020.242332] vfree+0x4d/0x100 [ 5020.244977] free_work+0x34/0x50 [libcfs] [ 5020.248568] process_one_work+0x2c8/0x700 [ 5020.251367] worker_thread+0x296/0x6e0 [ 5020.254749] ? rescuer_thread+0x570/0x570 [ 5020.259010] kthread+0x1d1/0x200 [ 5020.261592] ? set_kthread_struct+0x70/0x70 [ 5020.264626] ret_from_fork+0x1f/0x30 [ 5036.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5036.063638] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5036.115134] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5036.129292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5036.136732] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5036.140448] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5036.159610] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5036.171335] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5036.178671] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5036.189932] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5036.200332] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5036.209629] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5036.215693] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5036.225710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5036.234763] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5036.237153] Call Trace: [ 5036.242657] [ 5036.244569] ? show_regs.cold.9+0x22/0x2f [ 5036.246931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5036.248399] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5036.252035] ? watchdog+0x40/0x40 [ 5036.254930] ? hrtimer_interrupt+0x128/0x2c0 [ 5036.256771] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5036.258366] ? apic_timer_interrupt+0xf/0x20 [ 5036.262206] [ 5036.264143] ? on_each_cpu_cond_mask+0x46/0x150 [ 5036.267345] ? __x64_sys_fspick+0x210/0x210 [ 5036.270477] ? smp_call_function_many_cond+0x3db/0x540 [ 5036.275737] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5036.278754] ? __x64_sys_fspick+0x210/0x210 [ 5036.281375] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5036.285071] on_each_cpu_cond_mask+0x46/0x150 [ 5036.286210] on_each_cpu_cond+0x1c/0x30 [ 5036.287194] invalidate_bh_lrus+0x27/0x30 [ 5036.292481] invalidate_bdev+0x5a/0x90 [ 5036.296261] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5036.301415] generic_shutdown_super+0xb1/0x1b0 [ 5036.304334] kill_block_super+0x29/0x80 [ 5036.308304] deactivate_locked_super+0x52/0xd0 [ 5036.313430] deactivate_super+0x83/0x90 [ 5036.314573] cleanup_mnt+0x5f/0xe0 [ 5036.319116] __cleanup_mnt+0x16/0x20 [ 5036.322125] task_work_run+0xc6/0x110 [ 5036.323287] exit_to_usermode_loop+0x1dd/0x1f0 [ 5036.325153] do_syscall_64+0x3d6/0x440 [ 5036.328991] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5036.332934] RIP: 0033:0x7fbb019648fb [ 5036.337944] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5036.352024] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5036.356325] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5036.359579] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5036.365153] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5036.369575] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5036.374384] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5040.055355] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5040.060735] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5040.091199] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5040.099507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5040.109006] Workqueue: events free_work [libcfs] [ 5040.111032] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5040.113655] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5040.122156] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5040.125214] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5040.129870] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5040.133080] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5040.136282] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5040.139646] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5040.141803] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5040.144969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5040.147960] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5040.151020] Call Trace: [ 5040.152187] [ 5040.153138] ? show_regs.cold.9+0x22/0x2f [ 5040.155012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5040.157163] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5040.159808] ? watchdog+0x40/0x40 [ 5040.161212] ? hrtimer_interrupt+0x128/0x2c0 [ 5040.163005] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5040.165662] ? apic_timer_interrupt+0xf/0x20 [ 5040.168773] [ 5040.170135] ? on_each_cpu+0x51/0x120 [ 5040.174076] ? smp_call_function_many_cond+0x3db/0x540 [ 5040.179026] ? smp_call_function_many_cond+0x3a8/0x540 [ 5040.181620] ? load_new_mm_cr3+0x120/0x120 [ 5040.187248] ? load_new_mm_cr3+0x120/0x120 [ 5040.193966] on_each_cpu+0x51/0x120 [ 5040.196965] flush_tlb_kernel_range+0x93/0xb0 [ 5040.201132] free_unmap_vmap_area+0x63/0x70 [ 5040.203866] remove_vm_area+0xe7/0x110 [ 5040.205844] __vunmap+0x9c/0x400 [ 5040.207644] vfree+0x4d/0x100 [ 5040.208855] free_work+0x34/0x50 [libcfs] [ 5040.210973] process_one_work+0x2c8/0x700 [ 5040.212860] worker_thread+0x296/0x6e0 [ 5040.214864] ? rescuer_thread+0x570/0x570 [ 5040.216902] kthread+0x1d1/0x200 [ 5040.218764] ? set_kthread_struct+0x70/0x70 [ 5040.220883] ret_from_fork+0x1f/0x30 [ 5044.704688] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2686s! [ 5044.709310] Showing busy workqueues and worker pools: [ 5044.711541] workqueue events: flags=0x0 [ 5044.713186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5044.716519] in-flight: 56293:free_work [libcfs] [ 5044.718756] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5044.722857] in-flight: 22277:free_work [libcfs] [ 5044.726050] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5044.731432] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5044.734063] pending: nfs4_renew_state [nfsv4] [ 5044.736997] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5044.740384] in-flight: 47596:free_work [libcfs] [ 5044.743096] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5044.751604] workqueue events_unbound: flags=0x2 [ 5044.753957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5044.758559] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5044.762874] workqueue events_power_efficient: flags=0x80 [ 5044.765564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5044.769587] pending: neigh_periodic_work, do_cache_clean [ 5044.772766] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5044.775874] pending: check_lifetime [ 5044.778164] workqueue mm_percpu_wq: flags=0x8 [ 5044.779972] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5044.786087] pending: vmstat_update [ 5044.787968] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5044.793330] pending: vmstat_update [ 5044.795507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5044.799952] pending: vmstat_update [ 5044.801824] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5044.804550] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5044.807662] pending: cgroup_pidlist_destroy_work_fn [ 5044.809346] workqueue writeback: flags=0x4a [ 5044.810487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5044.815571] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5044.822561] workqueue xprtiod: flags=0xa [ 5044.825557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5044.829751] pending: xprt_autoclose [ 5044.834199] workqueue ipv6_addrconf: flags=0x40008 [ 5044.837560] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5044.840676] pending: addrconf_verify_work [ 5044.843169] workqueue knbd0-recv: flags=0x1a [ 5044.846513] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5044.849708] in-flight: 655:recv_work [ 5044.857736] workqueue ptlrpc_pinger: flags=0x2 [ 5044.866374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5044.870837] pending: ptlrpc_pinger_main [ptlrpc] [ 5044.874583] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2687s workers=5 idle: 53684 27131 54850 46584 [ 5044.886789] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2687s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5044.892528] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2687s workers=6 idle: 53660 37674 46417 53664 46418 [ 5044.904142] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5034s workers=2 idle: 657 [ 5048.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5048.063558] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5048.087896] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5048.093288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5048.096503] Workqueue: events free_work [libcfs] [ 5048.098754] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5048.101818] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5048.110906] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5048.115692] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5048.119132] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5048.122843] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5048.126510] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5048.129947] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5048.133474] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5048.137595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5048.139763] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5048.143148] Call Trace: [ 5048.144771] [ 5048.145941] ? show_regs.cold.9+0x22/0x2f [ 5048.148019] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5048.150445] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5048.152772] ? watchdog+0x40/0x40 [ 5048.154608] ? hrtimer_interrupt+0x128/0x2c0 [ 5048.157123] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5048.159151] ? apic_timer_interrupt+0xf/0x20 [ 5048.163648] [ 5048.166633] ? on_each_cpu+0x51/0x120 [ 5048.169007] ? smp_call_function_many_cond+0x3d9/0x540 [ 5048.172052] ? smp_call_function_many_cond+0x3a8/0x540 [ 5048.175000] ? load_new_mm_cr3+0x120/0x120 [ 5048.177907] ? load_new_mm_cr3+0x120/0x120 [ 5048.180601] on_each_cpu+0x51/0x120 [ 5048.182736] flush_tlb_kernel_range+0x93/0xb0 [ 5048.185776] free_unmap_vmap_area+0x63/0x70 [ 5048.187511] remove_vm_area+0xe7/0x110 [ 5048.189025] __vunmap+0x9c/0x400 [ 5048.190935] vfree+0x4d/0x100 [ 5048.192757] free_work+0x34/0x50 [libcfs] [ 5048.195066] process_one_work+0x2c8/0x700 [ 5048.197149] worker_thread+0x296/0x6e0 [ 5048.198989] ? rescuer_thread+0x570/0x570 [ 5048.200893] kthread+0x1d1/0x200 [ 5048.202838] ? set_kthread_struct+0x70/0x70 [ 5048.204926] ret_from_fork+0x1f/0x30 [ 5064.058008] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5064.061601] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5064.084542] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5064.089711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5064.095492] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5064.098457] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5064.109784] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5064.114732] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5064.118708] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5064.121916] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5064.125369] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5064.128965] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5064.132096] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5064.135802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5064.140206] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5064.143638] Call Trace: [ 5064.145063] [ 5064.146299] ? show_regs.cold.9+0x22/0x2f [ 5064.148589] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5064.152100] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5064.154660] ? watchdog+0x40/0x40 [ 5064.156366] ? hrtimer_interrupt+0x128/0x2c0 [ 5064.159547] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5064.163402] ? apic_timer_interrupt+0xf/0x20 [ 5064.165610] [ 5064.166579] ? on_each_cpu_cond_mask+0x46/0x150 [ 5064.169686] ? __x64_sys_fspick+0x210/0x210 [ 5064.172419] ? smp_call_function_many_cond+0x3db/0x540 [ 5064.174443] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5064.176274] ? __x64_sys_fspick+0x210/0x210 [ 5064.178274] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5064.180259] on_each_cpu_cond_mask+0x46/0x150 [ 5064.182367] on_each_cpu_cond+0x1c/0x30 [ 5064.183486] invalidate_bh_lrus+0x27/0x30 [ 5064.184886] invalidate_bdev+0x5a/0x90 [ 5064.186012] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5064.188180] generic_shutdown_super+0xb1/0x1b0 [ 5064.190121] kill_block_super+0x29/0x80 [ 5064.192154] deactivate_locked_super+0x52/0xd0 [ 5064.194231] deactivate_super+0x83/0x90 [ 5064.196603] cleanup_mnt+0x5f/0xe0 [ 5064.201122] __cleanup_mnt+0x16/0x20 [ 5064.202637] task_work_run+0xc6/0x110 [ 5064.205205] exit_to_usermode_loop+0x1dd/0x1f0 [ 5064.207362] do_syscall_64+0x3d6/0x440 [ 5064.209131] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5064.212496] RIP: 0033:0x7fbb019648fb [ 5064.214680] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5064.222559] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5064.225696] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5064.228565] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5064.231487] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5064.234715] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5064.237979] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5072.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5072.058254] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5072.085022] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5072.090490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5072.094372] Workqueue: events free_work [libcfs] [ 5072.096495] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5072.099362] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5072.107827] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5072.112610] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5072.115732] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5072.118937] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5072.122354] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5072.125546] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5072.128812] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5072.134992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5072.137567] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5072.140661] Call Trace: [ 5072.141775] [ 5072.143326] ? show_regs.cold.9+0x22/0x2f [ 5072.145672] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5072.147730] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5072.149716] ? watchdog+0x40/0x40 [ 5072.151347] ? hrtimer_interrupt+0x128/0x2c0 [ 5072.153439] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5072.155454] ? apic_timer_interrupt+0xf/0x20 [ 5072.157864] [ 5072.158798] ? on_each_cpu+0x51/0x120 [ 5072.160756] ? smp_call_function_many_cond+0x3d9/0x540 [ 5072.163489] ? smp_call_function_many_cond+0x3a8/0x540 [ 5072.165806] ? load_new_mm_cr3+0x120/0x120 [ 5072.167783] ? load_new_mm_cr3+0x120/0x120 [ 5072.169494] on_each_cpu+0x51/0x120 [ 5072.171470] flush_tlb_kernel_range+0x93/0xb0 [ 5072.173946] free_unmap_vmap_area+0x63/0x70 [ 5072.175784] remove_vm_area+0xe7/0x110 [ 5072.177372] __vunmap+0x9c/0x400 [ 5072.178847] vfree+0x4d/0x100 [ 5072.180149] free_work+0x34/0x50 [libcfs] [ 5072.181670] process_one_work+0x2c8/0x700 [ 5072.183461] worker_thread+0x296/0x6e0 [ 5072.186222] ? rescuer_thread+0x570/0x570 [ 5072.187811] kthread+0x1d1/0x200 [ 5072.189393] ? set_kthread_struct+0x70/0x70 [ 5072.191232] ret_from_fork+0x1f/0x30 [ 5074.912058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2716s! [ 5074.917296] Showing busy workqueues and worker pools: [ 5074.919389] workqueue events: flags=0x0 [ 5074.922972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5074.931136] in-flight: 56293:free_work [libcfs] [ 5074.937419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5074.940665] in-flight: 22277:free_work [libcfs] [ 5074.943701] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5074.951951] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5074.955332] pending: nfs4_renew_state [nfsv4] [ 5074.957926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5074.962230] in-flight: 47596:free_work [libcfs] [ 5074.964874] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5074.969272] workqueue events_unbound: flags=0x2 [ 5074.971533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5074.974510] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5074.978303] workqueue events_power_efficient: flags=0x80 [ 5074.982085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5074.987476] pending: neigh_periodic_work, do_cache_clean [ 5074.991100] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5074.996222] pending: check_lifetime [ 5074.998904] workqueue mm_percpu_wq: flags=0x8 [ 5075.002602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5075.008814] pending: vmstat_update [ 5075.010832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5075.014488] pending: vmstat_update [ 5075.016631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5075.019552] pending: vmstat_update [ 5075.022478] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5075.027027] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5075.033306] pending: cgroup_pidlist_destroy_work_fn [ 5075.036850] workqueue writeback: flags=0x4a [ 5075.039051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5075.042798] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5075.045321] workqueue xprtiod: flags=0xa [ 5075.048340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5075.053095] pending: xprt_autoclose [ 5075.055910] workqueue ipv6_addrconf: flags=0x40008 [ 5075.058797] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5075.062999] pending: addrconf_verify_work [ 5075.065016] workqueue knbd0-recv: flags=0x1a [ 5075.066515] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5075.069307] in-flight: 655:recv_work [ 5075.072027] workqueue ptlrpc_pinger: flags=0x2 [ 5075.074914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5075.079480] pending: ptlrpc_pinger_main [ptlrpc] [ 5075.083103] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2717s workers=5 idle: 53684 27131 54850 46584 [ 5075.088532] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2717s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5075.094552] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2717s workers=6 idle: 53660 37674 46417 53664 46418 [ 5075.102674] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5064s workers=2 idle: 657 [ 5076.059307] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5076.067077] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5076.121205] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5076.130736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5076.134770] Workqueue: events free_work [libcfs] [ 5076.137982] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5076.141206] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5076.151255] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5076.155537] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5076.159722] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5076.164288] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5076.168868] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5076.173068] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5076.177461] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5076.182384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5076.185882] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5076.189921] Call Trace: [ 5076.190863] [ 5076.191665] ? show_regs.cold.9+0x22/0x2f [ 5076.194107] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5076.196907] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5076.199288] ? watchdog+0x40/0x40 [ 5076.201162] ? hrtimer_interrupt+0x128/0x2c0 [ 5076.203264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5076.206149] ? apic_timer_interrupt+0xf/0x20 [ 5076.208117] [ 5076.209201] ? on_each_cpu+0x51/0x120 [ 5076.210977] ? smp_call_function_many_cond+0x3db/0x540 [ 5076.213532] ? smp_call_function_many_cond+0x3a8/0x540 [ 5076.215455] ? load_new_mm_cr3+0x120/0x120 [ 5076.217498] ? load_new_mm_cr3+0x120/0x120 [ 5076.219481] on_each_cpu+0x51/0x120 [ 5076.221157] flush_tlb_kernel_range+0x93/0xb0 [ 5076.223231] free_unmap_vmap_area+0x63/0x70 [ 5076.225806] remove_vm_area+0xe7/0x110 [ 5076.230533] __vunmap+0x9c/0x400 [ 5076.232665] vfree+0x4d/0x100 [ 5076.234057] free_work+0x34/0x50 [libcfs] [ 5076.237930] process_one_work+0x2c8/0x700 [ 5076.239927] worker_thread+0x296/0x6e0 [ 5076.241953] ? rescuer_thread+0x570/0x570 [ 5076.244354] kthread+0x1d1/0x200 [ 5076.246412] ? set_kthread_struct+0x70/0x70 [ 5076.249298] ret_from_fork+0x1f/0x30 [ 5092.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5092.063122] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5092.095210] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5092.099740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5092.103389] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5092.105741] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5092.113108] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5092.116036] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5092.118940] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5092.121839] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5092.124684] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5092.127568] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5092.130453] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5092.133521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5092.135996] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5092.138672] Call Trace: [ 5092.139614] [ 5092.140506] ? show_regs.cold.9+0x22/0x2f [ 5092.142033] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5092.143979] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5092.145859] ? watchdog+0x40/0x40 [ 5092.147132] ? hrtimer_interrupt+0x128/0x2c0 [ 5092.148482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5092.150112] ? apic_timer_interrupt+0xf/0x20 [ 5092.151929] [ 5092.152867] ? on_each_cpu_cond_mask+0x46/0x150 [ 5092.154889] ? __x64_sys_fspick+0x210/0x210 [ 5092.156698] ? smp_call_function_many_cond+0x3db/0x540 [ 5092.158823] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5092.160417] ? __x64_sys_fspick+0x210/0x210 [ 5092.161994] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5092.163802] on_each_cpu_cond_mask+0x46/0x150 [ 5092.165531] on_each_cpu_cond+0x1c/0x30 [ 5092.166942] invalidate_bh_lrus+0x27/0x30 [ 5092.168544] invalidate_bdev+0x5a/0x90 [ 5092.170075] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5092.172352] generic_shutdown_super+0xb1/0x1b0 [ 5092.174110] kill_block_super+0x29/0x80 [ 5092.175738] deactivate_locked_super+0x52/0xd0 [ 5092.177561] deactivate_super+0x83/0x90 [ 5092.179106] cleanup_mnt+0x5f/0xe0 [ 5092.180643] __cleanup_mnt+0x16/0x20 [ 5092.182013] task_work_run+0xc6/0x110 [ 5092.183555] exit_to_usermode_loop+0x1dd/0x1f0 [ 5092.185380] do_syscall_64+0x3d6/0x440 [ 5092.187184] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5092.189348] RIP: 0033:0x7fbb019648fb [ 5092.190756] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5092.198410] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5092.201580] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5092.204976] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5092.208451] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5092.211531] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5092.214131] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5100.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5100.058984] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5100.093161] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5100.097538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5100.102977] Workqueue: events free_work [libcfs] [ 5100.106479] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5100.110130] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5100.124537] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5100.129344] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5100.139677] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5100.147713] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5100.152772] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5100.158231] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5100.161808] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5100.167688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5100.171934] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5100.177256] Call Trace: [ 5100.178189] [ 5100.179454] ? show_regs.cold.9+0x22/0x2f [ 5100.181807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5100.188637] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5100.191887] ? watchdog+0x40/0x40 [ 5100.194483] ? hrtimer_interrupt+0x128/0x2c0 [ 5100.198161] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5100.200685] ? apic_timer_interrupt+0xf/0x20 [ 5100.203253] [ 5100.207140] ? on_each_cpu+0x51/0x120 [ 5100.209251] ? smp_call_function_many_cond+0x3d9/0x540 [ 5100.211993] ? smp_call_function_many_cond+0x3a8/0x540 [ 5100.214885] ? load_new_mm_cr3+0x120/0x120 [ 5100.217193] ? load_new_mm_cr3+0x120/0x120 [ 5100.219697] on_each_cpu+0x51/0x120 [ 5100.220972] flush_tlb_kernel_range+0x93/0xb0 [ 5100.222939] free_unmap_vmap_area+0x63/0x70 [ 5100.224906] remove_vm_area+0xe7/0x110 [ 5100.226802] __vunmap+0x9c/0x400 [ 5100.228348] vfree+0x4d/0x100 [ 5100.230165] free_work+0x34/0x50 [libcfs] [ 5100.234125] process_one_work+0x2c8/0x700 [ 5100.237493] worker_thread+0x296/0x6e0 [ 5100.238952] ? rescuer_thread+0x570/0x570 [ 5100.241476] kthread+0x1d1/0x200 [ 5100.243450] ? set_kthread_struct+0x70/0x70 [ 5100.245645] ret_from_fork+0x1f/0x30 [ 5104.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5104.060891] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5104.072837] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5104.075553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5104.077588] Workqueue: events free_work [libcfs] [ 5104.078717] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5104.080142] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5104.084638] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5104.086484] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5104.088289] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5104.089896] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5104.091600] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5104.093383] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5104.095170] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5104.097119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5104.098493] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5104.100250] Call Trace: [ 5104.100805] [ 5104.101364] ? show_regs.cold.9+0x22/0x2f [ 5104.102375] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5104.103569] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5104.104628] ? watchdog+0x40/0x40 [ 5104.105475] ? hrtimer_interrupt+0x128/0x2c0 [ 5104.106506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5104.107646] ? apic_timer_interrupt+0xf/0x20 [ 5104.108657] [ 5104.109155] ? on_each_cpu+0x51/0x120 [ 5104.110083] ? smp_call_function_many_cond+0x3d9/0x540 [ 5104.111276] ? smp_call_function_many_cond+0x3a8/0x540 [ 5104.112660] ? load_new_mm_cr3+0x120/0x120 [ 5104.114038] ? load_new_mm_cr3+0x120/0x120 [ 5104.115221] on_each_cpu+0x51/0x120 [ 5104.115946] flush_tlb_kernel_range+0x93/0xb0 [ 5104.117132] free_unmap_vmap_area+0x63/0x70 [ 5104.118803] remove_vm_area+0xe7/0x110 [ 5104.119838] __vunmap+0x9c/0x400 [ 5104.120789] vfree+0x4d/0x100 [ 5104.121565] free_work+0x34/0x50 [libcfs] [ 5104.122665] process_one_work+0x2c8/0x700 [ 5104.123742] worker_thread+0x296/0x6e0 [ 5104.124771] ? rescuer_thread+0x570/0x570 [ 5104.125822] kthread+0x1d1/0x200 [ 5104.126670] ? set_kthread_struct+0x70/0x70 [ 5104.127771] ret_from_fork+0x1f/0x30 [ 5105.119022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2746s! [ 5105.121183] Showing busy workqueues and worker pools: [ 5105.122535] workqueue events: flags=0x0 [ 5105.123700] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5105.125893] in-flight: 56293:free_work [libcfs] [ 5105.127768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5105.133268] in-flight: 22277:free_work [libcfs] [ 5105.134562] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5105.137645] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5105.140411] pending: nfs4_renew_state [nfsv4] [ 5105.142444] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5105.147698] in-flight: 47596:free_work [libcfs] [ 5105.150006] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5105.153391] workqueue events_unbound: flags=0x2 [ 5105.154686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5105.158630] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5105.161633] workqueue events_power_efficient: flags=0x80 [ 5105.164334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5105.169320] pending: neigh_periodic_work, do_cache_clean [ 5105.172761] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5105.177210] pending: check_lifetime [ 5105.179319] workqueue mm_percpu_wq: flags=0x8 [ 5105.181360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5105.183503] pending: vmstat_update [ 5105.185209] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5105.189056] pending: vmstat_update [ 5105.190608] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5105.194377] pending: vmstat_update [ 5105.196421] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5105.198725] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5105.201886] pending: cgroup_pidlist_destroy_work_fn [ 5105.204384] workqueue writeback: flags=0x4a [ 5105.206463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5105.209618] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5105.213634] workqueue xprtiod: flags=0xa [ 5105.215561] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5105.218492] pending: xprt_autoclose [ 5105.220486] workqueue ipv6_addrconf: flags=0x40008 [ 5105.223085] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5105.226383] pending: addrconf_verify_work [ 5105.230284] workqueue knbd0-recv: flags=0x1a [ 5105.232611] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5105.237370] in-flight: 655:recv_work [ 5105.239508] workqueue ptlrpc_pinger: flags=0x2 [ 5105.241381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5105.245082] pending: ptlrpc_pinger_main [ptlrpc] [ 5105.248318] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2747s workers=5 idle: 53684 27131 54850 46584 [ 5105.254090] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2747s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5105.261822] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2747s workers=6 idle: 53660 37674 46417 53664 46418 [ 5105.271830] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5095s workers=2 idle: 657 [ 5117.812011] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 5117.816898] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 5117.826128] (detected by 1, t=2760096 jiffies, g=626849, q=1398) [ 5117.831684] Sending NMI from CPU 1 to CPUs 3: [ 5117.835938] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 5117.835947] rcu: rcu_sched kthread starved for 2759747 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5117.848373] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5117.855863] rcu: RCU grace-period kthread stack dump: [ 5117.864010] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5117.871722] Call Trace: [ 5117.876326] __schedule+0x351/0xcb0 [ 5117.879541] schedule+0xc0/0x180 [ 5117.882520] schedule_timeout+0xb4/0x190 [ 5117.884113] ? do_raw_spin_unlock+0x75/0x190 [ 5117.888534] ? __next_timer_interrupt+0x160/0x160 [ 5117.895493] ? prepare_to_swait_event+0xf3/0x140 [ 5117.899373] rcu_gp_kthread+0x8dd/0x15b0 [ 5117.900780] ? force_qs_rnp+0x2c0/0x2c0 [ 5117.903234] kthread+0x1d1/0x200 [ 5117.906567] ? set_kthread_struct+0x70/0x70 [ 5117.914080] ret_from_fork+0x1f/0x30 [ 5117.919044] rcu: Stack dump where RCU GP kthread last ran: [ 5117.924027] NMI backtrace for cpu 1 [ 5117.927063] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5117.938524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5117.949507] Call Trace: [ 5117.950429] [ 5117.956097] ? dump_stack+0xbb/0x10e [ 5117.957761] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5117.965232] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5117.970638] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5117.976848] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5117.987429] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5117.989447] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5117.993556] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 5117.998790] ? update_process_times+0x6d/0xc0 [ 5118.003451] ? tick_sched_handle.isra.18+0x27/0x90 [ 5118.008283] ? tick_sched_timer+0x53/0xb0 [ 5118.012073] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5118.013892] ? tick_sched_handle.isra.18+0x90/0x90 [ 5118.016645] ? hrtimer_interrupt+0x128/0x2c0 [ 5118.019043] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5118.024186] ? apic_timer_interrupt+0xf/0x20 [ 5118.027416] [ 5118.030029] ? on_each_cpu_cond_mask+0x46/0x150 [ 5118.037223] ? __x64_sys_fspick+0x210/0x210 [ 5118.049099] ? smp_call_function_many_cond+0x3db/0x540 [ 5118.054727] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5118.057971] ? __x64_sys_fspick+0x210/0x210 [ 5118.063385] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5118.072416] ? on_each_cpu_cond_mask+0x46/0x150 [ 5118.075943] ? on_each_cpu_cond+0x1c/0x30 [ 5118.078422] ? invalidate_bh_lrus+0x27/0x30 [ 5118.081360] ? invalidate_bdev+0x5a/0x90 [ 5118.084586] ? ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5118.089772] ? generic_shutdown_super+0xb1/0x1b0 [ 5118.094153] ? kill_block_super+0x29/0x80 [ 5118.096820] ? deactivate_locked_super+0x52/0xd0 [ 5118.100504] ? deactivate_super+0x83/0x90 [ 5118.102548] ? cleanup_mnt+0x5f/0xe0 [ 5118.105388] ? __cleanup_mnt+0x16/0x20 [ 5118.108420] ? task_work_run+0xc6/0x110 [ 5118.109810] ? exit_to_usermode_loop+0x1dd/0x1f0 [ 5118.113602] ? do_syscall_64+0x3d6/0x440 [ 5118.119625] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5132.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5132.059861] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5132.061069] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5132.090506] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5132.095323] Modules linked in: zfs(O) [ 5132.110198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5132.110203] spl(O) lustre(O) [ 5132.113323] Workqueue: events free_work [libcfs] [ 5132.117638] osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) [ 5132.119382] [ 5132.119389] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5132.119403] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5132.119407] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5132.119412] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5132.119414] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5132.119416] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5132.119418] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5132.119420] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5132.119424] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5132.119427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5132.119430] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5132.119438] Call Trace: [ 5132.119442] [ 5132.119447] ? show_regs.cold.9+0x22/0x2f [ 5132.119455] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5132.119461] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5132.119465] ? watchdog+0x40/0x40 [ 5132.119469] ? hrtimer_interrupt+0x128/0x2c0 [ 5132.119474] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5132.119481] ? apic_timer_interrupt+0xf/0x20 [ 5132.119486] [ 5132.119488] ? on_each_cpu+0x51/0x120 [ 5132.119491] ? smp_call_function_many_cond+0x3d9/0x540 [ 5132.119495] ? smp_call_function_many_cond+0x3a8/0x540 [ 5132.119498] ? load_new_mm_cr3+0x120/0x120 [ 5132.119504] ? load_new_mm_cr3+0x120/0x120 [ 5132.121182] ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5132.121249] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5132.121256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5132.121293] Workqueue: events free_work [libcfs] [ 5132.121331] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5132.121339] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5132.121343] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5132.121347] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5132.121350] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5132.121352] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5132.121354] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5132.121356] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5132.121359] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5132.121361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5132.121364] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5132.121371] Call Trace: [ 5132.121375] [ 5132.121379] ? show_regs.cold.9+0x22/0x2f [ 5132.121387] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5132.121407] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5132.121411] ? watchdog+0x40/0x40 [ 5132.121416] ? hrtimer_interrupt+0x128/0x2c0 [ 5132.121420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5132.121428] ? apic_timer_interrupt+0xf/0x20 [ 5132.121433] [ 5132.121435] ? on_each_cpu+0x51/0x120 [ 5132.121439] ? smp_call_function_many_cond+0x3db/0x540 [ 5132.121442] ? smp_call_function_many_cond+0x3a8/0x540 [ 5132.121445] ? load_new_mm_cr3+0x120/0x120 [ 5132.121450] ? load_new_mm_cr3+0x120/0x120 [ 5132.121453] on_each_cpu+0x51/0x120 [ 5132.121457] flush_tlb_kernel_range+0x93/0xb0 [ 5132.121462] free_unmap_vmap_area+0x63/0x70 [ 5132.121467] remove_vm_area+0xe7/0x110 [ 5132.121471] __vunmap+0x9c/0x400 [ 5132.121474] vfree+0x4d/0x100 [ 5132.121477] free_work+0x34/0x50 [libcfs] [ 5132.121504] process_one_work+0x2c8/0x700 [ 5132.121510] worker_thread+0x296/0x6e0 [ 5132.121514] ? rescuer_thread+0x570/0x570 [ 5132.121517] kthread+0x1d1/0x200 [ 5132.121522] ? set_kthread_struct+0x70/0x70 [ 5132.121525] ret_from_fork+0x1f/0x30 [ 5132.359715] on_each_cpu+0x51/0x120 [ 5132.360985] flush_tlb_kernel_range+0x93/0xb0 [ 5132.362922] free_unmap_vmap_area+0x63/0x70 [ 5132.364859] remove_vm_area+0xe7/0x110 [ 5132.366170] __vunmap+0x9c/0x400 [ 5132.367306] vfree+0x4d/0x100 [ 5132.368785] free_work+0x34/0x50 [libcfs] [ 5132.370803] process_one_work+0x2c8/0x700 [ 5132.372981] worker_thread+0x296/0x6e0 [ 5132.374854] ? rescuer_thread+0x570/0x570 [ 5132.376757] kthread+0x1d1/0x200 [ 5132.378159] ? set_kthread_struct+0x70/0x70 [ 5132.380273] ret_from_fork+0x1f/0x30 [ 5135.327056] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2777s! [ 5135.330741] Showing busy workqueues and worker pools: [ 5135.332878] workqueue events: flags=0x0 [ 5135.334563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5135.336974] in-flight: 56293:free_work [libcfs] [ 5135.339164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5135.342193] in-flight: 22277:free_work [libcfs] [ 5135.344906] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5135.348607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5135.353058] pending: nfs4_renew_state [nfsv4] [ 5135.355209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5135.358200] in-flight: 47596:free_work [libcfs] [ 5135.360274] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5135.364172] workqueue events_unbound: flags=0x2 [ 5135.366130] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5135.369922] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5135.374070] workqueue events_power_efficient: flags=0x80 [ 5135.376534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5135.379034] pending: neigh_periodic_work, do_cache_clean [ 5135.380672] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5135.382590] pending: check_lifetime [ 5135.384376] workqueue mm_percpu_wq: flags=0x8 [ 5135.385930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5135.388712] pending: vmstat_update [ 5135.390601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5135.395782] pending: vmstat_update [ 5135.397268] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5135.402419] pending: vmstat_update [ 5135.404539] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5135.407456] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5135.410942] pending: cgroup_pidlist_destroy_work_fn [ 5135.415053] workqueue writeback: flags=0x4a [ 5135.418694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5135.421164] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5135.424250] workqueue xprtiod: flags=0xa [ 5135.427255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5135.431735] pending: xprt_autoclose [ 5135.437974] workqueue ipv6_addrconf: flags=0x40008 [ 5135.440943] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5135.445065] pending: addrconf_verify_work [ 5135.447465] workqueue knbd0-recv: flags=0x1a [ 5135.449264] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5135.453772] in-flight: 655:recv_work [ 5135.456936] workqueue ptlrpc_pinger: flags=0x2 [ 5135.460349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5135.464015] pending: ptlrpc_pinger_main [ptlrpc] [ 5135.466112] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2777s workers=5 idle: 53684 27131 54850 46584 [ 5135.470881] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2777s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5135.475392] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2777s workers=6 idle: 53660 37674 46417 53664 46418 [ 5135.480551] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5125s workers=2 idle: 657 [ 5144.061730] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5144.072561] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5144.097932] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5144.105898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5144.110772] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5144.113520] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5144.128377] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5144.132860] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5144.136502] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5144.144075] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5144.148454] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5144.152473] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5144.155775] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5144.162858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5144.166513] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5144.170293] Call Trace: [ 5144.171383] [ 5144.172303] ? show_regs.cold.9+0x22/0x2f [ 5144.174329] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5144.176615] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5144.178704] ? watchdog+0x40/0x40 [ 5144.180560] ? hrtimer_interrupt+0x128/0x2c0 [ 5144.182613] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5144.187339] ? apic_timer_interrupt+0xf/0x20 [ 5144.189011] [ 5144.191554] ? on_each_cpu_cond_mask+0x46/0x150 [ 5144.194029] ? __x64_sys_fspick+0x210/0x210 [ 5144.196995] ? smp_call_function_many_cond+0x3db/0x540 [ 5144.199981] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5144.202490] ? __x64_sys_fspick+0x210/0x210 [ 5144.204805] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5144.208085] on_each_cpu_cond_mask+0x46/0x150 [ 5144.211684] on_each_cpu_cond+0x1c/0x30 [ 5144.213467] invalidate_bh_lrus+0x27/0x30 [ 5144.215329] invalidate_bdev+0x5a/0x90 [ 5144.217041] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5144.222681] generic_shutdown_super+0xb1/0x1b0 [ 5144.227282] kill_block_super+0x29/0x80 [ 5144.228829] deactivate_locked_super+0x52/0xd0 [ 5144.232419] deactivate_super+0x83/0x90 [ 5144.234555] cleanup_mnt+0x5f/0xe0 [ 5144.237591] __cleanup_mnt+0x16/0x20 [ 5144.239820] task_work_run+0xc6/0x110 [ 5144.241350] exit_to_usermode_loop+0x1dd/0x1f0 [ 5144.244395] do_syscall_64+0x3d6/0x440 [ 5144.248045] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5144.253335] RIP: 0033:0x7fbb019648fb [ 5144.255029] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5144.268080] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5144.271856] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5144.275580] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5144.281333] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5144.286405] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5144.289029] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5160.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5160.058167] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5160.059104] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5160.081550] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5160.086060] Modules linked in: [ 5160.091526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5160.091530] zfs(O) spl(O) [ 5160.093750] Workqueue: events free_work [libcfs] [ 5160.099130] lustre(O) [ 5160.100662] [ 5160.103200] osp(O) ofd(O) [ 5160.104542] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5160.105416] lod(O) [ 5160.106560] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5160.109646] ost(O) [ 5160.111736] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5160.111755] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5160.111757] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5160.111759] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5160.111762] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5160.111764] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5160.111767] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5160.111769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5160.111771] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5160.111779] Call Trace: [ 5160.111785] [ 5160.111790] ? show_regs.cold.9+0x22/0x2f [ 5160.111803] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5160.111809] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5160.111813] ? watchdog+0x40/0x40 [ 5160.111817] ? hrtimer_interrupt+0x128/0x2c0 [ 5160.111820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5160.111827] ? apic_timer_interrupt+0xf/0x20 [ 5160.111831] [ 5160.111833] ? on_each_cpu+0x51/0x120 [ 5160.111838] ? smp_call_function_many_cond+0x3db/0x540 [ 5160.111842] ? smp_call_function_many_cond+0x3a8/0x540 [ 5160.111845] ? load_new_mm_cr3+0x120/0x120 [ 5160.111850] ? load_new_mm_cr3+0x120/0x120 [ 5160.111854] on_each_cpu+0x51/0x120 [ 5160.111858] flush_tlb_kernel_range+0x93/0xb0 [ 5160.111864] free_unmap_vmap_area+0x63/0x70 [ 5160.111879] remove_vm_area+0xe7/0x110 [ 5160.111884] __vunmap+0x9c/0x400 [ 5160.111888] vfree+0x4d/0x100 [ 5160.111894] free_work+0x34/0x50 [libcfs] [ 5160.111936] process_one_work+0x2c8/0x700 [ 5160.111943] worker_thread+0x296/0x6e0 [ 5160.111948] ? rescuer_thread+0x570/0x570 [ 5160.111951] kthread+0x1d1/0x200 [ 5160.111956] ? set_kthread_struct+0x70/0x70 [ 5160.111960] ret_from_fork+0x1f/0x30 [ 5160.211836] mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5160.242885] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5160.248743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5160.253476] Workqueue: events free_work [libcfs] [ 5160.255864] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5160.258694] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5160.273569] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5160.280668] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5160.283708] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5160.287641] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5160.292147] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5160.296573] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5160.300417] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5160.304527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5160.307502] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5160.311036] Call Trace: [ 5160.312384] [ 5160.313378] ? show_regs.cold.9+0x22/0x2f [ 5160.315395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5160.317701] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5160.320099] ? watchdog+0x40/0x40 [ 5160.321616] ? hrtimer_interrupt+0x128/0x2c0 [ 5160.323708] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5160.326164] ? apic_timer_interrupt+0xf/0x20 [ 5160.327661] [ 5160.328460] ? on_each_cpu+0x51/0x120 [ 5160.330448] ? smp_call_function_many_cond+0x3db/0x540 [ 5160.333060] ? smp_call_function_many_cond+0x3a8/0x540 [ 5160.335473] ? load_new_mm_cr3+0x120/0x120 [ 5160.337501] ? load_new_mm_cr3+0x120/0x120 [ 5160.339541] on_each_cpu+0x51/0x120 [ 5160.340987] flush_tlb_kernel_range+0x93/0xb0 [ 5160.342558] free_unmap_vmap_area+0x63/0x70 [ 5160.344433] remove_vm_area+0xe7/0x110 [ 5160.346249] __vunmap+0x9c/0x400 [ 5160.347736] vfree+0x4d/0x100 [ 5160.349488] free_work+0x34/0x50 [libcfs] [ 5160.352190] process_one_work+0x2c8/0x700 [ 5160.353773] worker_thread+0x296/0x6e0 [ 5160.355622] ? rescuer_thread+0x570/0x570 [ 5160.357577] kthread+0x1d1/0x200 [ 5160.359377] ? set_kthread_struct+0x70/0x70 [ 5160.361201] ret_from_fork+0x1f/0x30 [ 5165.535043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2807s! [ 5165.540439] Showing busy workqueues and worker pools: [ 5165.545157] workqueue events: flags=0x0 [ 5165.548319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5165.553516] in-flight: 56293:free_work [libcfs] [ 5165.555964] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5165.562983] in-flight: 22277:free_work [libcfs] [ 5165.568791] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5165.575103] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5165.579388] pending: nfs4_renew_state [nfsv4] [ 5165.582537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5165.587216] in-flight: 47596:free_work [libcfs] [ 5165.590279] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5165.597128] workqueue events_unbound: flags=0x2 [ 5165.599861] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5165.604051] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5165.610781] workqueue events_power_efficient: flags=0x80 [ 5165.614929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5165.622283] pending: neigh_periodic_work, do_cache_clean [ 5165.626981] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5165.631376] pending: check_lifetime [ 5165.634353] workqueue mm_percpu_wq: flags=0x8 [ 5165.637394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5165.643371] pending: vmstat_update [ 5165.645954] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5165.648355] pending: vmstat_update [ 5165.651227] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5165.654885] pending: vmstat_update [ 5165.657636] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5165.660351] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5165.663539] pending: cgroup_pidlist_destroy_work_fn [ 5165.665756] workqueue writeback: flags=0x4a [ 5165.669988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5165.675598] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5165.679623] workqueue xprtiod: flags=0xa [ 5165.682120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5165.685331] pending: xprt_autoclose [ 5165.687591] workqueue ipv6_addrconf: flags=0x40008 [ 5165.690247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5165.692709] pending: addrconf_verify_work [ 5165.695580] workqueue knbd0-recv: flags=0x1a [ 5165.697579] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5165.700588] in-flight: 655:recv_work [ 5165.702744] workqueue ptlrpc_pinger: flags=0x2 [ 5165.706775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5165.709111] pending: ptlrpc_pinger_main [ptlrpc] [ 5165.713121] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2807s workers=5 idle: 53684 27131 54850 46584 [ 5165.722099] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2807s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5165.729940] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2808s workers=6 idle: 53660 37674 46417 53664 46418 [ 5165.734465] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5155s workers=2 idle: 657 [ 5172.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5172.061987] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5172.092368] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5172.099827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5172.107278] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5172.113425] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5172.122592] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5172.127947] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5172.132183] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5172.136798] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5172.140283] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5172.145616] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5172.148888] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5172.154386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5172.158593] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5172.162594] Call Trace: [ 5172.163724] [ 5172.164721] ? show_regs.cold.9+0x22/0x2f [ 5172.166589] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5172.169941] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5172.172924] ? watchdog+0x40/0x40 [ 5172.174099] ? hrtimer_interrupt+0x128/0x2c0 [ 5172.179227] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5172.183170] ? apic_timer_interrupt+0xf/0x20 [ 5172.185163] [ 5172.186130] ? on_each_cpu_cond_mask+0x46/0x150 [ 5172.188744] ? __x64_sys_fspick+0x210/0x210 [ 5172.191762] ? smp_call_function_many_cond+0x3d9/0x540 [ 5172.194003] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5172.195968] ? __x64_sys_fspick+0x210/0x210 [ 5172.197426] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5172.199885] on_each_cpu_cond_mask+0x46/0x150 [ 5172.202505] on_each_cpu_cond+0x1c/0x30 [ 5172.204702] invalidate_bh_lrus+0x27/0x30 [ 5172.208042] invalidate_bdev+0x5a/0x90 [ 5172.209776] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5172.214775] generic_shutdown_super+0xb1/0x1b0 [ 5172.217396] kill_block_super+0x29/0x80 [ 5172.221286] deactivate_locked_super+0x52/0xd0 [ 5172.222849] deactivate_super+0x83/0x90 [ 5172.224214] cleanup_mnt+0x5f/0xe0 [ 5172.227054] __cleanup_mnt+0x16/0x20 [ 5172.228387] task_work_run+0xc6/0x110 [ 5172.231541] exit_to_usermode_loop+0x1dd/0x1f0 [ 5172.235495] do_syscall_64+0x3d6/0x440 [ 5172.238016] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5172.242151] RIP: 0033:0x7fbb019648fb [ 5172.246361] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5172.255356] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5172.258011] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5172.260057] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5172.262367] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5172.264867] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5172.267176] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5188.061331] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5188.069335] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5188.094995] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5188.100723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5188.110934] Workqueue: events free_work [libcfs] [ 5188.115028] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5188.119997] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5188.126751] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5188.130864] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5188.139805] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5188.141686] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5188.150372] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5188.153870] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5188.159697] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5188.162055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5188.166852] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5188.172237] Call Trace: [ 5188.173071] [ 5188.174416] ? show_regs.cold.9+0x22/0x2f [ 5188.178096] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5188.180311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5188.182267] ? watchdog+0x40/0x40 [ 5188.183607] ? hrtimer_interrupt+0x128/0x2c0 [ 5188.185846] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5188.187929] ? apic_timer_interrupt+0xf/0x20 [ 5188.189885] [ 5188.190861] ? on_each_cpu+0x51/0x120 [ 5188.192326] ? smp_call_function_many_cond+0x3db/0x540 [ 5188.195949] ? smp_call_function_many_cond+0x3a8/0x540 [ 5188.201422] ? load_new_mm_cr3+0x120/0x120 [ 5188.205338] ? load_new_mm_cr3+0x120/0x120 [ 5188.208463] on_each_cpu+0x51/0x120 [ 5188.210514] flush_tlb_kernel_range+0x93/0xb0 [ 5188.214697] free_unmap_vmap_area+0x63/0x70 [ 5188.216957] remove_vm_area+0xe7/0x110 [ 5188.221922] __vunmap+0x9c/0x400 [ 5188.223147] vfree+0x4d/0x100 [ 5188.228435] free_work+0x34/0x50 [libcfs] [ 5188.230687] process_one_work+0x2c8/0x700 [ 5188.236568] worker_thread+0x296/0x6e0 [ 5188.237972] ? rescuer_thread+0x570/0x570 [ 5188.240953] kthread+0x1d1/0x200 [ 5188.242633] ? set_kthread_struct+0x70/0x70 [ 5188.246206] ret_from_fork+0x1f/0x30 [ 5192.056025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5192.061530] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5192.114557] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5192.127828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5192.135532] Workqueue: events free_work [libcfs] [ 5192.140710] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5192.147512] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5192.167745] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5192.172673] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5192.179424] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5192.186612] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5192.194395] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5192.200430] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5192.208270] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5192.214315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5192.218545] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5192.222531] Call Trace: [ 5192.224959] [ 5192.225772] ? show_regs.cold.9+0x22/0x2f [ 5192.228727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5192.240674] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5192.242701] ? watchdog+0x40/0x40 [ 5192.244115] ? hrtimer_interrupt+0x128/0x2c0 [ 5192.247859] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5192.249827] ? apic_timer_interrupt+0xf/0x20 [ 5192.259846] [ 5192.261877] ? on_each_cpu+0x51/0x120 [ 5192.265057] ? smp_call_function_many_cond+0x3db/0x540 [ 5192.267026] ? smp_call_function_many_cond+0x3a8/0x540 [ 5192.275312] ? load_new_mm_cr3+0x120/0x120 [ 5192.278728] ? load_new_mm_cr3+0x120/0x120 [ 5192.281325] on_each_cpu+0x51/0x120 [ 5192.282662] flush_tlb_kernel_range+0x93/0xb0 [ 5192.288942] free_unmap_vmap_area+0x63/0x70 [ 5192.291708] remove_vm_area+0xe7/0x110 [ 5192.296306] __vunmap+0x9c/0x400 [ 5192.298470] vfree+0x4d/0x100 [ 5192.299680] free_work+0x34/0x50 [libcfs] [ 5192.301181] process_one_work+0x2c8/0x700 [ 5192.305892] worker_thread+0x296/0x6e0 [ 5192.312267] ? rescuer_thread+0x570/0x570 [ 5192.316203] kthread+0x1d1/0x200 [ 5192.319263] ? set_kthread_struct+0x70/0x70 [ 5192.321411] ret_from_fork+0x1f/0x30 [ 5195.744059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2837s! [ 5195.748901] Showing busy workqueues and worker pools: [ 5195.752058] workqueue events: flags=0x0 [ 5195.754114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5195.757556] in-flight: 56293:free_work [libcfs] [ 5195.759773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5195.762814] in-flight: 22277:free_work [libcfs] [ 5195.765476] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5195.769234] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5195.771579] pending: nfs4_renew_state [nfsv4] [ 5195.773917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5195.776640] in-flight: 47596:free_work [libcfs] [ 5195.779298] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5195.784402] workqueue events_unbound: flags=0x2 [ 5195.786683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5195.791008] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5195.795586] workqueue events_power_efficient: flags=0x80 [ 5195.797938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5195.802298] pending: neigh_periodic_work, do_cache_clean [ 5195.805774] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5195.809520] pending: check_lifetime [ 5195.810933] workqueue mm_percpu_wq: flags=0x8 [ 5195.812772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5195.819232] pending: vmstat_update [ 5195.821971] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5195.825586] pending: vmstat_update [ 5195.827079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5195.837932] pending: vmstat_update [ 5195.842236] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5195.849794] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5195.856652] pending: cgroup_pidlist_destroy_work_fn [ 5195.861539] workqueue writeback: flags=0x4a [ 5195.866272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5195.872246] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5195.876552] workqueue xprtiod: flags=0xa [ 5195.879324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5195.883542] pending: xprt_autoclose [ 5195.886085] workqueue ipv6_addrconf: flags=0x40008 [ 5195.889078] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5195.892527] pending: addrconf_verify_work [ 5195.895088] workqueue knbd0-recv: flags=0x1a [ 5195.897871] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5195.901537] in-flight: 655:recv_work [ 5195.905990] workqueue ptlrpc_pinger: flags=0x2 [ 5195.908458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5195.912766] pending: ptlrpc_pinger_main [ptlrpc] [ 5195.917453] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2838s workers=5 idle: 53684 27131 54850 46584 [ 5195.922481] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2838s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5195.928248] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2838s workers=6 idle: 53660 37674 46417 53664 46418 [ 5195.935728] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5185s workers=2 idle: 657 [ 5200.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5200.063196] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.100467] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.109828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5200.118491] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5200.126993] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5200.141212] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5200.147551] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5200.152160] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5200.155746] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5200.165095] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5200.175761] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5200.181978] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5200.190574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.202192] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5200.207243] Call Trace: [ 5200.209281] [ 5200.211855] ? show_regs.cold.9+0x22/0x2f [ 5200.217129] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.220990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.223026] ? watchdog+0x40/0x40 [ 5200.228116] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.233847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.237030] ? apic_timer_interrupt+0xf/0x20 [ 5200.240847] [ 5200.243589] ? on_each_cpu_cond_mask+0x46/0x150 [ 5200.246993] ? __x64_sys_fspick+0x210/0x210 [ 5200.250719] ? smp_call_function_many_cond+0x3db/0x540 [ 5200.260657] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5200.267282] ? __x64_sys_fspick+0x210/0x210 [ 5200.269964] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5200.272720] on_each_cpu_cond_mask+0x46/0x150 [ 5200.277196] on_each_cpu_cond+0x1c/0x30 [ 5200.278715] invalidate_bh_lrus+0x27/0x30 [ 5200.283391] invalidate_bdev+0x5a/0x90 [ 5200.287571] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5200.289606] generic_shutdown_super+0xb1/0x1b0 [ 5200.296616] kill_block_super+0x29/0x80 [ 5200.304650] deactivate_locked_super+0x52/0xd0 [ 5200.309354] deactivate_super+0x83/0x90 [ 5200.313805] cleanup_mnt+0x5f/0xe0 [ 5200.317681] __cleanup_mnt+0x16/0x20 [ 5200.325473] task_work_run+0xc6/0x110 [ 5200.331486] exit_to_usermode_loop+0x1dd/0x1f0 [ 5200.336388] do_syscall_64+0x3d6/0x440 [ 5200.339375] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5200.345834] RIP: 0033:0x7fbb019648fb [ 5200.347125] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5200.368402] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5200.375411] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5200.381897] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5200.384825] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5200.393726] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5200.399320] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5216.059008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5216.064506] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5216.101459] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5216.111613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5216.116485] Workqueue: events free_work [libcfs] [ 5216.119325] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5216.123606] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5216.139753] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5216.147460] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5216.153504] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5216.165280] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5216.175934] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5216.186478] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5216.193638] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5216.200642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5216.205422] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5216.215484] Call Trace: [ 5216.222056] [ 5216.225914] ? show_regs.cold.9+0x22/0x2f [ 5216.232265] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5216.235238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5216.237317] ? watchdog+0x40/0x40 [ 5216.238851] ? hrtimer_interrupt+0x128/0x2c0 [ 5216.240545] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5216.244465] ? apic_timer_interrupt+0xf/0x20 [ 5216.248747] [ 5216.251061] ? on_each_cpu+0x51/0x120 [ 5216.253704] ? smp_call_function_many_cond+0x3db/0x540 [ 5216.258411] ? smp_call_function_many_cond+0x3a8/0x540 [ 5216.262252] ? load_new_mm_cr3+0x120/0x120 [ 5216.268485] ? load_new_mm_cr3+0x120/0x120 [ 5216.273367] on_each_cpu+0x51/0x120 [ 5216.276926] flush_tlb_kernel_range+0x93/0xb0 [ 5216.280409] free_unmap_vmap_area+0x63/0x70 [ 5216.284174] remove_vm_area+0xe7/0x110 [ 5216.288681] __vunmap+0x9c/0x400 [ 5216.290138] vfree+0x4d/0x100 [ 5216.291080] free_work+0x34/0x50 [libcfs] [ 5216.297956] process_one_work+0x2c8/0x700 [ 5216.301972] worker_thread+0x296/0x6e0 [ 5216.306367] ? rescuer_thread+0x570/0x570 [ 5216.309674] kthread+0x1d1/0x200 [ 5216.311986] ? set_kthread_struct+0x70/0x70 [ 5216.313437] ret_from_fork+0x1f/0x30 [ 5220.057947] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5220.066627] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5220.113000] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5220.122268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5220.128102] Workqueue: events free_work [libcfs] [ 5220.131714] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5220.135898] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5220.152730] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5220.162552] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5220.170932] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5220.179179] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5220.191750] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5220.198439] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5220.204728] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5220.210894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5220.214944] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5220.221106] Call Trace: [ 5220.222812] [ 5220.224216] ? show_regs.cold.9+0x22/0x2f [ 5220.227096] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5220.230101] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5220.232980] ? watchdog+0x40/0x40 [ 5220.234595] ? hrtimer_interrupt+0x128/0x2c0 [ 5220.237650] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5220.240390] ? apic_timer_interrupt+0xf/0x20 [ 5220.242292] [ 5220.243938] ? on_each_cpu+0x51/0x120 [ 5220.247840] ? smp_call_function_many_cond+0x3db/0x540 [ 5220.251590] ? smp_call_function_many_cond+0x3a8/0x540 [ 5220.255408] ? load_new_mm_cr3+0x120/0x120 [ 5220.257943] ? load_new_mm_cr3+0x120/0x120 [ 5220.260551] on_each_cpu+0x51/0x120 [ 5220.263140] flush_tlb_kernel_range+0x93/0xb0 [ 5220.266464] free_unmap_vmap_area+0x63/0x70 [ 5220.268794] remove_vm_area+0xe7/0x110 [ 5220.275738] __vunmap+0x9c/0x400 [ 5220.282119] vfree+0x4d/0x100 [ 5220.283898] free_work+0x34/0x50 [libcfs] [ 5220.288890] process_one_work+0x2c8/0x700 [ 5220.293783] worker_thread+0x296/0x6e0 [ 5220.297569] ? rescuer_thread+0x570/0x570 [ 5220.307949] kthread+0x1d1/0x200 [ 5220.312351] ? set_kthread_struct+0x70/0x70 [ 5220.315403] ret_from_fork+0x1f/0x30 [ 5225.951026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2867s! [ 5225.956571] Showing busy workqueues and worker pools: [ 5225.960233] workqueue events: flags=0x0 [ 5225.964634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5225.971794] in-flight: 56293:free_work [libcfs] [ 5225.977945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5225.987440] in-flight: 22277:free_work [libcfs] [ 5225.993001] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5226.003023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5226.006706] pending: nfs4_renew_state [nfsv4] [ 5226.009958] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5226.013207] in-flight: 47596:free_work [libcfs] [ 5226.015723] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5226.020695] workqueue events_unbound: flags=0x2 [ 5226.023041] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5226.026274] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5226.032370] workqueue events_power_efficient: flags=0x80 [ 5226.035628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5226.049234] pending: neigh_periodic_work, do_cache_clean [ 5226.057353] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5226.065788] pending: check_lifetime [ 5226.070526] workqueue mm_percpu_wq: flags=0x8 [ 5226.073660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5226.076287] pending: vmstat_update [ 5226.079960] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5226.083932] pending: vmstat_update [ 5226.085958] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5226.089571] pending: vmstat_update [ 5226.091179] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5226.093672] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5226.096775] pending: cgroup_pidlist_destroy_work_fn [ 5226.099762] workqueue writeback: flags=0x4a [ 5226.101958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5226.105295] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5226.108579] workqueue xprtiod: flags=0xa [ 5226.111455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5226.115990] pending: xprt_autoclose [ 5226.118013] workqueue ipv6_addrconf: flags=0x40008 [ 5226.120669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5226.124331] pending: addrconf_verify_work [ 5226.132894] workqueue knbd0-recv: flags=0x1a [ 5226.140017] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5226.145798] in-flight: 655:recv_work [ 5226.148690] workqueue ptlrpc_pinger: flags=0x2 [ 5226.151674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5226.155202] pending: ptlrpc_pinger_main [ptlrpc] [ 5226.161875] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2868s workers=5 idle: 53684 27131 54850 46584 [ 5226.172208] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2868s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5226.178438] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2868s workers=6 idle: 53660 37674 46417 53664 46418 [ 5226.183200] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5215s workers=2 idle: 657 [ 5228.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5228.060185] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5228.078837] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5228.083169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5228.086854] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5228.089270] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5228.096832] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5228.099942] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5228.103987] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5228.106628] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5228.112750] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5228.116668] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5228.125679] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5228.129991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5228.132910] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5228.141170] Call Trace: [ 5228.142795] [ 5228.145491] ? show_regs.cold.9+0x22/0x2f [ 5228.148996] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5228.155986] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5228.160239] ? watchdog+0x40/0x40 [ 5228.164964] ? hrtimer_interrupt+0x128/0x2c0 [ 5228.168021] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5228.170910] ? apic_timer_interrupt+0xf/0x20 [ 5228.174829] [ 5228.179712] ? on_each_cpu_cond_mask+0x46/0x150 [ 5228.181381] ? __x64_sys_fspick+0x210/0x210 [ 5228.185833] ? smp_call_function_many_cond+0x3db/0x540 [ 5228.189109] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5228.191669] ? __x64_sys_fspick+0x210/0x210 [ 5228.197676] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5228.199072] on_each_cpu_cond_mask+0x46/0x150 [ 5228.201541] on_each_cpu_cond+0x1c/0x30 [ 5228.203917] invalidate_bh_lrus+0x27/0x30 [ 5228.207595] invalidate_bdev+0x5a/0x90 [ 5228.211145] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5228.218594] generic_shutdown_super+0xb1/0x1b0 [ 5228.220352] kill_block_super+0x29/0x80 [ 5228.222147] deactivate_locked_super+0x52/0xd0 [ 5228.226883] deactivate_super+0x83/0x90 [ 5228.230242] cleanup_mnt+0x5f/0xe0 [ 5228.234787] __cleanup_mnt+0x16/0x20 [ 5228.236365] task_work_run+0xc6/0x110 [ 5228.238754] exit_to_usermode_loop+0x1dd/0x1f0 [ 5228.242751] do_syscall_64+0x3d6/0x440 [ 5228.247462] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5228.250973] RIP: 0033:0x7fbb019648fb [ 5228.253760] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5228.268295] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5228.274407] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5228.279997] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5228.285391] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5228.291760] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5228.300272] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5244.059077] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5244.062527] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5244.086969] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5244.092378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5244.097497] Workqueue: events free_work [libcfs] [ 5244.099305] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5244.102665] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5244.110900] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5244.114417] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5244.117094] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5244.119721] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5244.122227] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5244.124914] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5244.127500] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5244.134870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5244.141065] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5244.145297] Call Trace: [ 5244.146034] [ 5244.146696] ? show_regs.cold.9+0x22/0x2f [ 5244.151708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5244.154076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5244.156490] ? watchdog+0x40/0x40 [ 5244.157706] ? hrtimer_interrupt+0x128/0x2c0 [ 5244.159388] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5244.162412] ? apic_timer_interrupt+0xf/0x20 [ 5244.167644] [ 5244.168601] ? on_each_cpu+0x51/0x120 [ 5244.170324] ? smp_call_function_many_cond+0x3db/0x540 [ 5244.173028] ? smp_call_function_many_cond+0x3a8/0x540 [ 5244.175479] ? load_new_mm_cr3+0x120/0x120 [ 5244.177254] ? load_new_mm_cr3+0x120/0x120 [ 5244.179162] on_each_cpu+0x51/0x120 [ 5244.180647] flush_tlb_kernel_range+0x93/0xb0 [ 5244.182632] free_unmap_vmap_area+0x63/0x70 [ 5244.185076] remove_vm_area+0xe7/0x110 [ 5244.187293] __vunmap+0x9c/0x400 [ 5244.189129] vfree+0x4d/0x100 [ 5244.190565] free_work+0x34/0x50 [libcfs] [ 5244.192198] process_one_work+0x2c8/0x700 [ 5244.193842] worker_thread+0x296/0x6e0 [ 5244.195236] ? rescuer_thread+0x570/0x570 [ 5244.196891] kthread+0x1d1/0x200 [ 5244.198270] ? set_kthread_struct+0x70/0x70 [ 5244.199860] ret_from_fork+0x1f/0x30 [ 5252.055194] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5252.062962] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5252.096793] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5252.108399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5252.118554] Workqueue: events free_work [libcfs] [ 5252.127505] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5252.129963] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5252.140708] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5252.145675] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5252.154019] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5252.159216] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5252.162659] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5252.165416] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5252.170519] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5252.178917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5252.184067] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5252.193200] Call Trace: [ 5252.195125] [ 5252.195755] ? show_regs.cold.9+0x22/0x2f [ 5252.199127] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5252.202104] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5252.204632] ? watchdog+0x40/0x40 [ 5252.206957] ? hrtimer_interrupt+0x128/0x2c0 [ 5252.212133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5252.215250] ? apic_timer_interrupt+0xf/0x20 [ 5252.220614] [ 5252.222260] ? on_each_cpu+0x51/0x120 [ 5252.226879] ? smp_call_function_many_cond+0x3d9/0x540 [ 5252.231722] ? smp_call_function_many_cond+0x3a8/0x540 [ 5252.236030] ? load_new_mm_cr3+0x120/0x120 [ 5252.241513] ? load_new_mm_cr3+0x120/0x120 [ 5252.245639] on_each_cpu+0x51/0x120 [ 5252.247701] flush_tlb_kernel_range+0x93/0xb0 [ 5252.250573] free_unmap_vmap_area+0x63/0x70 [ 5252.253017] remove_vm_area+0xe7/0x110 [ 5252.255792] __vunmap+0x9c/0x400 [ 5252.263177] vfree+0x4d/0x100 [ 5252.267066] free_work+0x34/0x50 [libcfs] [ 5252.271013] process_one_work+0x2c8/0x700 [ 5252.272499] worker_thread+0x296/0x6e0 [ 5252.276516] ? rescuer_thread+0x570/0x570 [ 5252.279257] kthread+0x1d1/0x200 [ 5252.281245] ? set_kthread_struct+0x70/0x70 [ 5252.283509] ret_from_fork+0x1f/0x30 [ 5256.060464] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5256.068163] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5256.108625] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5256.119544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5256.126327] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5256.131612] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5256.148768] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5256.155098] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5256.159322] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5256.163272] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5256.167297] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5256.171456] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5256.176197] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5256.180455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5256.183200] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5256.186423] Call Trace: [ 5256.187454] [ 5256.188395] ? show_regs.cold.9+0x22/0x2f [ 5256.190434] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5256.192466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5256.194288] ? watchdog+0x40/0x40 [ 5256.195400] ? hrtimer_interrupt+0x128/0x2c0 [ 5256.197262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5256.200811] ? apic_timer_interrupt+0xf/0x20 [ 5256.203235] [ 5256.204942] ? on_each_cpu_cond_mask+0x46/0x150 [ 5256.209318] ? __x64_sys_fspick+0x210/0x210 [ 5256.212068] ? smp_call_function_many_cond+0x3db/0x540 [ 5256.215036] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5256.218474] ? __x64_sys_fspick+0x210/0x210 [ 5256.223165] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5256.226327] on_each_cpu_cond_mask+0x46/0x150 [ 5256.229085] on_each_cpu_cond+0x1c/0x30 [ 5256.231055] invalidate_bh_lrus+0x27/0x30 [ 5256.232871] invalidate_bdev+0x5a/0x90 [ 5256.234210] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5256.243196] generic_shutdown_super+0xb1/0x1b0 [ 5256.246906] kill_block_super+0x29/0x80 [ 5256.251955] deactivate_locked_super+0x52/0xd0 [ 5256.254643] deactivate_super+0x83/0x90 [ 5256.260182] cleanup_mnt+0x5f/0xe0 [ 5256.262971] __cleanup_mnt+0x16/0x20 [ 5256.264588] task_work_run+0xc6/0x110 [ 5256.266501] exit_to_usermode_loop+0x1dd/0x1f0 [ 5256.268332] do_syscall_64+0x3d6/0x440 [ 5256.269853] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5256.273502] RIP: 0033:0x7fbb019648fb [ 5256.275151] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5256.286049] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5256.289931] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5256.293612] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5256.298408] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5256.306209] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5256.312935] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5256.676494] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2898s! [ 5256.682458] Showing busy workqueues and worker pools: [ 5256.686771] workqueue events: flags=0x0 [ 5256.691751] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5256.696657] in-flight: 56293:free_work [libcfs] [ 5256.702031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5256.707207] in-flight: 22277:free_work [libcfs] [ 5256.711981] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5256.719909] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5256.728410] pending: nfs4_renew_state [nfsv4] [ 5256.736214] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5256.749946] in-flight: 47596:free_work [libcfs] [ 5256.751833] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5256.764192] workqueue events_unbound: flags=0x2 [ 5256.767696] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5256.775211] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5256.781647] workqueue events_power_efficient: flags=0x80 [ 5256.784662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5256.792419] pending: neigh_periodic_work, do_cache_clean [ 5256.802134] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5256.810264] pending: check_lifetime [ 5256.811774] workqueue mm_percpu_wq: flags=0x8 [ 5256.818206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5256.821927] pending: vmstat_update [ 5256.827271] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5256.834379] pending: vmstat_update [ 5256.836876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5256.841247] pending: vmstat_update [ 5256.845920] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5256.854722] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5256.862905] pending: cgroup_pidlist_destroy_work_fn [ 5256.866513] workqueue writeback: flags=0x4a [ 5256.874153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5256.884324] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5256.896743] workqueue xprtiod: flags=0xa [ 5256.901916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5256.906647] pending: xprt_autoclose [ 5256.913585] workqueue ipv6_addrconf: flags=0x40008 [ 5256.916576] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5256.923649] pending: addrconf_verify_work [ 5256.928946] workqueue knbd0-recv: flags=0x1a [ 5256.931858] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5256.939183] in-flight: 655:recv_work [ 5256.943320] workqueue ptlrpc_pinger: flags=0x2 [ 5256.946859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5256.955780] pending: ptlrpc_pinger_main [ptlrpc] [ 5256.957971] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2899s workers=5 idle: 53684 27131 54850 46584 [ 5256.967819] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2899s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5256.981554] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2899s workers=6 idle: 53660 37674 46417 53664 46418 [ 5256.993262] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5246s workers=2 idle: 657 [ 5272.059150] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5272.063077] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5272.090043] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5272.094879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5272.098607] Workqueue: events free_work [libcfs] [ 5272.101437] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5272.107091] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5272.121029] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5272.126023] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5272.129928] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5272.132967] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5272.134826] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5272.140924] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5272.146851] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5272.154882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5272.161354] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5272.168935] Call Trace: [ 5272.171247] [ 5272.176501] ? show_regs.cold.9+0x22/0x2f [ 5272.180164] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5272.185108] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5272.189808] ? watchdog+0x40/0x40 [ 5272.193335] ? hrtimer_interrupt+0x128/0x2c0 [ 5272.196619] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5272.199487] ? apic_timer_interrupt+0xf/0x20 [ 5272.201686] [ 5272.205007] ? on_each_cpu+0x51/0x120 [ 5272.206155] ? smp_call_function_many_cond+0x3d9/0x540 [ 5272.208974] ? smp_call_function_many_cond+0x3a8/0x540 [ 5272.213048] ? load_new_mm_cr3+0x120/0x120 [ 5272.215767] ? load_new_mm_cr3+0x120/0x120 [ 5272.220299] on_each_cpu+0x51/0x120 [ 5272.222432] flush_tlb_kernel_range+0x93/0xb0 [ 5272.228074] free_unmap_vmap_area+0x63/0x70 [ 5272.230679] remove_vm_area+0xe7/0x110 [ 5272.234953] __vunmap+0x9c/0x400 [ 5272.236680] vfree+0x4d/0x100 [ 5272.238315] free_work+0x34/0x50 [libcfs] [ 5272.241114] process_one_work+0x2c8/0x700 [ 5272.244268] worker_thread+0x296/0x6e0 [ 5272.247268] ? rescuer_thread+0x570/0x570 [ 5272.249140] kthread+0x1d1/0x200 [ 5272.250264] ? set_kthread_struct+0x70/0x70 [ 5272.256284] ret_from_fork+0x1f/0x30 [ 5284.055638] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 5284.058089] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5284.066644] Modules linked in: [ 5284.070094] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5284.070183] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5284.070191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5284.070193] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5284.070205] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5284.070209] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5284.070214] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5284.070216] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5284.070219] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5284.070221] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5284.070223] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5284.070226] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5284.070229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5284.070232] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5284.070241] Call Trace: [ 5284.070245] [ 5284.070251] ? show_regs.cold.9+0x22/0x2f [ 5284.070259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5284.070265] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5284.070269] ? watchdog+0x40/0x40 [ 5284.070273] ? hrtimer_interrupt+0x128/0x2c0 [ 5284.073588] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5284.073671] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5284.073678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5284.073685] Workqueue: events free_work [libcfs] [ 5284.073740] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5284.073748] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5284.073752] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5284.073757] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5284.073759] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5284.073761] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5284.073764] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5284.073766] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5284.073770] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5284.073772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5284.073775] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5284.073784] Call Trace: [ 5284.073790] [ 5284.073795] ? show_regs.cold.9+0x22/0x2f [ 5284.073817] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5284.073824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5284.073827] ? watchdog+0x40/0x40 [ 5284.073832] ? hrtimer_interrupt+0x128/0x2c0 [ 5284.073836] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5284.073844] ? apic_timer_interrupt+0xf/0x20 [ 5284.073848] [ 5284.073850] ? on_each_cpu+0x51/0x120 [ 5284.073855] ? smp_call_function_many_cond+0x3db/0x540 [ 5284.073859] ? smp_call_function_many_cond+0x3a8/0x540 [ 5284.073862] ? load_new_mm_cr3+0x120/0x120 [ 5284.073868] ? load_new_mm_cr3+0x120/0x120 [ 5284.073872] on_each_cpu+0x51/0x120 [ 5284.073876] flush_tlb_kernel_range+0x93/0xb0 [ 5284.073881] free_unmap_vmap_area+0x63/0x70 [ 5284.073886] remove_vm_area+0xe7/0x110 [ 5284.073889] __vunmap+0x9c/0x400 [ 5284.073893] vfree+0x4d/0x100 [ 5284.073897] free_work+0x34/0x50 [libcfs] [ 5284.073922] process_one_work+0x2c8/0x700 [ 5284.073929] worker_thread+0x296/0x6e0 [ 5284.073933] ? rescuer_thread+0x570/0x570 [ 5284.073937] kthread+0x1d1/0x200 [ 5284.073942] ? set_kthread_struct+0x70/0x70 [ 5284.073946] ret_from_fork+0x1f/0x30 [ 5284.409425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5284.413022] ? apic_timer_interrupt+0xf/0x20 [ 5284.417637] [ 5284.418532] ? on_each_cpu_cond_mask+0x46/0x150 [ 5284.424561] ? __x64_sys_fspick+0x210/0x210 [ 5284.426898] ? smp_call_function_many_cond+0x3d9/0x540 [ 5284.431241] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5284.434224] ? __x64_sys_fspick+0x210/0x210 [ 5284.437133] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5284.439438] on_each_cpu_cond_mask+0x46/0x150 [ 5284.442778] on_each_cpu_cond+0x1c/0x30 [ 5284.446035] invalidate_bh_lrus+0x27/0x30 [ 5284.448809] invalidate_bdev+0x5a/0x90 [ 5284.452292] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5284.455213] generic_shutdown_super+0xb1/0x1b0 [ 5284.457851] kill_block_super+0x29/0x80 [ 5284.464774] deactivate_locked_super+0x52/0xd0 [ 5284.470950] deactivate_super+0x83/0x90 [ 5284.472526] cleanup_mnt+0x5f/0xe0 [ 5284.488796] __cleanup_mnt+0x16/0x20 [ 5284.490305] task_work_run+0xc6/0x110 [ 5284.496390] exit_to_usermode_loop+0x1dd/0x1f0 [ 5284.502682] do_syscall_64+0x3d6/0x440 [ 5284.504833] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5284.506910] RIP: 0033:0x7fbb019648fb [ 5284.517302] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5284.540505] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5284.550818] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5284.556468] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5284.560253] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5284.564873] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5284.568093] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5287.392064] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2929s! [ 5287.400932] Showing busy workqueues and worker pools: [ 5287.407415] workqueue events: flags=0x0 [ 5287.408911] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5287.416044] in-flight: 56293:free_work [libcfs] [ 5287.419011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5287.423880] in-flight: 22277:free_work [libcfs] [ 5287.428250] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5287.437457] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5287.441371] pending: nfs4_renew_state [nfsv4] [ 5287.444471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5287.450919] in-flight: 47596:free_work [libcfs] [ 5287.454826] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5287.460328] workqueue events_unbound: flags=0x2 [ 5287.464387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5287.470301] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5287.476282] workqueue events_power_efficient: flags=0x80 [ 5287.479351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5287.487622] pending: neigh_periodic_work, do_cache_clean [ 5287.492933] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5287.496751] pending: check_lifetime [ 5287.499581] workqueue mm_percpu_wq: flags=0x8 [ 5287.501851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5287.505073] pending: vmstat_update [ 5287.507501] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5287.511518] pending: vmstat_update [ 5287.513375] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5287.520158] pending: vmstat_update [ 5287.521931] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5287.527164] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5287.531038] pending: cgroup_pidlist_destroy_work_fn [ 5287.533099] workqueue writeback: flags=0x4a [ 5287.536276] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5287.542072] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5287.546973] workqueue xprtiod: flags=0xa [ 5287.550871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5287.558360] pending: xprt_autoclose [ 5287.561695] workqueue ipv6_addrconf: flags=0x40008 [ 5287.567176] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5287.576504] pending: addrconf_verify_work [ 5287.578265] workqueue knbd0-recv: flags=0x1a [ 5287.580545] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5287.584737] in-flight: 655:recv_work [ 5287.587757] workqueue ptlrpc_pinger: flags=0x2 [ 5287.590633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5287.593805] pending: ptlrpc_pinger_main [ptlrpc] [ 5287.597126] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2929s workers=5 idle: 53684 27131 54850 46584 [ 5287.601547] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2929s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5287.607342] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2929s workers=6 idle: 53660 37674 46417 53664 46418 [ 5287.612270] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5277s workers=2 idle: 657 [ 5297.817474] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 5297.821262] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 5297.832638] (detected by 2, t=2940087 jiffies, g=626849, q=1398) [ 5297.835245] Sending NMI from CPU 2 to CPUs 3: [ 5297.837312] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 5297.837322] rcu: rcu_sched kthread starved for 2939729 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5297.849652] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5297.859134] rcu: RCU grace-period kthread stack dump: [ 5297.862216] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5297.868194] Call Trace: [ 5297.869492] __schedule+0x351/0xcb0 [ 5297.871462] schedule+0xc0/0x180 [ 5297.873089] schedule_timeout+0xb4/0x190 [ 5297.883297] ? do_raw_spin_unlock+0x75/0x190 [ 5297.887357] ? __next_timer_interrupt+0x160/0x160 [ 5297.889226] ? prepare_to_swait_event+0xf3/0x140 [ 5297.890392] rcu_gp_kthread+0x8dd/0x15b0 [ 5297.897689] ? force_qs_rnp+0x2c0/0x2c0 [ 5297.903126] kthread+0x1d1/0x200 [ 5297.905969] ? set_kthread_struct+0x70/0x70 [ 5297.907649] ret_from_fork+0x1f/0x30 [ 5297.912018] rcu: Stack dump where RCU GP kthread last ran: [ 5297.914201] NMI backtrace for cpu 2 [ 5297.915508] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5297.922026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5297.928131] Workqueue: events free_work [libcfs] [ 5297.931795] Call Trace: [ 5297.932790] [ 5297.937766] ? dump_stack+0xbb/0x10e [ 5297.939674] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5297.947313] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5297.953850] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5297.955903] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5297.959808] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5297.964123] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5297.969577] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 5297.973950] ? update_process_times+0x6d/0xc0 [ 5297.975679] ? tick_sched_handle.isra.18+0x27/0x90 [ 5297.983084] ? tick_sched_timer+0x53/0xb0 [ 5297.985418] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5297.987248] ? tick_sched_handle.isra.18+0x90/0x90 [ 5297.990745] ? hrtimer_interrupt+0x128/0x2c0 [ 5297.996430] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5298.001701] ? apic_timer_interrupt+0xf/0x20 [ 5298.005411] [ 5298.006852] ? on_each_cpu+0x51/0x120 [ 5298.009497] ? smp_call_function_many_cond+0x3db/0x540 [ 5298.012513] ? smp_call_function_many_cond+0x3a8/0x540 [ 5298.017668] ? load_new_mm_cr3+0x120/0x120 [ 5298.022726] ? load_new_mm_cr3+0x120/0x120 [ 5298.024367] ? on_each_cpu+0x51/0x120 [ 5298.025787] ? flush_tlb_kernel_range+0x93/0xb0 [ 5298.036931] ? free_unmap_vmap_area+0x63/0x70 [ 5298.042655] ? remove_vm_area+0xe7/0x110 [ 5298.044218] ? __vunmap+0x9c/0x400 [ 5298.047764] ? vfree+0x4d/0x100 [ 5298.051043] ? free_work+0x34/0x50 [libcfs] [ 5298.055102] ? process_one_work+0x2c8/0x700 [ 5298.059756] ? worker_thread+0x296/0x6e0 [ 5298.062857] ? rescuer_thread+0x570/0x570 [ 5298.064438] ? kthread+0x1d1/0x200 [ 5298.069417] ? set_kthread_struct+0x70/0x70 [ 5298.071064] ? ret_from_fork+0x1f/0x30 [ 5312.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 5312.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5312.058386] Modules linked in: zfs(O) [ 5312.063031] Modules linked in: [ 5312.063034] spl(O) lustre(O) osp(O) [ 5312.064260] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) [ 5312.065792] ofd(O) [ 5312.066856] ptlrpc(O) [ 5312.073179] lod(O) [ 5312.074300] obdclass(O) [ 5312.075044] ost(O) [ 5312.075960] ksocklnd(O) [ 5312.076884] mdt(O) [ 5312.077584] lnet(O) [ 5312.078272] mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) [ 5312.078971] libcfs(O) [ 5312.079736] obdecho(O) [ 5312.081354] dm_flakey [ 5312.082275] mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 5312.083023] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5312.083061] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5312.083067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5312.083070] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5312.083082] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5312.083086] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5312.083090] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5312.083092] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5312.083094] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5312.083096] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5312.083099] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5312.083102] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5312.083104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5312.083107] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5312.083115] Call Trace: [ 5312.083120] [ 5312.083124] ? show_regs.cold.9+0x22/0x2f [ 5312.083134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5312.083140] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5312.083143] ? watchdog+0x40/0x40 [ 5312.083868] lnet(O) [ 5312.089702] ? hrtimer_interrupt+0x128/0x2c0 [ 5312.089722] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5312.089731] ? apic_timer_interrupt+0xf/0x20 [ 5312.089736] [ 5312.089740] ? on_each_cpu_cond_mask+0x46/0x150 [ 5312.089745] ? __x64_sys_fspick+0x210/0x210 [ 5312.089753] ? smp_call_function_many_cond+0x3d9/0x540 [ 5312.089757] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5312.089761] ? __x64_sys_fspick+0x210/0x210 [ 5312.089766] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5312.089769] on_each_cpu_cond_mask+0x46/0x150 [ 5312.089779] on_each_cpu_cond+0x1c/0x30 [ 5312.089783] invalidate_bh_lrus+0x27/0x30 [ 5312.089787] invalidate_bdev+0x5a/0x90 [ 5312.099143] libcfs(O) [ 5312.104201] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5312.104327] generic_shutdown_super+0xb1/0x1b0 [ 5312.104337] kill_block_super+0x29/0x80 [ 5312.104342] deactivate_locked_super+0x52/0xd0 [ 5312.104349] deactivate_super+0x83/0x90 [ 5312.104354] cleanup_mnt+0x5f/0xe0 [ 5312.104360] __cleanup_mnt+0x16/0x20 [ 5312.104364] task_work_run+0xc6/0x110 [ 5312.104372] exit_to_usermode_loop+0x1dd/0x1f0 [ 5312.104379] do_syscall_64+0x3d6/0x440 [ 5312.104384] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5312.104392] RIP: 0033:0x7fbb019648fb [ 5312.104399] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5312.104403] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5312.104409] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5312.104411] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5312.104413] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5312.104415] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5312.104417] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5312.250069] dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5312.269223] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5312.278985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5312.283441] Workqueue: events free_work [libcfs] [ 5312.285274] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5312.292765] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5312.301760] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5312.305155] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5312.309388] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5312.312371] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5312.317182] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5312.321035] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5312.326037] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5312.332132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5312.333932] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5312.340737] Call Trace: [ 5312.342767] [ 5312.343800] ? show_regs.cold.9+0x22/0x2f [ 5312.346546] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5312.349424] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5312.353228] ? watchdog+0x40/0x40 [ 5312.361241] ? hrtimer_interrupt+0x128/0x2c0 [ 5312.362971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5312.367965] ? apic_timer_interrupt+0xf/0x20 [ 5312.372671] [ 5312.373531] ? on_each_cpu+0x51/0x120 [ 5312.378823] ? smp_call_function_many_cond+0x3d9/0x540 [ 5312.382333] ? smp_call_function_many_cond+0x3a8/0x540 [ 5312.387907] ? load_new_mm_cr3+0x120/0x120 [ 5312.391951] ? load_new_mm_cr3+0x120/0x120 [ 5312.400028] on_each_cpu+0x51/0x120 [ 5312.401424] flush_tlb_kernel_range+0x93/0xb0 [ 5312.405529] free_unmap_vmap_area+0x63/0x70 [ 5312.407194] remove_vm_area+0xe7/0x110 [ 5312.413004] __vunmap+0x9c/0x400 [ 5312.415404] vfree+0x4d/0x100 [ 5312.416520] free_work+0x34/0x50 [libcfs] [ 5312.420991] process_one_work+0x2c8/0x700 [ 5312.425129] worker_thread+0x296/0x6e0 [ 5312.427182] ? rescuer_thread+0x570/0x570 [ 5312.429241] kthread+0x1d1/0x200 [ 5312.432086] ? set_kthread_struct+0x70/0x70 [ 5312.433854] ret_from_fork+0x1f/0x30 [ 5318.112032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2959s! [ 5318.118068] Showing busy workqueues and worker pools: [ 5318.121544] workqueue events: flags=0x0 [ 5318.124807] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5318.131528] in-flight: 56293:free_work [libcfs] [ 5318.135106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5318.139820] in-flight: 22277:free_work [libcfs] [ 5318.142034] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5318.147664] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5318.150992] pending: nfs4_renew_state [nfsv4] [ 5318.153806] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5318.157465] in-flight: 47596:free_work [libcfs] [ 5318.160777] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5318.166329] workqueue events_unbound: flags=0x2 [ 5318.169769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5318.173981] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5318.178435] workqueue events_power_efficient: flags=0x80 [ 5318.180461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5318.186545] pending: neigh_periodic_work, do_cache_clean [ 5318.194256] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5318.197652] pending: check_lifetime [ 5318.199435] workqueue mm_percpu_wq: flags=0x8 [ 5318.201238] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5318.204223] pending: vmstat_update [ 5318.205966] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5318.209640] pending: vmstat_update [ 5318.211551] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5318.215445] pending: vmstat_update [ 5318.216991] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5318.219713] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5318.222578] pending: cgroup_pidlist_destroy_work_fn [ 5318.224638] workqueue writeback: flags=0x4a [ 5318.226225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5318.229189] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5318.233075] workqueue xprtiod: flags=0xa [ 5318.234692] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5318.236492] pending: xprt_autoclose [ 5318.237626] workqueue ipv6_addrconf: flags=0x40008 [ 5318.238973] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5318.240734] pending: addrconf_verify_work [ 5318.241846] workqueue knbd0-recv: flags=0x1a [ 5318.242950] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5318.244688] in-flight: 655:recv_work [ 5318.245845] workqueue ptlrpc_pinger: flags=0x2 [ 5318.247014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5318.248768] pending: ptlrpc_pinger_main [ptlrpc] [ 5318.250304] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2960s workers=5 idle: 53684 27131 54850 46584 [ 5318.252704] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2960s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5318.255525] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2960s workers=6 idle: 53660 37674 46417 53664 46418 [ 5318.258129] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5308s workers=2 idle: 657 [ 5324.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5324.063900] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5324.096074] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5324.104879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5324.111009] Workqueue: events free_work [libcfs] [ 5324.113195] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5324.115762] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5324.124969] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5324.132117] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5324.135794] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5324.139766] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5324.143694] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5324.147069] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5324.150829] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5324.156730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5324.163906] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5324.167748] Call Trace: [ 5324.168828] [ 5324.169786] ? show_regs.cold.9+0x22/0x2f [ 5324.171599] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5324.174941] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5324.177326] ? watchdog+0x40/0x40 [ 5324.178816] ? hrtimer_interrupt+0x128/0x2c0 [ 5324.182400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5324.185466] ? apic_timer_interrupt+0xf/0x20 [ 5324.188348] [ 5324.190075] ? on_each_cpu+0x51/0x120 [ 5324.192796] ? smp_call_function_many_cond+0x3d9/0x540 [ 5324.195400] ? smp_call_function_many_cond+0x3a8/0x540 [ 5324.200736] ? load_new_mm_cr3+0x120/0x120 [ 5324.203146] ? load_new_mm_cr3+0x120/0x120 [ 5324.206134] on_each_cpu+0x51/0x120 [ 5324.208805] flush_tlb_kernel_range+0x93/0xb0 [ 5324.211244] free_unmap_vmap_area+0x63/0x70 [ 5324.213193] remove_vm_area+0xe7/0x110 [ 5324.214793] __vunmap+0x9c/0x400 [ 5324.216090] vfree+0x4d/0x100 [ 5324.218668] free_work+0x34/0x50 [libcfs] [ 5324.220273] process_one_work+0x2c8/0x700 [ 5324.222155] worker_thread+0x296/0x6e0 [ 5324.223924] ? rescuer_thread+0x570/0x570 [ 5324.228162] kthread+0x1d1/0x200 [ 5324.230059] ? set_kthread_struct+0x70/0x70 [ 5324.234831] ret_from_fork+0x1f/0x30 [ 5340.058689] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5340.061787] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5340.103436] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5340.111788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5340.118424] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5340.121325] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5340.132895] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5340.139650] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5340.152251] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5340.160449] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5340.164313] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5340.168592] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5340.175088] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5340.181243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5340.185161] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5340.189692] Call Trace: [ 5340.191230] [ 5340.191812] ? show_regs.cold.9+0x22/0x2f [ 5340.194632] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5340.198531] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5340.200270] ? watchdog+0x40/0x40 [ 5340.202480] ? hrtimer_interrupt+0x128/0x2c0 [ 5340.206305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5340.210254] ? apic_timer_interrupt+0xf/0x20 [ 5340.212533] [ 5340.213282] ? on_each_cpu_cond_mask+0x46/0x150 [ 5340.214735] ? __x64_sys_fspick+0x210/0x210 [ 5340.216284] ? smp_call_function_many_cond+0x3d9/0x540 [ 5340.218489] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5340.220249] ? __x64_sys_fspick+0x210/0x210 [ 5340.225243] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5340.226532] on_each_cpu_cond_mask+0x46/0x150 [ 5340.232897] on_each_cpu_cond+0x1c/0x30 [ 5340.234060] invalidate_bh_lrus+0x27/0x30 [ 5340.235862] invalidate_bdev+0x5a/0x90 [ 5340.237691] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5340.242235] generic_shutdown_super+0xb1/0x1b0 [ 5340.243796] kill_block_super+0x29/0x80 [ 5340.249470] deactivate_locked_super+0x52/0xd0 [ 5340.254469] deactivate_super+0x83/0x90 [ 5340.256069] cleanup_mnt+0x5f/0xe0 [ 5340.257399] __cleanup_mnt+0x16/0x20 [ 5340.261544] task_work_run+0xc6/0x110 [ 5340.265040] exit_to_usermode_loop+0x1dd/0x1f0 [ 5340.269087] do_syscall_64+0x3d6/0x440 [ 5340.272487] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5340.277205] RIP: 0033:0x7fbb019648fb [ 5340.279713] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5340.289593] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5340.293828] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5340.298642] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5340.301941] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5340.305951] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5340.309125] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5344.055016] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 5344.060724] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5344.116570] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5344.128046] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5344.132302] Workqueue: events free_work [libcfs] [ 5344.135609] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5344.139796] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5344.173667] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5344.181335] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5344.187634] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5344.195940] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5344.203744] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5344.209111] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5344.212752] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5344.218338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5344.223610] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5344.232769] Call Trace: [ 5344.234280] [ 5344.235050] ? show_regs.cold.9+0x22/0x2f [ 5344.236638] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5344.243295] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5344.247044] ? watchdog+0x40/0x40 [ 5344.254296] ? hrtimer_interrupt+0x128/0x2c0 [ 5344.259118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5344.263278] ? apic_timer_interrupt+0xf/0x20 [ 5344.269791] [ 5344.270957] ? on_each_cpu+0x51/0x120 [ 5344.272245] ? smp_call_function_many_cond+0x3d9/0x540 [ 5344.278963] ? smp_call_function_many_cond+0x3a8/0x540 [ 5344.283441] ? load_new_mm_cr3+0x120/0x120 [ 5344.286491] ? load_new_mm_cr3+0x120/0x120 [ 5344.289198] on_each_cpu+0x51/0x120 [ 5344.296322] flush_tlb_kernel_range+0x93/0xb0 [ 5344.298645] free_unmap_vmap_area+0x63/0x70 [ 5344.303400] remove_vm_area+0xe7/0x110 [ 5344.310308] __vunmap+0x9c/0x400 [ 5344.315162] vfree+0x4d/0x100 [ 5344.317863] free_work+0x34/0x50 [libcfs] [ 5344.319440] process_one_work+0x2c8/0x700 [ 5344.326041] worker_thread+0x296/0x6e0 [ 5344.329134] ? rescuer_thread+0x570/0x570 [ 5344.333084] kthread+0x1d1/0x200 [ 5344.337002] ? set_kthread_struct+0x70/0x70 [ 5344.343090] ret_from_fork+0x1f/0x30 [ 5348.319259] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2990s! [ 5348.323824] Showing busy workqueues and worker pools: [ 5348.325286] workqueue events: flags=0x0 [ 5348.326578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5348.329343] in-flight: 56293:free_work [libcfs] [ 5348.331534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5348.334401] in-flight: 22277:free_work [libcfs] [ 5348.335670] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5348.340896] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5348.343397] pending: nfs4_renew_state [nfsv4] [ 5348.345658] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5348.348099] in-flight: 47596:free_work [libcfs] [ 5348.350811] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5348.355672] workqueue events_unbound: flags=0x2 [ 5348.361884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5348.363607] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5348.371112] workqueue events_power_efficient: flags=0x80 [ 5348.373558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5348.376665] pending: neigh_periodic_work, do_cache_clean [ 5348.379606] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5348.382035] pending: check_lifetime [ 5348.384025] workqueue mm_percpu_wq: flags=0x8 [ 5348.385606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5348.387914] pending: vmstat_update [ 5348.388867] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5348.392127] pending: vmstat_update [ 5348.394552] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5348.397888] pending: vmstat_update [ 5348.399644] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5348.401387] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5348.404671] pending: cgroup_pidlist_destroy_work_fn [ 5348.406872] workqueue writeback: flags=0x4a [ 5348.410197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5348.413899] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5348.417600] workqueue xprtiod: flags=0xa [ 5348.424452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5348.432248] pending: xprt_autoclose [ 5348.434787] workqueue ipv6_addrconf: flags=0x40008 [ 5348.436647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5348.440303] pending: addrconf_verify_work [ 5348.444972] workqueue knbd0-recv: flags=0x1a [ 5348.446662] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5348.451652] in-flight: 655:recv_work [ 5348.454452] workqueue ptlrpc_pinger: flags=0x2 [ 5348.456551] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5348.465664] pending: ptlrpc_pinger_main [ptlrpc] [ 5348.470033] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=2990s workers=5 idle: 53684 27131 54850 46584 [ 5348.478270] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=2990s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5348.488474] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=2990s workers=6 idle: 53660 37674 46417 53664 46418 [ 5348.502223] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5338s workers=2 idle: 657 [ 5352.059071] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5352.065387] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.116794] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.130419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5352.143108] Workqueue: events free_work [libcfs] [ 5352.146191] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5352.151456] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5352.171097] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5352.176080] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5352.186244] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5352.192880] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5352.197877] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5352.201423] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5352.204620] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5352.221090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.227316] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5352.232414] Call Trace: [ 5352.234184] [ 5352.235259] ? show_regs.cold.9+0x22/0x2f [ 5352.237451] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.244366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.254717] ? watchdog+0x40/0x40 [ 5352.261384] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.266925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.273949] ? apic_timer_interrupt+0xf/0x20 [ 5352.278790] [ 5352.282975] ? on_each_cpu+0x51/0x120 [ 5352.290311] ? smp_call_function_many_cond+0x3db/0x540 [ 5352.298194] ? smp_call_function_many_cond+0x3a8/0x540 [ 5352.300125] ? load_new_mm_cr3+0x120/0x120 [ 5352.318240] ? load_new_mm_cr3+0x120/0x120 [ 5352.321916] on_each_cpu+0x51/0x120 [ 5352.326070] flush_tlb_kernel_range+0x93/0xb0 [ 5352.327766] free_unmap_vmap_area+0x63/0x70 [ 5352.335359] remove_vm_area+0xe7/0x110 [ 5352.340317] __vunmap+0x9c/0x400 [ 5352.344075] vfree+0x4d/0x100 [ 5352.347902] free_work+0x34/0x50 [libcfs] [ 5352.349767] process_one_work+0x2c8/0x700 [ 5352.354329] worker_thread+0x296/0x6e0 [ 5352.357298] ? rescuer_thread+0x570/0x570 [ 5352.361405] kthread+0x1d1/0x200 [ 5352.364235] ? set_kthread_struct+0x70/0x70 [ 5352.368343] ret_from_fork+0x1f/0x30 [ 5368.058009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5368.061679] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5368.089097] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5368.096618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5368.100067] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5368.102772] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5368.111251] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5368.114723] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5368.117857] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5368.120916] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5368.124035] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5368.127314] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5368.130649] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5368.134156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5368.136667] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5368.139437] Call Trace: [ 5368.140764] [ 5368.141484] ? show_regs.cold.9+0x22/0x2f [ 5368.144242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5368.147937] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5368.150164] ? watchdog+0x40/0x40 [ 5368.153395] ? hrtimer_interrupt+0x128/0x2c0 [ 5368.156786] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5368.160100] ? apic_timer_interrupt+0xf/0x20 [ 5368.162567] [ 5368.164564] ? on_each_cpu_cond_mask+0x46/0x150 [ 5368.166327] ? __x64_sys_fspick+0x210/0x210 [ 5368.168350] ? smp_call_function_many_cond+0x3d9/0x540 [ 5368.172187] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5368.174342] ? __x64_sys_fspick+0x210/0x210 [ 5368.178869] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5368.183855] on_each_cpu_cond_mask+0x46/0x150 [ 5368.187048] on_each_cpu_cond+0x1c/0x30 [ 5368.188892] invalidate_bh_lrus+0x27/0x30 [ 5368.190458] invalidate_bdev+0x5a/0x90 [ 5368.194764] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5368.197870] generic_shutdown_super+0xb1/0x1b0 [ 5368.199908] kill_block_super+0x29/0x80 [ 5368.201904] deactivate_locked_super+0x52/0xd0 [ 5368.203919] deactivate_super+0x83/0x90 [ 5368.205600] cleanup_mnt+0x5f/0xe0 [ 5368.207123] __cleanup_mnt+0x16/0x20 [ 5368.208803] task_work_run+0xc6/0x110 [ 5368.210194] exit_to_usermode_loop+0x1dd/0x1f0 [ 5368.212198] do_syscall_64+0x3d6/0x440 [ 5368.214035] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5368.216360] RIP: 0033:0x7fbb019648fb [ 5368.218040] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5368.226161] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5368.232132] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5368.235875] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5368.240762] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5368.244093] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5368.248193] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5376.055155] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5376.061720] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5376.099241] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5376.107144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5376.116605] Workqueue: events free_work [libcfs] [ 5376.120079] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5376.127841] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5376.140757] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5376.148354] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5376.150960] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5376.155645] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5376.158834] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5376.165511] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5376.168791] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5376.172744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5376.177409] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5376.181599] Call Trace: [ 5376.185662] [ 5376.187254] ? show_regs.cold.9+0x22/0x2f [ 5376.189717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5376.192682] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5376.195563] ? watchdog+0x40/0x40 [ 5376.199744] ? hrtimer_interrupt+0x128/0x2c0 [ 5376.202752] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5376.207586] ? apic_timer_interrupt+0xf/0x20 [ 5376.210819] [ 5376.212521] ? on_each_cpu+0x51/0x120 [ 5376.215949] ? smp_call_function_many_cond+0x3db/0x540 [ 5376.220875] ? smp_call_function_many_cond+0x3a8/0x540 [ 5376.224710] ? load_new_mm_cr3+0x120/0x120 [ 5376.228830] ? load_new_mm_cr3+0x120/0x120 [ 5376.231600] on_each_cpu+0x51/0x120 [ 5376.232779] flush_tlb_kernel_range+0x93/0xb0 [ 5376.234092] free_unmap_vmap_area+0x63/0x70 [ 5376.239698] remove_vm_area+0xe7/0x110 [ 5376.241955] __vunmap+0x9c/0x400 [ 5376.243632] vfree+0x4d/0x100 [ 5376.245210] free_work+0x34/0x50 [libcfs] [ 5376.247205] process_one_work+0x2c8/0x700 [ 5376.248281] worker_thread+0x296/0x6e0 [ 5376.249552] ? rescuer_thread+0x570/0x570 [ 5376.252011] kthread+0x1d1/0x200 [ 5376.255628] ? set_kthread_struct+0x70/0x70 [ 5376.257700] ret_from_fork+0x1f/0x30 [ 5378.527769] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3020s! [ 5378.540375] Showing busy workqueues and worker pools: [ 5378.544187] workqueue events: flags=0x0 [ 5378.546913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5378.554739] in-flight: 56293:free_work [libcfs] [ 5378.557822] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5378.560934] in-flight: 22277:free_work [libcfs] [ 5378.564755] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5378.570799] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5378.577862] pending: nfs4_renew_state [nfsv4] [ 5378.581993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5378.586649] in-flight: 47596:free_work [libcfs] [ 5378.589846] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5378.594927] workqueue events_unbound: flags=0x2 [ 5378.597755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5378.600692] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5378.610636] workqueue events_power_efficient: flags=0x80 [ 5378.618451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5378.622939] pending: neigh_periodic_work, do_cache_clean [ 5378.627902] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5378.633133] pending: check_lifetime [ 5378.636128] workqueue mm_percpu_wq: flags=0x8 [ 5378.639870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5378.645587] pending: vmstat_update [ 5378.650785] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5378.655864] pending: vmstat_update [ 5378.657869] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5378.662370] pending: vmstat_update [ 5378.664241] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5378.666231] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5378.677363] pending: cgroup_pidlist_destroy_work_fn [ 5378.680747] workqueue writeback: flags=0x4a [ 5378.684231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5378.690771] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5378.697703] workqueue xprtiod: flags=0xa [ 5378.700914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5378.705209] pending: xprt_autoclose [ 5378.707572] workqueue ipv6_addrconf: flags=0x40008 [ 5378.710427] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5378.713266] pending: addrconf_verify_work [ 5378.715456] workqueue knbd0-recv: flags=0x1a [ 5378.717389] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5378.721455] in-flight: 655:recv_work [ 5378.724283] workqueue ptlrpc_pinger: flags=0x2 [ 5378.726157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5378.728492] pending: ptlrpc_pinger_main [ptlrpc] [ 5378.730584] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3020s workers=5 idle: 53684 27131 54850 46584 [ 5378.734541] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3020s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5378.739229] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3021s workers=6 idle: 53660 37674 46417 53664 46418 [ 5378.741955] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5368s workers=2 idle: 657 [ 5380.059899] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5380.067140] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.100595] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.106327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5380.110690] Workqueue: events free_work [libcfs] [ 5380.114283] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5380.117846] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5380.129376] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5380.133893] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5380.136757] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5380.141176] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5380.145610] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5380.151885] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5380.156686] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5380.162865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.165967] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5380.171965] Call Trace: [ 5380.175116] [ 5380.176752] ? show_regs.cold.9+0x22/0x2f [ 5380.179341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.181790] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.184920] ? watchdog+0x40/0x40 [ 5380.188924] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.194774] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.199364] ? apic_timer_interrupt+0xf/0x20 [ 5380.204862] [ 5380.205660] ? on_each_cpu+0x51/0x120 [ 5380.208070] ? smp_call_function_many_cond+0x3db/0x540 [ 5380.209864] ? smp_call_function_many_cond+0x3a8/0x540 [ 5380.213391] ? load_new_mm_cr3+0x120/0x120 [ 5380.215471] ? load_new_mm_cr3+0x120/0x120 [ 5380.218542] on_each_cpu+0x51/0x120 [ 5380.220826] flush_tlb_kernel_range+0x93/0xb0 [ 5380.222976] free_unmap_vmap_area+0x63/0x70 [ 5380.225046] remove_vm_area+0xe7/0x110 [ 5380.226336] __vunmap+0x9c/0x400 [ 5380.228418] vfree+0x4d/0x100 [ 5380.230109] free_work+0x34/0x50 [libcfs] [ 5380.232338] process_one_work+0x2c8/0x700 [ 5380.233706] worker_thread+0x296/0x6e0 [ 5380.235156] ? rescuer_thread+0x570/0x570 [ 5380.237646] kthread+0x1d1/0x200 [ 5380.239418] ? set_kthread_struct+0x70/0x70 [ 5380.240917] ret_from_fork+0x1f/0x30 [ 5396.058051] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5396.064694] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5396.102924] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5396.110094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5396.115101] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5396.120299] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5396.132366] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5396.138600] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5396.146247] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5396.152471] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5396.158039] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5396.163184] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5396.169127] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5396.174736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5396.178562] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5396.182352] Call Trace: [ 5396.184656] [ 5396.186800] ? show_regs.cold.9+0x22/0x2f [ 5396.188512] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5396.190410] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5396.196402] ? watchdog+0x40/0x40 [ 5396.200166] ? hrtimer_interrupt+0x128/0x2c0 [ 5396.203241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5396.205847] ? apic_timer_interrupt+0xf/0x20 [ 5396.207559] [ 5396.211283] ? on_each_cpu_cond_mask+0x46/0x150 [ 5396.216666] ? __x64_sys_fspick+0x210/0x210 [ 5396.219711] ? smp_call_function_many_cond+0x3db/0x540 [ 5396.221770] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5396.224744] ? __x64_sys_fspick+0x210/0x210 [ 5396.227535] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5396.231644] on_each_cpu_cond_mask+0x46/0x150 [ 5396.233974] on_each_cpu_cond+0x1c/0x30 [ 5396.237024] invalidate_bh_lrus+0x27/0x30 [ 5396.240504] invalidate_bdev+0x5a/0x90 [ 5396.242420] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5396.246442] generic_shutdown_super+0xb1/0x1b0 [ 5396.249134] kill_block_super+0x29/0x80 [ 5396.250605] deactivate_locked_super+0x52/0xd0 [ 5396.253775] deactivate_super+0x83/0x90 [ 5396.255249] cleanup_mnt+0x5f/0xe0 [ 5396.259240] __cleanup_mnt+0x16/0x20 [ 5396.264504] task_work_run+0xc6/0x110 [ 5396.267004] exit_to_usermode_loop+0x1dd/0x1f0 [ 5396.268823] do_syscall_64+0x3d6/0x440 [ 5396.272171] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5396.279524] RIP: 0033:0x7fbb019648fb [ 5396.282023] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5396.294074] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5396.297952] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5396.302621] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5396.308361] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5396.315935] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5396.320928] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5404.055014] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5404.059494] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5404.092530] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5404.099808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5404.108058] Workqueue: events free_work [libcfs] [ 5404.113590] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5404.119796] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5404.133328] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5404.140929] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5404.146062] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5404.150156] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5404.156042] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5404.161071] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5404.164853] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5404.171688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5404.174906] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5404.180981] Call Trace: [ 5404.183524] [ 5404.185046] ? show_regs.cold.9+0x22/0x2f [ 5404.187390] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5404.190130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5404.194257] ? watchdog+0x40/0x40 [ 5404.197089] ? hrtimer_interrupt+0x128/0x2c0 [ 5404.201319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5404.204784] ? apic_timer_interrupt+0xf/0x20 [ 5404.208349] [ 5404.209218] ? on_each_cpu+0x51/0x120 [ 5404.210421] ? smp_call_function_many_cond+0x3db/0x540 [ 5404.213652] ? smp_call_function_many_cond+0x3a8/0x540 [ 5404.216806] ? load_new_mm_cr3+0x120/0x120 [ 5404.218303] ? load_new_mm_cr3+0x120/0x120 [ 5404.223839] on_each_cpu+0x51/0x120 [ 5404.226108] flush_tlb_kernel_range+0x93/0xb0 [ 5404.229696] free_unmap_vmap_area+0x63/0x70 [ 5404.232450] remove_vm_area+0xe7/0x110 [ 5404.237737] __vunmap+0x9c/0x400 [ 5404.242093] vfree+0x4d/0x100 [ 5404.245597] free_work+0x34/0x50 [libcfs] [ 5404.247150] process_one_work+0x2c8/0x700 [ 5404.250712] worker_thread+0x296/0x6e0 [ 5404.252288] ? rescuer_thread+0x570/0x570 [ 5404.256363] kthread+0x1d1/0x200 [ 5404.261871] ? set_kthread_struct+0x70/0x70 [ 5404.265243] ret_from_fork+0x1f/0x30 [ 5408.059201] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5408.062795] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5408.098786] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5408.105778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5408.111097] Workqueue: events free_work [libcfs] [ 5408.119095] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5408.128634] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5408.143641] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5408.153020] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5408.157745] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5408.162623] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5408.167108] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5408.175161] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5408.181843] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5408.187086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5408.190594] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5408.194099] Call Trace: [ 5408.195776] [ 5408.196730] ? show_regs.cold.9+0x22/0x2f [ 5408.200236] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5408.202044] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5408.204779] ? watchdog+0x40/0x40 [ 5408.207038] ? hrtimer_interrupt+0x128/0x2c0 [ 5408.209260] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5408.211909] ? apic_timer_interrupt+0xf/0x20 [ 5408.215937] [ 5408.217061] ? on_each_cpu+0x51/0x120 [ 5408.219663] ? smp_call_function_many_cond+0x3d9/0x540 [ 5408.223168] ? smp_call_function_many_cond+0x3a8/0x540 [ 5408.226951] ? load_new_mm_cr3+0x120/0x120 [ 5408.229191] ? load_new_mm_cr3+0x120/0x120 [ 5408.231386] on_each_cpu+0x51/0x120 [ 5408.233704] flush_tlb_kernel_range+0x93/0xb0 [ 5408.238254] free_unmap_vmap_area+0x63/0x70 [ 5408.240591] remove_vm_area+0xe7/0x110 [ 5408.242790] __vunmap+0x9c/0x400 [ 5408.244379] vfree+0x4d/0x100 [ 5408.247253] free_work+0x34/0x50 [libcfs] [ 5408.249800] process_one_work+0x2c8/0x700 [ 5408.253438] worker_thread+0x296/0x6e0 [ 5408.256121] ? rescuer_thread+0x570/0x570 [ 5408.259755] kthread+0x1d1/0x200 [ 5408.263660] ? set_kthread_struct+0x70/0x70 [ 5408.268449] ret_from_fork+0x1f/0x30 [ 5409.247027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3050s! [ 5409.255175] Showing busy workqueues and worker pools: [ 5409.258797] workqueue events: flags=0x0 [ 5409.261305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5409.265115] in-flight: 56293:free_work [libcfs] [ 5409.268361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5409.274030] in-flight: 22277:free_work [libcfs] [ 5409.277927] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5409.284961] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5409.290336] pending: nfs4_renew_state [nfsv4] [ 5409.293041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5409.297223] in-flight: 47596:free_work [libcfs] [ 5409.300120] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5409.304833] workqueue events_unbound: flags=0x2 [ 5409.306539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5409.310343] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5409.314041] workqueue events_power_efficient: flags=0x80 [ 5409.316248] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5409.318901] pending: neigh_periodic_work, do_cache_clean [ 5409.321554] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5409.324709] pending: check_lifetime [ 5409.326666] workqueue mm_percpu_wq: flags=0x8 [ 5409.329609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5409.332934] pending: vmstat_update [ 5409.334939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5409.338065] pending: vmstat_update [ 5409.340492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5409.345187] pending: vmstat_update [ 5409.346902] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5409.349217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5409.352091] pending: cgroup_pidlist_destroy_work_fn [ 5409.354511] workqueue writeback: flags=0x4a [ 5409.356184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5409.362094] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5409.367617] workqueue xprtiod: flags=0xa [ 5409.370145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5409.373741] pending: xprt_autoclose [ 5409.375906] workqueue ipv6_addrconf: flags=0x40008 [ 5409.378032] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5409.380987] pending: addrconf_verify_work [ 5409.382636] workqueue knbd0-recv: flags=0x1a [ 5409.384474] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5409.388851] in-flight: 655:recv_work [ 5409.391030] workqueue ptlrpc_pinger: flags=0x2 [ 5409.394781] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5409.400238] pending: ptlrpc_pinger_main [ptlrpc] [ 5409.406052] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3051s workers=5 idle: 53684 27131 54850 46584 [ 5409.413708] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3051s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5409.420087] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3051s workers=6 idle: 53660 37674 46417 53664 46418 [ 5409.424965] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5399s workers=2 idle: 657 [ 5424.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5424.061989] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5424.125911] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5424.139094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5424.143143] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5424.146244] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5424.158042] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5424.166407] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5424.172785] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5424.177855] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5424.182578] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5424.194002] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5424.202779] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5424.207863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5424.214181] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5424.218142] Call Trace: [ 5424.219286] [ 5424.221288] ? show_regs.cold.9+0x22/0x2f [ 5424.223659] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5424.228426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5424.229867] ? watchdog+0x40/0x40 [ 5424.233391] ? hrtimer_interrupt+0x128/0x2c0 [ 5424.235897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5424.238630] ? apic_timer_interrupt+0xf/0x20 [ 5424.241228] [ 5424.251136] ? on_each_cpu_cond_mask+0x46/0x150 [ 5424.254702] ? __x64_sys_fspick+0x210/0x210 [ 5424.258758] ? smp_call_function_many_cond+0x3db/0x540 [ 5424.262372] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5424.270200] ? __x64_sys_fspick+0x210/0x210 [ 5424.273334] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5424.275621] on_each_cpu_cond_mask+0x46/0x150 [ 5424.279192] on_each_cpu_cond+0x1c/0x30 [ 5424.281041] invalidate_bh_lrus+0x27/0x30 [ 5424.283222] invalidate_bdev+0x5a/0x90 [ 5424.289052] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5424.291782] generic_shutdown_super+0xb1/0x1b0 [ 5424.296975] kill_block_super+0x29/0x80 [ 5424.300156] deactivate_locked_super+0x52/0xd0 [ 5424.303916] deactivate_super+0x83/0x90 [ 5424.308446] cleanup_mnt+0x5f/0xe0 [ 5424.310033] __cleanup_mnt+0x16/0x20 [ 5424.318103] task_work_run+0xc6/0x110 [ 5424.323385] exit_to_usermode_loop+0x1dd/0x1f0 [ 5424.325167] do_syscall_64+0x3d6/0x440 [ 5424.331790] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5424.337048] RIP: 0033:0x7fbb019648fb [ 5424.340646] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5424.355380] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5424.360790] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5424.367561] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5424.374592] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5424.384589] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5424.399319] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5436.055021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5436.061128] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5436.063194] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata [ 5436.068607] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5436.069189] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5436.069196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5436.069215] Workqueue: events free_work [libcfs] [ 5436.069256] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5436.069267] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5436.069271] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5436.069275] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5436.069277] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5436.069279] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5436.069282] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5436.069284] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5436.069287] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5436.069290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5436.069292] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5436.069301] Call Trace: [ 5436.069307] [ 5436.069313] ? show_regs.cold.9+0x22/0x2f [ 5436.069321] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5436.069327] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5436.069330] ? watchdog+0x40/0x40 [ 5436.069334] ? hrtimer_interrupt+0x128/0x2c0 [ 5436.069337] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5436.069344] ? apic_timer_interrupt+0xf/0x20 [ 5436.069347] [ 5436.069349] ? on_each_cpu+0x51/0x120 [ 5436.069353] ? smp_call_function_many_cond+0x3db/0x540 [ 5436.069357] ? smp_call_function_many_cond+0x3a8/0x540 [ 5436.069360] ? load_new_mm_cr3+0x120/0x120 [ 5436.069365] ? load_new_mm_cr3+0x120/0x120 [ 5436.069368] on_each_cpu+0x51/0x120 [ 5436.069372] flush_tlb_kernel_range+0x93/0xb0 [ 5436.069377] free_unmap_vmap_area+0x63/0x70 [ 5436.069382] remove_vm_area+0xe7/0x110 [ 5436.069384] __vunmap+0x9c/0x400 [ 5436.069387] vfree+0x4d/0x100 [ 5436.069390] free_work+0x34/0x50 [libcfs] [ 5436.069412] process_one_work+0x2c8/0x700 [ 5436.069418] worker_thread+0x296/0x6e0 [ 5436.069422] ? rescuer_thread+0x570/0x570 [ 5436.069426] kthread+0x1d1/0x200 [ 5436.069429] ? set_kthread_struct+0x70/0x70 [ 5436.069432] ret_from_fork+0x1f/0x30 [ 5436.330029] serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5436.340232] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5436.351727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5436.359424] Workqueue: events free_work [libcfs] [ 5436.368683] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5436.371565] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5436.395513] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5436.400251] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5436.410478] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5436.414129] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5436.418405] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5436.424904] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5436.428971] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5436.433920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5436.445112] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5436.450533] Call Trace: [ 5436.451585] [ 5436.454646] ? show_regs.cold.9+0x22/0x2f [ 5436.458506] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5436.461121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5436.464247] ? watchdog+0x40/0x40 [ 5436.467956] ? hrtimer_interrupt+0x128/0x2c0 [ 5436.470999] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5436.473645] ? apic_timer_interrupt+0xf/0x20 [ 5436.475880] [ 5436.476612] ? on_each_cpu+0x51/0x120 [ 5436.478304] ? smp_call_function_many_cond+0x3db/0x540 [ 5436.480700] ? smp_call_function_many_cond+0x3a8/0x540 [ 5436.483050] ? load_new_mm_cr3+0x120/0x120 [ 5436.484740] ? load_new_mm_cr3+0x120/0x120 [ 5436.485884] on_each_cpu+0x51/0x120 [ 5436.487594] flush_tlb_kernel_range+0x93/0xb0 [ 5436.491070] free_unmap_vmap_area+0x63/0x70 [ 5436.493194] remove_vm_area+0xe7/0x110 [ 5436.494751] __vunmap+0x9c/0x400 [ 5436.500705] vfree+0x4d/0x100 [ 5436.503139] free_work+0x34/0x50 [libcfs] [ 5436.505139] process_one_work+0x2c8/0x700 [ 5436.506978] worker_thread+0x296/0x6e0 [ 5436.513204] ? rescuer_thread+0x570/0x570 [ 5436.515205] kthread+0x1d1/0x200 [ 5436.517022] ? set_kthread_struct+0x70/0x70 [ 5436.519114] ret_from_fork+0x1f/0x30 [ 5439.456076] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3081s! [ 5439.461977] Showing busy workqueues and worker pools: [ 5439.467666] workqueue events: flags=0x0 [ 5439.471229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5439.474948] in-flight: 56293:free_work [libcfs] [ 5439.481119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5439.485397] in-flight: 22277:free_work [libcfs] [ 5439.489398] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5439.500246] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5439.504545] pending: nfs4_renew_state [nfsv4] [ 5439.508364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5439.522345] in-flight: 47596:free_work [libcfs] [ 5439.526236] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5439.538199] workqueue events_unbound: flags=0x2 [ 5439.545347] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5439.549263] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5439.557674] workqueue events_power_efficient: flags=0x80 [ 5439.559868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5439.570489] pending: neigh_periodic_work, do_cache_clean [ 5439.575948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5439.583201] pending: check_lifetime [ 5439.590664] workqueue mm_percpu_wq: flags=0x8 [ 5439.597591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5439.603078] pending: vmstat_update [ 5439.606100] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5439.616157] pending: vmstat_update [ 5439.619250] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5439.623773] pending: vmstat_update [ 5439.629418] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5439.636625] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5439.641010] pending: cgroup_pidlist_destroy_work_fn [ 5439.647594] workqueue writeback: flags=0x4a [ 5439.649208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5439.654102] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5439.658276] workqueue xprtiod: flags=0xa [ 5439.662390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5439.673052] pending: xprt_autoclose [ 5439.682058] workqueue ipv6_addrconf: flags=0x40008 [ 5439.685369] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5439.694025] pending: addrconf_verify_work [ 5439.697207] workqueue knbd0-recv: flags=0x1a [ 5439.698951] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5439.702801] in-flight: 655:recv_work [ 5439.705905] workqueue ptlrpc_pinger: flags=0x2 [ 5439.708309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5439.711155] pending: ptlrpc_pinger_main [ptlrpc] [ 5439.713808] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3081s workers=5 idle: 53684 27131 54850 46584 [ 5439.719056] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3081s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5439.732377] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3082s workers=6 idle: 53660 37674 46417 53664 46418 [ 5439.744173] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5429s workers=2 idle: 657 [ 5452.058330] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5452.063328] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5452.096426] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5452.102430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5452.106172] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5452.112844] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5452.130048] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5452.136362] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5452.142848] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5452.149880] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5452.156495] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5452.161858] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5452.165792] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5452.169357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5452.172438] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5452.175937] Call Trace: [ 5452.177772] [ 5452.179103] ? show_regs.cold.9+0x22/0x2f [ 5452.181047] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5452.184532] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5452.187797] ? watchdog+0x40/0x40 [ 5452.189427] ? hrtimer_interrupt+0x128/0x2c0 [ 5452.191449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5452.193831] ? apic_timer_interrupt+0xf/0x20 [ 5452.195704] [ 5452.196748] ? on_each_cpu_cond_mask+0x46/0x150 [ 5452.198845] ? __x64_sys_fspick+0x210/0x210 [ 5452.200422] ? smp_call_function_many_cond+0x3db/0x540 [ 5452.202849] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5452.204843] ? __x64_sys_fspick+0x210/0x210 [ 5452.207044] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5452.209145] on_each_cpu_cond_mask+0x46/0x150 [ 5452.211147] on_each_cpu_cond+0x1c/0x30 [ 5452.213878] invalidate_bh_lrus+0x27/0x30 [ 5452.216985] invalidate_bdev+0x5a/0x90 [ 5452.218904] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5452.221324] generic_shutdown_super+0xb1/0x1b0 [ 5452.225136] kill_block_super+0x29/0x80 [ 5452.227075] deactivate_locked_super+0x52/0xd0 [ 5452.229901] deactivate_super+0x83/0x90 [ 5452.232097] cleanup_mnt+0x5f/0xe0 [ 5452.233783] __cleanup_mnt+0x16/0x20 [ 5452.235629] task_work_run+0xc6/0x110 [ 5452.237393] exit_to_usermode_loop+0x1dd/0x1f0 [ 5452.239768] do_syscall_64+0x3d6/0x440 [ 5452.241802] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5452.244380] RIP: 0033:0x7fbb019648fb [ 5452.246020] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5452.261415] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5452.268163] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5452.272195] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5452.275958] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5452.278354] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5452.283110] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5464.055040] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5464.058636] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5464.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5464.081777] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5464.087379] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5464.087470] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5464.087478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5464.087485] Workqueue: events free_work [libcfs] [ 5464.093882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5464.093896] Workqueue: events free_work [libcfs] [ 5464.093944] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5464.093954] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5464.093959] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5464.093963] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5464.093965] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5464.093968] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5464.093970] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5464.093972] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5464.093975] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5464.093978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5464.093980] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5464.095273] Call Trace: [ 5464.125320] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5464.125341] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5464.125345] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5464.125351] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5464.125353] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5464.125356] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5464.125358] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5464.125360] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5464.125362] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5464.125365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5464.125367] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5464.125376] Call Trace: [ 5464.125382] [ 5464.125388] ? show_regs.cold.9+0x22/0x2f [ 5464.125398] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5464.125415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5464.125419] ? watchdog+0x40/0x40 [ 5464.125424] ? hrtimer_interrupt+0x128/0x2c0 [ 5464.125427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5464.125434] ? apic_timer_interrupt+0xf/0x20 [ 5464.125439] [ 5464.125441] ? on_each_cpu+0x51/0x120 [ 5464.125445] ? smp_call_function_many_cond+0x3db/0x540 [ 5464.125448] ? smp_call_function_many_cond+0x3a8/0x540 [ 5464.125452] ? load_new_mm_cr3+0x120/0x120 [ 5464.125457] ? load_new_mm_cr3+0x120/0x120 [ 5464.125461] on_each_cpu+0x51/0x120 [ 5464.125465] flush_tlb_kernel_range+0x93/0xb0 [ 5464.125470] free_unmap_vmap_area+0x63/0x70 [ 5464.125475] remove_vm_area+0xe7/0x110 [ 5464.125479] __vunmap+0x9c/0x400 [ 5464.125482] vfree+0x4d/0x100 [ 5464.125488] free_work+0x34/0x50 [libcfs] [ 5464.125527] process_one_work+0x2c8/0x700 [ 5464.125535] worker_thread+0x296/0x6e0 [ 5464.125539] ? rescuer_thread+0x570/0x570 [ 5464.125543] kthread+0x1d1/0x200 [ 5464.125548] ? set_kthread_struct+0x70/0x70 [ 5464.125552] ret_from_fork+0x1f/0x30 [ 5464.274087] [ 5464.275130] ? show_regs.cold.9+0x22/0x2f [ 5464.277296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5464.279468] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5464.281482] ? watchdog+0x40/0x40 [ 5464.283495] ? hrtimer_interrupt+0x128/0x2c0 [ 5464.285012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5464.287488] ? apic_timer_interrupt+0xf/0x20 [ 5464.289545] [ 5464.290806] ? on_each_cpu+0x51/0x120 [ 5464.292494] ? smp_call_function_many_cond+0x3db/0x540 [ 5464.295314] ? smp_call_function_many_cond+0x3a8/0x540 [ 5464.297925] ? load_new_mm_cr3+0x120/0x120 [ 5464.300013] ? load_new_mm_cr3+0x120/0x120 [ 5464.302035] on_each_cpu+0x51/0x120 [ 5464.303836] flush_tlb_kernel_range+0x93/0xb0 [ 5464.305994] free_unmap_vmap_area+0x63/0x70 [ 5464.308057] remove_vm_area+0xe7/0x110 [ 5464.309978] __vunmap+0x9c/0x400 [ 5464.311599] vfree+0x4d/0x100 [ 5464.312676] free_work+0x34/0x50 [libcfs] [ 5464.314899] process_one_work+0x2c8/0x700 [ 5464.317003] worker_thread+0x296/0x6e0 [ 5464.320092] ? rescuer_thread+0x570/0x570 [ 5464.321605] kthread+0x1d1/0x200 [ 5464.328058] ? set_kthread_struct+0x70/0x70 [ 5464.329633] ret_from_fork+0x1f/0x30 [ 5470.176080] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3111s! [ 5470.180963] Showing busy workqueues and worker pools: [ 5470.185713] workqueue events: flags=0x0 [ 5470.189755] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5470.196829] in-flight: 56293:free_work [libcfs] [ 5470.200681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5470.206466] in-flight: 22277:free_work [libcfs] [ 5470.209335] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5470.213824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5470.220327] pending: nfs4_renew_state [nfsv4] [ 5470.223922] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5470.229039] in-flight: 47596:free_work [libcfs] [ 5470.232380] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5470.239475] workqueue events_unbound: flags=0x2 [ 5470.244407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5470.249555] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5470.252732] workqueue events_power_efficient: flags=0x80 [ 5470.256522] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5470.259618] pending: neigh_periodic_work, do_cache_clean [ 5470.261806] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5470.264855] pending: check_lifetime [ 5470.266950] workqueue mm_percpu_wq: flags=0x8 [ 5470.269425] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5470.272344] pending: vmstat_update [ 5470.273861] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5470.276412] pending: vmstat_update [ 5470.278011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5470.281656] pending: vmstat_update [ 5470.283388] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5470.285525] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5470.288470] pending: cgroup_pidlist_destroy_work_fn [ 5470.290671] workqueue writeback: flags=0x4a [ 5470.292605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5470.295357] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5470.298252] workqueue xprtiod: flags=0xa [ 5470.300174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5470.302724] pending: xprt_autoclose [ 5470.304562] workqueue ipv6_addrconf: flags=0x40008 [ 5470.306643] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5470.309442] pending: addrconf_verify_work [ 5470.311388] workqueue knbd0-recv: flags=0x1a [ 5470.313138] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5470.315117] in-flight: 655:recv_work [ 5470.316146] workqueue ptlrpc_pinger: flags=0x2 [ 5470.317520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5470.320027] pending: ptlrpc_pinger_main [ptlrpc] [ 5470.322503] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3112s workers=5 idle: 53684 27131 54850 46584 [ 5470.326023] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3112s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5470.330533] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3112s workers=6 idle: 53660 37674 46417 53664 46418 [ 5470.335804] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5460s workers=2 idle: 657 [ 5477.823891] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 5477.827707] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 5477.834327] (detected by 2, t=3120093 jiffies, g=626849, q=1398) [ 5477.837118] Sending NMI from CPU 2 to CPUs 3: [ 5477.839988] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 5477.839998] rcu: rcu_sched kthread starved for 3119735 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5477.849619] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5477.856825] rcu: RCU grace-period kthread stack dump: [ 5477.865525] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5477.870410] Call Trace: [ 5477.871737] __schedule+0x351/0xcb0 [ 5477.873333] schedule+0xc0/0x180 [ 5477.874569] schedule_timeout+0xb4/0x190 [ 5477.876473] ? do_raw_spin_unlock+0x75/0x190 [ 5477.879181] ? __next_timer_interrupt+0x160/0x160 [ 5477.885436] ? prepare_to_swait_event+0xf3/0x140 [ 5477.892129] rcu_gp_kthread+0x8dd/0x15b0 [ 5477.893876] ? force_qs_rnp+0x2c0/0x2c0 [ 5477.897696] kthread+0x1d1/0x200 [ 5477.899171] ? set_kthread_struct+0x70/0x70 [ 5477.901100] ret_from_fork+0x1f/0x30 [ 5477.902391] rcu: Stack dump where RCU GP kthread last ran: [ 5477.912201] Sending NMI from CPU 2 to CPUs 0: [ 5477.919586] NMI backtrace for cpu 0 [ 5477.919592] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5477.919594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5477.919596] Workqueue: events free_work [libcfs] [ 5477.919599] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5477.919600] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5477.919602] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 [ 5477.919604] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5477.919605] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5477.919606] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5477.919608] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5477.919609] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5477.919610] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5477.919611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5477.919613] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5477.919614] Call Trace: [ 5477.919615] [ 5477.919616] ? show_regs.cold.9+0x22/0x2f [ 5477.919616] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5477.919617] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5477.919618] ? nmi_handle+0xbd/0x210 [ 5477.919619] ? default_do_nmi+0x4b/0x1b0 [ 5477.919621] ? do_nmi+0x280/0x380 [ 5477.919622] ? end_repeat_nmi+0x16/0x1f [ 5477.919623] ? on_each_cpu+0x51/0x120 [ 5477.919624] ? smp_call_function_many_cond+0x3db/0x540 [ 5477.919625] ? smp_call_function_many_cond+0x3db/0x540 [ 5477.919626] ? smp_call_function_many_cond+0x3db/0x540 [ 5477.919627] [ 5477.919628] ? load_new_mm_cr3+0x120/0x120 [ 5477.919629] ? load_new_mm_cr3+0x120/0x120 [ 5477.919630] on_each_cpu+0x51/0x120 [ 5477.919631] flush_tlb_kernel_range+0x93/0xb0 [ 5477.919632] free_unmap_vmap_area+0x63/0x70 [ 5477.919633] remove_vm_area+0xe7/0x110 [ 5477.919634] __vunmap+0x9c/0x400 [ 5477.919635] vfree+0x4d/0x100 [ 5477.919636] free_work+0x34/0x50 [libcfs] [ 5477.919637] process_one_work+0x2c8/0x700 [ 5477.919638] worker_thread+0x296/0x6e0 [ 5477.919639] ? rescuer_thread+0x570/0x570 [ 5477.919640] kthread+0x1d1/0x200 [ 5477.919641] ? set_kthread_struct+0x70/0x70 [ 5477.919642] ret_from_fork+0x1f/0x30 [ 5480.062846] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5480.069727] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5480.108740] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5480.116057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5480.124656] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5480.128802] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5480.141913] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5480.146384] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5480.150162] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5480.154703] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5480.163531] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5480.171235] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5480.176032] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5480.183499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5480.193979] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5480.202154] Call Trace: [ 5480.206041] [ 5480.208640] ? show_regs.cold.9+0x22/0x2f [ 5480.214932] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5480.219951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5480.237534] ? watchdog+0x40/0x40 [ 5480.239068] ? hrtimer_interrupt+0x128/0x2c0 [ 5480.241341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5480.245764] ? apic_timer_interrupt+0xf/0x20 [ 5480.247436] [ 5480.248249] ? on_each_cpu_cond_mask+0x46/0x150 [ 5480.253996] ? __x64_sys_fspick+0x210/0x210 [ 5480.256341] ? smp_call_function_many_cond+0x3db/0x540 [ 5480.258275] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5480.261029] ? __x64_sys_fspick+0x210/0x210 [ 5480.262638] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5480.268956] on_each_cpu_cond_mask+0x46/0x150 [ 5480.272097] on_each_cpu_cond+0x1c/0x30 [ 5480.274525] invalidate_bh_lrus+0x27/0x30 [ 5480.281090] invalidate_bdev+0x5a/0x90 [ 5480.282826] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5480.287005] generic_shutdown_super+0xb1/0x1b0 [ 5480.289197] kill_block_super+0x29/0x80 [ 5480.291732] deactivate_locked_super+0x52/0xd0 [ 5480.294609] deactivate_super+0x83/0x90 [ 5480.296085] cleanup_mnt+0x5f/0xe0 [ 5480.300075] __cleanup_mnt+0x16/0x20 [ 5480.302400] task_work_run+0xc6/0x110 [ 5480.304334] exit_to_usermode_loop+0x1dd/0x1f0 [ 5480.306834] do_syscall_64+0x3d6/0x440 [ 5480.308180] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5480.314113] RIP: 0033:0x7fbb019648fb [ 5480.318936] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5480.335676] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5480.342319] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5480.345026] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5480.354735] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5480.357411] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5480.361758] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5500.384029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3142s! [ 5500.388065] Showing busy workqueues and worker pools: [ 5500.390860] workqueue events: flags=0x0 [ 5500.393080] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5500.397082] in-flight: 56293:free_work [libcfs] [ 5500.400226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5500.404104] in-flight: 22277:free_work [libcfs] [ 5500.407350] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5500.412949] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5500.417891] pending: nfs4_renew_state [nfsv4] [ 5500.421806] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5500.425800] in-flight: 47596:free_work [libcfs] [ 5500.427605] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5500.432770] workqueue events_unbound: flags=0x2 [ 5500.436306] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5500.441087] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5500.447025] workqueue events_power_efficient: flags=0x80 [ 5500.451373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5500.455287] pending: neigh_periodic_work, do_cache_clean [ 5500.460876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5500.464298] pending: check_lifetime [ 5500.466170] workqueue mm_percpu_wq: flags=0x8 [ 5500.468806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5500.472734] pending: vmstat_update [ 5500.475351] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5500.478824] pending: vmstat_update [ 5500.480885] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5500.484488] pending: vmstat_update [ 5500.485967] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5500.488356] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5500.494674] pending: cgroup_pidlist_destroy_work_fn [ 5500.497895] workqueue writeback: flags=0x4a [ 5500.500304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5500.503863] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5500.506941] workqueue xprtiod: flags=0xa [ 5500.508564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5500.514686] pending: xprt_autoclose [ 5500.516653] workqueue ipv6_addrconf: flags=0x40008 [ 5500.518512] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5500.522823] pending: addrconf_verify_work [ 5500.528339] workqueue knbd0-recv: flags=0x1a [ 5500.530244] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5500.535147] in-flight: 655:recv_work [ 5500.537679] workqueue ptlrpc_pinger: flags=0x2 [ 5500.540474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5500.544366] pending: ptlrpc_pinger_main [ptlrpc] [ 5500.549897] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3142s workers=5 idle: 53684 27131 54850 46584 [ 5500.555692] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3142s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5500.562014] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3142s workers=6 idle: 53660 37674 46417 53664 46418 [ 5500.566708] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5490s workers=2 idle: 657 [ 5504.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 5504.061574] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5504.081841] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5504.087444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5504.092476] Workqueue: events free_work [libcfs] [ 5504.094769] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5504.098770] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5504.107525] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5504.112386] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5504.116034] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5504.119760] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5504.127250] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5504.131241] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5504.135853] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5504.141249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5504.143742] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5504.149266] Call Trace: [ 5504.151789] [ 5504.154144] ? show_regs.cold.9+0x22/0x2f [ 5504.158090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5504.165706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5504.167526] ? watchdog+0x40/0x40 [ 5504.171427] ? hrtimer_interrupt+0x128/0x2c0 [ 5504.173507] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5504.175801] ? apic_timer_interrupt+0xf/0x20 [ 5504.178959] [ 5504.180105] ? on_each_cpu+0x51/0x120 [ 5504.181600] ? smp_call_function_many_cond+0x3d9/0x540 [ 5504.185619] ? smp_call_function_many_cond+0x3a8/0x540 [ 5504.190259] ? load_new_mm_cr3+0x120/0x120 [ 5504.193717] ? load_new_mm_cr3+0x120/0x120 [ 5504.195787] on_each_cpu+0x51/0x120 [ 5504.196951] flush_tlb_kernel_range+0x93/0xb0 [ 5504.198263] free_unmap_vmap_area+0x63/0x70 [ 5504.200105] remove_vm_area+0xe7/0x110 [ 5504.201611] __vunmap+0x9c/0x400 [ 5504.202929] vfree+0x4d/0x100 [ 5504.204695] free_work+0x34/0x50 [libcfs] [ 5504.208080] process_one_work+0x2c8/0x700 [ 5504.211547] worker_thread+0x296/0x6e0 [ 5504.213627] ? rescuer_thread+0x570/0x570 [ 5504.215807] kthread+0x1d1/0x200 [ 5504.217588] ? set_kthread_struct+0x70/0x70 [ 5504.220684] ret_from_fork+0x1f/0x30 [ 5508.058113] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5508.062129] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5508.095764] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5508.102822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5508.108359] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5508.111933] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5508.123114] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5508.126741] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5508.130743] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5508.141139] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5508.144370] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5508.150833] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5508.157698] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5508.163474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5508.167642] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5508.172819] Call Trace: [ 5508.174087] [ 5508.175202] ? show_regs.cold.9+0x22/0x2f [ 5508.176829] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5508.180700] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5508.182923] ? watchdog+0x40/0x40 [ 5508.184196] ? hrtimer_interrupt+0x128/0x2c0 [ 5508.187075] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5508.189189] ? apic_timer_interrupt+0xf/0x20 [ 5508.191134] [ 5508.193126] ? on_each_cpu_cond_mask+0x46/0x150 [ 5508.195809] ? __x64_sys_fspick+0x210/0x210 [ 5508.197396] ? smp_call_function_many_cond+0x3db/0x540 [ 5508.202195] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5508.204208] ? __x64_sys_fspick+0x210/0x210 [ 5508.205816] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5508.208333] on_each_cpu_cond_mask+0x46/0x150 [ 5508.210650] on_each_cpu_cond+0x1c/0x30 [ 5508.213422] invalidate_bh_lrus+0x27/0x30 [ 5508.215190] invalidate_bdev+0x5a/0x90 [ 5508.216950] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5508.219227] generic_shutdown_super+0xb1/0x1b0 [ 5508.221119] kill_block_super+0x29/0x80 [ 5508.222841] deactivate_locked_super+0x52/0xd0 [ 5508.224706] deactivate_super+0x83/0x90 [ 5508.226193] cleanup_mnt+0x5f/0xe0 [ 5508.228870] __cleanup_mnt+0x16/0x20 [ 5508.231218] task_work_run+0xc6/0x110 [ 5508.233124] exit_to_usermode_loop+0x1dd/0x1f0 [ 5508.235165] do_syscall_64+0x3d6/0x440 [ 5508.237445] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5508.239742] RIP: 0033:0x7fbb019648fb [ 5508.244350] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5508.255996] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5508.259738] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5508.263536] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5508.269746] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5508.276062] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5508.281191] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5528.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5528.058538] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5528.078777] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5528.083172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5528.086457] Workqueue: events free_work [libcfs] [ 5528.088330] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5528.090577] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5528.097778] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5528.101146] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5528.104206] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5528.107159] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5528.110112] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5528.113217] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5528.116391] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5528.120018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5528.122619] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5528.125723] Call Trace: [ 5528.126898] [ 5528.127708] ? show_regs.cold.9+0x22/0x2f [ 5528.129474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5528.131489] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5528.134132] ? watchdog+0x40/0x40 [ 5528.136460] ? hrtimer_interrupt+0x128/0x2c0 [ 5528.140372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5528.143063] ? apic_timer_interrupt+0xf/0x20 [ 5528.145337] [ 5528.146544] ? on_each_cpu+0x51/0x120 [ 5528.148511] ? smp_call_function_many_cond+0x3db/0x540 [ 5528.151459] ? smp_call_function_many_cond+0x3a8/0x540 [ 5528.155744] ? load_new_mm_cr3+0x120/0x120 [ 5528.158338] ? load_new_mm_cr3+0x120/0x120 [ 5528.160593] on_each_cpu+0x51/0x120 [ 5528.163189] flush_tlb_kernel_range+0x93/0xb0 [ 5528.167117] free_unmap_vmap_area+0x63/0x70 [ 5528.169631] remove_vm_area+0xe7/0x110 [ 5528.172897] __vunmap+0x9c/0x400 [ 5528.174521] vfree+0x4d/0x100 [ 5528.176079] free_work+0x34/0x50 [libcfs] [ 5528.177550] process_one_work+0x2c8/0x700 [ 5528.179616] worker_thread+0x296/0x6e0 [ 5528.181521] ? rescuer_thread+0x570/0x570 [ 5528.183610] kthread+0x1d1/0x200 [ 5528.185453] ? set_kthread_struct+0x70/0x70 [ 5528.187592] ret_from_fork+0x1f/0x30 [ 5530.591717] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3172s! [ 5530.595636] Showing busy workqueues and worker pools: [ 5530.601850] workqueue events: flags=0x0 [ 5530.609140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5530.612481] in-flight: 56293:free_work [libcfs] [ 5530.618341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5530.621472] in-flight: 22277:free_work [libcfs] [ 5530.625986] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5530.632208] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5530.635849] pending: nfs4_renew_state [nfsv4] [ 5530.638118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5530.641845] in-flight: 47596:free_work [libcfs] [ 5530.646321] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5530.651203] workqueue events_unbound: flags=0x2 [ 5530.653670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5530.657064] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5530.663425] workqueue events_power_efficient: flags=0x80 [ 5530.668069] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5530.675383] pending: neigh_periodic_work, do_cache_clean [ 5530.681830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5530.685276] pending: check_lifetime [ 5530.688031] workqueue mm_percpu_wq: flags=0x8 [ 5530.691056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5530.697995] pending: vmstat_update [ 5530.703571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5530.706913] pending: vmstat_update [ 5530.709116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5530.713928] pending: vmstat_update [ 5530.716114] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5530.718892] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5530.722137] pending: cgroup_pidlist_destroy_work_fn [ 5530.725530] workqueue writeback: flags=0x4a [ 5530.728074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5530.731871] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5530.737559] workqueue xprtiod: flags=0xa [ 5530.743063] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5530.748626] pending: xprt_autoclose [ 5530.754157] workqueue ipv6_addrconf: flags=0x40008 [ 5530.756055] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5530.761028] pending: addrconf_verify_work [ 5530.765281] workqueue knbd0-recv: flags=0x1a [ 5530.767709] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5530.772314] in-flight: 655:recv_work [ 5530.775059] workqueue ptlrpc_pinger: flags=0x2 [ 5530.778028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5530.782524] pending: ptlrpc_pinger_main [ptlrpc] [ 5530.785135] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3173s workers=5 idle: 53684 27131 54850 46584 [ 5530.791253] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3173s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5530.798122] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3173s workers=6 idle: 53660 37674 46417 53664 46418 [ 5530.803213] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5520s workers=2 idle: 657 [ 5532.059014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 5532.063431] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5532.088322] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5532.095079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5532.100442] Workqueue: events free_work [libcfs] [ 5532.104226] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5532.108020] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5532.121600] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5532.124958] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5532.128039] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5532.132233] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5532.139224] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5532.146821] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5532.151592] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5532.156845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5532.158602] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5532.165013] Call Trace: [ 5532.166820] [ 5532.167896] ? show_regs.cold.9+0x22/0x2f [ 5532.170336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5532.175695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5532.179567] ? watchdog+0x40/0x40 [ 5532.185059] ? hrtimer_interrupt+0x128/0x2c0 [ 5532.189190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5532.194531] ? apic_timer_interrupt+0xf/0x20 [ 5532.199735] [ 5532.200525] ? on_each_cpu+0x51/0x120 [ 5532.202706] ? smp_call_function_many_cond+0x3db/0x540 [ 5532.205659] ? smp_call_function_many_cond+0x3a8/0x540 [ 5532.212578] ? load_new_mm_cr3+0x120/0x120 [ 5532.214569] ? load_new_mm_cr3+0x120/0x120 [ 5532.217426] on_each_cpu+0x51/0x120 [ 5532.219490] flush_tlb_kernel_range+0x93/0xb0 [ 5532.223849] free_unmap_vmap_area+0x63/0x70 [ 5532.227652] remove_vm_area+0xe7/0x110 [ 5532.232194] __vunmap+0x9c/0x400 [ 5532.235666] vfree+0x4d/0x100 [ 5532.238875] free_work+0x34/0x50 [libcfs] [ 5532.241679] process_one_work+0x2c8/0x700 [ 5532.246671] worker_thread+0x296/0x6e0 [ 5532.249604] ? rescuer_thread+0x570/0x570 [ 5532.252000] kthread+0x1d1/0x200 [ 5532.253599] ? set_kthread_struct+0x70/0x70 [ 5532.257872] ret_from_fork+0x1f/0x30 [ 5536.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5536.062451] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5536.118455] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5536.126515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5536.135686] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5536.143237] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5536.156171] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5536.163707] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5536.166949] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5536.169986] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5536.172922] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5536.176824] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5536.181746] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5536.190020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5536.195977] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5536.205893] Call Trace: [ 5536.206845] [ 5536.207718] ? show_regs.cold.9+0x22/0x2f [ 5536.214303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5536.218094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5536.220442] ? watchdog+0x40/0x40 [ 5536.221627] ? hrtimer_interrupt+0x128/0x2c0 [ 5536.226888] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5536.232155] ? apic_timer_interrupt+0xf/0x20 [ 5536.234878] [ 5536.236504] ? on_each_cpu_cond_mask+0x46/0x150 [ 5536.239212] ? __x64_sys_fspick+0x210/0x210 [ 5536.247348] ? smp_call_function_many_cond+0x3d9/0x540 [ 5536.253058] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5536.256395] ? __x64_sys_fspick+0x210/0x210 [ 5536.260600] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5536.262321] on_each_cpu_cond_mask+0x46/0x150 [ 5536.267867] on_each_cpu_cond+0x1c/0x30 [ 5536.270281] invalidate_bh_lrus+0x27/0x30 [ 5536.272583] invalidate_bdev+0x5a/0x90 [ 5536.275334] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5536.278510] generic_shutdown_super+0xb1/0x1b0 [ 5536.281447] kill_block_super+0x29/0x80 [ 5536.283500] deactivate_locked_super+0x52/0xd0 [ 5536.286115] deactivate_super+0x83/0x90 [ 5536.288174] cleanup_mnt+0x5f/0xe0 [ 5536.289879] __cleanup_mnt+0x16/0x20 [ 5536.292099] task_work_run+0xc6/0x110 [ 5536.294062] exit_to_usermode_loop+0x1dd/0x1f0 [ 5536.296477] do_syscall_64+0x3d6/0x440 [ 5536.298491] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5536.301135] RIP: 0033:0x7fbb019648fb [ 5536.302609] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5536.311583] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5536.317763] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5536.322285] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5536.326167] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5536.331754] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5536.335615] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5556.055019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5556.059898] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5556.101791] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5556.106712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5556.111704] Workqueue: events free_work [libcfs] [ 5556.114368] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5556.117116] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5556.125256] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5556.129266] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5556.133675] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5556.140288] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5556.145757] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5556.152196] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5556.159023] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5556.165023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5556.168726] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5556.172578] Call Trace: [ 5556.174032] [ 5556.174846] ? show_regs.cold.9+0x22/0x2f [ 5556.176844] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5556.180092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5556.182178] ? watchdog+0x40/0x40 [ 5556.184432] ? hrtimer_interrupt+0x128/0x2c0 [ 5556.186020] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5556.189612] ? apic_timer_interrupt+0xf/0x20 [ 5556.193134] [ 5556.194429] ? on_each_cpu+0x51/0x120 [ 5556.198593] ? smp_call_function_many_cond+0x3db/0x540 [ 5556.204049] ? smp_call_function_many_cond+0x3a8/0x540 [ 5556.207368] ? load_new_mm_cr3+0x120/0x120 [ 5556.210051] ? load_new_mm_cr3+0x120/0x120 [ 5556.212099] on_each_cpu+0x51/0x120 [ 5556.213751] flush_tlb_kernel_range+0x93/0xb0 [ 5556.215934] free_unmap_vmap_area+0x63/0x70 [ 5556.221092] remove_vm_area+0xe7/0x110 [ 5556.222686] __vunmap+0x9c/0x400 [ 5556.225867] vfree+0x4d/0x100 [ 5556.230777] free_work+0x34/0x50 [libcfs] [ 5556.235811] process_one_work+0x2c8/0x700 [ 5556.239814] worker_thread+0x296/0x6e0 [ 5556.242620] ? rescuer_thread+0x570/0x570 [ 5556.246140] kthread+0x1d1/0x200 [ 5556.247635] ? set_kthread_struct+0x70/0x70 [ 5556.249480] ret_from_fork+0x1f/0x30 [ 5560.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 5560.063355] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5560.097740] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5560.102259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5560.106504] Workqueue: events free_work [libcfs] [ 5560.108464] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5560.110776] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5560.120009] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5560.123402] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5560.126596] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5560.131533] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5560.134966] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5560.142501] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5560.150253] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5560.156057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5560.160359] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5560.167425] Call Trace: [ 5560.168980] [ 5560.170307] ? show_regs.cold.9+0x22/0x2f [ 5560.172549] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5560.174799] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5560.176953] ? watchdog+0x40/0x40 [ 5560.180689] ? hrtimer_interrupt+0x128/0x2c0 [ 5560.182818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5560.185091] ? apic_timer_interrupt+0xf/0x20 [ 5560.187815] [ 5560.188910] ? on_each_cpu+0x51/0x120 [ 5560.190891] ? smp_call_function_many_cond+0x3db/0x540 [ 5560.194909] ? smp_call_function_many_cond+0x3a8/0x540 [ 5560.199848] ? load_new_mm_cr3+0x120/0x120 [ 5560.202095] ? load_new_mm_cr3+0x120/0x120 [ 5560.204632] on_each_cpu+0x51/0x120 [ 5560.206673] flush_tlb_kernel_range+0x93/0xb0 [ 5560.209267] free_unmap_vmap_area+0x63/0x70 [ 5560.213868] remove_vm_area+0xe7/0x110 [ 5560.216118] __vunmap+0x9c/0x400 [ 5560.219903] vfree+0x4d/0x100 [ 5560.225075] free_work+0x34/0x50 [libcfs] [ 5560.230055] process_one_work+0x2c8/0x700 [ 5560.234148] worker_thread+0x296/0x6e0 [ 5560.236062] ? rescuer_thread+0x570/0x570 [ 5560.247015] kthread+0x1d1/0x200 [ 5560.248899] ? set_kthread_struct+0x70/0x70 [ 5560.251658] ret_from_fork+0x1f/0x30 [ 5561.317021] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3203s! [ 5561.323347] Showing busy workqueues and worker pools: [ 5561.326790] workqueue events: flags=0x0 [ 5561.330519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5561.336071] in-flight: 56293:free_work [libcfs] [ 5561.343653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5561.351614] in-flight: 22277:free_work [libcfs] [ 5561.356500] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5561.362251] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5561.370575] pending: nfs4_renew_state [nfsv4] [ 5561.375355] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5561.382001] in-flight: 47596:free_work [libcfs] [ 5561.385910] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5561.391793] workqueue events_unbound: flags=0x2 [ 5561.393164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5561.396851] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5561.401372] workqueue events_power_efficient: flags=0x80 [ 5561.404324] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5561.408944] pending: neigh_periodic_work, do_cache_clean [ 5561.412420] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5561.417316] pending: check_lifetime [ 5561.419363] workqueue mm_percpu_wq: flags=0x8 [ 5561.422015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5561.425364] pending: vmstat_update [ 5561.426895] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5561.434361] pending: vmstat_update [ 5561.438961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5561.445980] pending: vmstat_update [ 5561.451193] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5561.453948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5561.457211] pending: cgroup_pidlist_destroy_work_fn [ 5561.459890] workqueue writeback: flags=0x4a [ 5561.461980] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5561.465461] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5561.468794] workqueue xprtiod: flags=0xa [ 5561.471254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5561.473441] pending: xprt_autoclose [ 5561.475408] workqueue ipv6_addrconf: flags=0x40008 [ 5561.477932] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5561.481162] pending: addrconf_verify_work [ 5561.482675] workqueue knbd0-recv: flags=0x1a [ 5561.483965] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5561.485248] in-flight: 655:recv_work [ 5561.486237] workqueue ptlrpc_pinger: flags=0x2 [ 5561.489385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5561.493123] pending: ptlrpc_pinger_main [ptlrpc] [ 5561.497396] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3203s workers=5 idle: 53684 27131 54850 46584 [ 5561.502017] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3203s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5561.506991] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3203s workers=6 idle: 53660 37674 46417 53664 46418 [ 5561.512511] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5551s workers=2 idle: 657 [ 5564.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5564.062194] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5564.095057] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5564.101791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5564.109610] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5564.115436] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5564.130719] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5564.134784] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5564.138702] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5564.144455] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5564.149517] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5564.154003] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5564.160166] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5564.164257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5564.167177] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5564.170482] Call Trace: [ 5564.171573] [ 5564.172599] ? show_regs.cold.9+0x22/0x2f [ 5564.174181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5564.175660] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5564.177511] ? watchdog+0x40/0x40 [ 5564.178993] ? hrtimer_interrupt+0x128/0x2c0 [ 5564.180567] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5564.182800] ? apic_timer_interrupt+0xf/0x20 [ 5564.184719] [ 5564.185761] ? on_each_cpu_cond_mask+0x46/0x150 [ 5564.187738] ? __x64_sys_fspick+0x210/0x210 [ 5564.189771] ? smp_call_function_many_cond+0x3d9/0x540 [ 5564.191851] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5564.194218] ? __x64_sys_fspick+0x210/0x210 [ 5564.196760] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5564.198873] on_each_cpu_cond_mask+0x46/0x150 [ 5564.200637] on_each_cpu_cond+0x1c/0x30 [ 5564.202299] invalidate_bh_lrus+0x27/0x30 [ 5564.204015] invalidate_bdev+0x5a/0x90 [ 5564.205634] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5564.211099] generic_shutdown_super+0xb1/0x1b0 [ 5564.215449] kill_block_super+0x29/0x80 [ 5564.219749] deactivate_locked_super+0x52/0xd0 [ 5564.222813] deactivate_super+0x83/0x90 [ 5564.224681] cleanup_mnt+0x5f/0xe0 [ 5564.234473] __cleanup_mnt+0x16/0x20 [ 5564.235807] task_work_run+0xc6/0x110 [ 5564.237057] exit_to_usermode_loop+0x1dd/0x1f0 [ 5564.242384] do_syscall_64+0x3d6/0x440 [ 5564.244404] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5564.246742] RIP: 0033:0x7fbb019648fb [ 5564.248515] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5564.256725] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5564.261223] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5564.264220] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5564.267572] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5564.270560] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5564.273460] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5588.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 5588.059016] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 5588.060052] Modules linked in: zfs(O) spl(O) [ 5588.063395] Modules linked in: [ 5588.063440] lustre(O) [ 5588.065803] zfs(O) spl(O) lustre(O) osp(O) [ 5588.067314] osp(O) [ 5588.068619] ofd(O) [ 5588.071453] ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5588.071783] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5588.071790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5588.071823] Workqueue: events free_work [libcfs] [ 5588.071862] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5588.071882] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5588.071886] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5588.071891] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5588.071893] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5588.071896] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5588.071898] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5588.071900] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5588.071903] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5588.071906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5588.071908] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5588.071916] Call Trace: [ 5588.071932] [ 5588.071938] ? show_regs.cold.9+0x22/0x2f [ 5588.071947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5588.071953] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5588.071956] ? watchdog+0x40/0x40 [ 5588.071961] ? hrtimer_interrupt+0x128/0x2c0 [ 5588.071965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5588.071975] ? apic_timer_interrupt+0xf/0x20 [ 5588.071981] [ 5588.071983] ? on_each_cpu+0x51/0x120 [ 5588.071986] ? smp_call_function_many_cond+0x3d9/0x540 [ 5588.071990] ? smp_call_function_many_cond+0x3a8/0x540 [ 5588.071993] ? load_new_mm_cr3+0x120/0x120 [ 5588.071998] ? load_new_mm_cr3+0x120/0x120 [ 5588.072002] on_each_cpu+0x51/0x120 [ 5588.072006] flush_tlb_kernel_range+0x93/0xb0 [ 5588.072012] free_unmap_vmap_area+0x63/0x70 [ 5588.072113] remove_vm_area+0xe7/0x110 [ 5588.074794] lod(O) [ 5588.075367] __vunmap+0x9c/0x400 [ 5588.092377] ost(O) [ 5588.098550] vfree+0x4d/0x100 [ 5588.104230] mdt(O) [ 5588.105835] free_work+0x34/0x50 [libcfs] [ 5588.107443] mdd(O) [ 5588.112983] process_one_work+0x2c8/0x700 [ 5588.115233] mgs(O) [ 5588.117885] worker_thread+0x296/0x6e0 [ 5588.119752] osd_ldiskfs(O) [ 5588.122193] ? rescuer_thread+0x570/0x570 [ 5588.124337] ldiskfs(O) [ 5588.127936] kthread+0x1d1/0x200 [ 5588.130226] lquota(O) [ 5588.132872] ? set_kthread_struct+0x70/0x70 [ 5588.137144] lfsck(O) [ 5588.138200] ret_from_fork+0x1f/0x30 [ 5588.139326] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5588.220965] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5588.227379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5588.233916] Workqueue: events free_work [libcfs] [ 5588.237698] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5588.241414] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5588.254887] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5588.263364] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5588.270319] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5588.276585] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5588.280646] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5588.286233] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5588.294211] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5588.300376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5588.303879] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5588.307795] Call Trace: [ 5588.308672] [ 5588.309870] ? show_regs.cold.9+0x22/0x2f [ 5588.313438] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5588.316416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5588.320063] ? watchdog+0x40/0x40 [ 5588.321773] ? hrtimer_interrupt+0x128/0x2c0 [ 5588.326463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5588.329039] ? apic_timer_interrupt+0xf/0x20 [ 5588.332463] [ 5588.333897] ? on_each_cpu+0x51/0x120 [ 5588.336308] ? smp_call_function_many_cond+0x3d9/0x540 [ 5588.339301] ? smp_call_function_many_cond+0x3a8/0x540 [ 5588.341345] ? load_new_mm_cr3+0x120/0x120 [ 5588.345309] ? load_new_mm_cr3+0x120/0x120 [ 5588.346993] on_each_cpu+0x51/0x120 [ 5588.348477] flush_tlb_kernel_range+0x93/0xb0 [ 5588.350877] free_unmap_vmap_area+0x63/0x70 [ 5588.352762] remove_vm_area+0xe7/0x110 [ 5588.354455] __vunmap+0x9c/0x400 [ 5588.356163] vfree+0x4d/0x100 [ 5588.357496] free_work+0x34/0x50 [libcfs] [ 5588.363777] process_one_work+0x2c8/0x700 [ 5588.365702] worker_thread+0x296/0x6e0 [ 5588.367140] ? rescuer_thread+0x570/0x570 [ 5588.368982] kthread+0x1d1/0x200 [ 5588.370461] ? set_kthread_struct+0x70/0x70 [ 5588.372081] ret_from_fork+0x1f/0x30 [ 5591.520073] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3233s! [ 5591.524288] Showing busy workqueues and worker pools: [ 5591.526581] workqueue events: flags=0x0 [ 5591.528551] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5591.531956] in-flight: 56293:free_work [libcfs] [ 5591.534622] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5591.537611] in-flight: 22277:free_work [libcfs] [ 5591.539861] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5591.544424] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5591.547072] pending: nfs4_renew_state [nfsv4] [ 5591.549182] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5591.552080] in-flight: 47596:free_work [libcfs] [ 5591.555674] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5591.560420] workqueue events_unbound: flags=0x2 [ 5591.562953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5591.566377] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5591.570897] workqueue events_power_efficient: flags=0x80 [ 5591.573707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5591.577010] pending: neigh_periodic_work, do_cache_clean [ 5591.579892] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5591.583676] pending: check_lifetime [ 5591.585838] workqueue mm_percpu_wq: flags=0x8 [ 5591.588042] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5591.592167] pending: vmstat_update [ 5591.595384] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5591.597982] pending: vmstat_update [ 5591.600372] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5591.605079] pending: vmstat_update [ 5591.607509] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5591.610094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5591.614949] pending: cgroup_pidlist_destroy_work_fn [ 5591.618091] workqueue writeback: flags=0x4a [ 5591.620644] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5591.623856] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5591.627189] workqueue xprtiod: flags=0xa [ 5591.629212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5591.633199] pending: xprt_autoclose [ 5591.635594] workqueue ipv6_addrconf: flags=0x40008 [ 5591.638929] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5591.641516] pending: addrconf_verify_work [ 5591.643849] workqueue knbd0-recv: flags=0x1a [ 5591.646193] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5591.654427] in-flight: 655:recv_work [ 5591.659932] workqueue ptlrpc_pinger: flags=0x2 [ 5591.668286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5591.671391] pending: ptlrpc_pinger_main [ptlrpc] [ 5591.678018] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3233s workers=5 idle: 53684 27131 54850 46584 [ 5591.684950] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3233s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5591.691281] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3233s workers=6 idle: 53660 37674 46417 53664 46418 [ 5591.697802] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5581s workers=2 idle: 657 [ 5592.058041] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5592.063187] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5592.096017] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5592.101835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5592.107123] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5592.110142] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5592.120391] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5592.124889] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5592.128557] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5592.131973] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5592.137116] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5592.144722] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5592.155422] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5592.160352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5592.162601] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5592.168293] Call Trace: [ 5592.171999] [ 5592.173264] ? show_regs.cold.9+0x22/0x2f [ 5592.181380] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5592.186957] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5592.190964] ? watchdog+0x40/0x40 [ 5592.192612] ? hrtimer_interrupt+0x128/0x2c0 [ 5592.194368] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5592.196414] ? apic_timer_interrupt+0xf/0x20 [ 5592.198694] [ 5592.199732] ? on_each_cpu_cond_mask+0x46/0x150 [ 5592.201698] ? __x64_sys_fspick+0x210/0x210 [ 5592.204321] ? smp_call_function_many_cond+0x3d9/0x540 [ 5592.212381] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5592.214133] ? __x64_sys_fspick+0x210/0x210 [ 5592.215001] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5592.218985] on_each_cpu_cond_mask+0x46/0x150 [ 5592.223059] on_each_cpu_cond+0x1c/0x30 [ 5592.225185] invalidate_bh_lrus+0x27/0x30 [ 5592.226531] invalidate_bdev+0x5a/0x90 [ 5592.228270] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5592.231140] generic_shutdown_super+0xb1/0x1b0 [ 5592.233157] kill_block_super+0x29/0x80 [ 5592.235103] deactivate_locked_super+0x52/0xd0 [ 5592.237079] deactivate_super+0x83/0x90 [ 5592.238742] cleanup_mnt+0x5f/0xe0 [ 5592.240285] __cleanup_mnt+0x16/0x20 [ 5592.241818] task_work_run+0xc6/0x110 [ 5592.243270] exit_to_usermode_loop+0x1dd/0x1f0 [ 5592.245174] do_syscall_64+0x3d6/0x440 [ 5592.246876] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5592.249086] RIP: 0033:0x7fbb019648fb [ 5592.250755] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5592.258707] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5592.262016] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5592.264902] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5592.267607] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5592.270733] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5592.275440] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5616.055015] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5616.059079] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5616.061471] Modules linked in: [ 5616.070190] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5616.070288] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5616.070295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5616.070302] Workqueue: events free_work [libcfs] [ 5616.070345] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5616.070353] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5616.070358] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5616.070362] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5616.070364] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5616.070366] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5616.070369] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5616.070371] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5616.070374] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5616.070377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5616.070379] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5616.070388] Call Trace: [ 5616.070393] [ 5616.070398] ? show_regs.cold.9+0x22/0x2f [ 5616.070407] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5616.070415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5616.070419] ? watchdog+0x40/0x40 [ 5616.070424] ? hrtimer_interrupt+0x128/0x2c0 [ 5616.070428] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5616.070437] ? apic_timer_interrupt+0xf/0x20 [ 5616.070442] [ 5616.070444] ? on_each_cpu+0x51/0x120 [ 5616.070449] ? smp_call_function_many_cond+0x3db/0x540 [ 5616.070453] ? smp_call_function_many_cond+0x3a8/0x540 [ 5616.070456] ? load_new_mm_cr3+0x120/0x120 [ 5616.070462] ? load_new_mm_cr3+0x120/0x120 [ 5616.070466] on_each_cpu+0x51/0x120 [ 5616.070470] flush_tlb_kernel_range+0x93/0xb0 [ 5616.070476] free_unmap_vmap_area+0x63/0x70 [ 5616.070482] remove_vm_area+0xe7/0x110 [ 5616.070485] __vunmap+0x9c/0x400 [ 5616.070489] vfree+0x4d/0x100 [ 5616.070493] free_work+0x34/0x50 [libcfs] [ 5616.070521] process_one_work+0x2c8/0x700 [ 5616.070529] worker_thread+0x296/0x6e0 [ 5616.070533] ? rescuer_thread+0x570/0x570 [ 5616.070537] kthread+0x1d1/0x200 [ 5616.070541] ? set_kthread_struct+0x70/0x70 [ 5616.070545] ret_from_fork+0x1f/0x30 [ 5616.316441] zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5616.350819] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5616.360728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5616.369737] Workqueue: events free_work [libcfs] [ 5616.375536] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5616.380816] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5616.397024] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5616.403773] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5616.407726] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5616.413794] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5616.419866] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5616.428969] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5616.434565] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5616.442786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5616.447280] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5616.457707] Call Trace: [ 5616.459431] [ 5616.460246] ? show_regs.cold.9+0x22/0x2f [ 5616.462604] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5616.468622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5616.472881] ? watchdog+0x40/0x40 [ 5616.477294] ? hrtimer_interrupt+0x128/0x2c0 [ 5616.484778] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5616.486932] ? apic_timer_interrupt+0xf/0x20 [ 5616.490848] [ 5616.498826] ? on_each_cpu+0x51/0x120 [ 5616.502931] ? smp_call_function_many_cond+0x3db/0x540 [ 5616.505413] ? smp_call_function_many_cond+0x3a8/0x540 [ 5616.508404] ? load_new_mm_cr3+0x120/0x120 [ 5616.511535] ? load_new_mm_cr3+0x120/0x120 [ 5616.513583] on_each_cpu+0x51/0x120 [ 5616.515099] flush_tlb_kernel_range+0x93/0xb0 [ 5616.516699] free_unmap_vmap_area+0x63/0x70 [ 5616.520816] remove_vm_area+0xe7/0x110 [ 5616.526209] __vunmap+0x9c/0x400 [ 5616.531604] vfree+0x4d/0x100 [ 5616.533055] free_work+0x34/0x50 [libcfs] [ 5616.535387] process_one_work+0x2c8/0x700 [ 5616.537857] worker_thread+0x296/0x6e0 [ 5616.543881] ? rescuer_thread+0x570/0x570 [ 5616.549707] kthread+0x1d1/0x200 [ 5616.553321] ? set_kthread_struct+0x70/0x70 [ 5616.558376] ret_from_fork+0x1f/0x30 [ 5620.058029] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5620.064649] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5620.111586] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5620.121233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5620.128512] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5620.137542] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5620.152897] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5620.157182] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5620.163301] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5620.165841] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5620.184117] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5620.190637] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5620.196210] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5620.205414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5620.210271] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5620.214594] Call Trace: [ 5620.216208] [ 5620.217600] ? show_regs.cold.9+0x22/0x2f [ 5620.220324] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5620.224236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5620.228509] ? watchdog+0x40/0x40 [ 5620.232196] ? hrtimer_interrupt+0x128/0x2c0 [ 5620.233612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5620.238478] ? apic_timer_interrupt+0xf/0x20 [ 5620.241356] [ 5620.242129] ? on_each_cpu_cond_mask+0x46/0x150 [ 5620.250390] ? __x64_sys_fspick+0x210/0x210 [ 5620.260612] ? smp_call_function_many_cond+0x3db/0x540 [ 5620.262613] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5620.264988] ? __x64_sys_fspick+0x210/0x210 [ 5620.272329] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5620.275405] on_each_cpu_cond_mask+0x46/0x150 [ 5620.282939] on_each_cpu_cond+0x1c/0x30 [ 5620.287497] invalidate_bh_lrus+0x27/0x30 [ 5620.291828] invalidate_bdev+0x5a/0x90 [ 5620.294166] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5620.302830] generic_shutdown_super+0xb1/0x1b0 [ 5620.307738] kill_block_super+0x29/0x80 [ 5620.312870] deactivate_locked_super+0x52/0xd0 [ 5620.316833] deactivate_super+0x83/0x90 [ 5620.319146] cleanup_mnt+0x5f/0xe0 [ 5620.320629] __cleanup_mnt+0x16/0x20 [ 5620.322889] task_work_run+0xc6/0x110 [ 5620.324687] exit_to_usermode_loop+0x1dd/0x1f0 [ 5620.327856] do_syscall_64+0x3d6/0x440 [ 5620.330130] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5620.335241] RIP: 0033:0x7fbb019648fb [ 5620.338864] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5620.355214] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5620.363699] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5620.369193] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5620.373710] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5620.378096] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5620.382007] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5621.729672] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3263s! [ 5621.735922] Showing busy workqueues and worker pools: [ 5621.737752] workqueue events: flags=0x0 [ 5621.745256] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5621.755870] in-flight: 56293:free_work [libcfs] [ 5621.764530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5621.769663] in-flight: 22277:free_work [libcfs] [ 5621.772940] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5621.783354] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5621.787373] pending: nfs4_renew_state [nfsv4] [ 5621.791134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5621.801553] in-flight: 47596:free_work [libcfs] [ 5621.804973] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5621.818830] workqueue events_unbound: flags=0x2 [ 5621.821454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5621.827880] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5621.833036] workqueue events_power_efficient: flags=0x80 [ 5621.835953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5621.841242] pending: neigh_periodic_work, do_cache_clean [ 5621.845852] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5621.850583] pending: check_lifetime [ 5621.852674] workqueue mm_percpu_wq: flags=0x8 [ 5621.854619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5621.857199] pending: vmstat_update [ 5621.858590] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5621.861452] pending: vmstat_update [ 5621.862984] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5621.867419] pending: vmstat_update [ 5621.872634] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5621.876189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5621.883982] pending: cgroup_pidlist_destroy_work_fn [ 5621.886716] workqueue writeback: flags=0x4a [ 5621.888961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5621.891733] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5621.894696] workqueue xprtiod: flags=0xa [ 5621.896407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5621.899224] pending: xprt_autoclose [ 5621.901652] workqueue ipv6_addrconf: flags=0x40008 [ 5621.904097] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5621.906737] pending: addrconf_verify_work [ 5621.909079] workqueue knbd0-recv: flags=0x1a [ 5621.910912] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5621.913655] in-flight: 655:recv_work [ 5621.915396] workqueue ptlrpc_pinger: flags=0x2 [ 5621.917302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5621.920031] pending: ptlrpc_pinger_main [ptlrpc] [ 5621.922481] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3264s workers=5 idle: 53684 27131 54850 46584 [ 5621.927465] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3264s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5621.933752] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3264s workers=6 idle: 53660 37674 46417 53664 46418 [ 5621.939611] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5611s workers=2 idle: 657 [ 5644.059014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5644.062620] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5644.099631] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5644.109264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5644.115067] Workqueue: events free_work [libcfs] [ 5644.120412] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5644.127674] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5644.143560] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5644.150404] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5644.159459] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5644.166045] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5644.170285] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5644.173894] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5644.177573] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5644.181256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5644.183994] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5644.188962] Call Trace: [ 5644.190132] [ 5644.191126] ? show_regs.cold.9+0x22/0x2f [ 5644.193205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5644.195743] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5644.197960] ? watchdog+0x40/0x40 [ 5644.199785] ? hrtimer_interrupt+0x128/0x2c0 [ 5644.201200] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5644.203418] ? apic_timer_interrupt+0xf/0x20 [ 5644.205652] [ 5644.206650] ? on_each_cpu+0x51/0x120 [ 5644.208609] ? smp_call_function_many_cond+0x3d9/0x540 [ 5644.211011] ? smp_call_function_many_cond+0x3a8/0x540 [ 5644.213200] ? load_new_mm_cr3+0x120/0x120 [ 5644.214631] ? load_new_mm_cr3+0x120/0x120 [ 5644.216941] on_each_cpu+0x51/0x120 [ 5644.218655] flush_tlb_kernel_range+0x93/0xb0 [ 5644.220845] free_unmap_vmap_area+0x63/0x70 [ 5644.222872] remove_vm_area+0xe7/0x110 [ 5644.224807] __vunmap+0x9c/0x400 [ 5644.226483] vfree+0x4d/0x100 [ 5644.227965] free_work+0x34/0x50 [libcfs] [ 5644.230078] process_one_work+0x2c8/0x700 [ 5644.231957] worker_thread+0x296/0x6e0 [ 5644.233955] ? rescuer_thread+0x570/0x570 [ 5644.235922] kthread+0x1d1/0x200 [ 5644.237585] ? set_kthread_struct+0x70/0x70 [ 5644.239407] ret_from_fork+0x1f/0x30 [ 5648.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 5648.058015] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5648.060076] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) [ 5648.068726] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5648.068827] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5648.068836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5648.068839] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5648.068855] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5648.068859] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5648.068864] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5648.068867] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5648.068869] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5648.068872] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5648.068874] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5648.068878] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5648.068880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5648.068882] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5648.068891] Call Trace: [ 5648.068897] [ 5648.068905] ? show_regs.cold.9+0x22/0x2f [ 5648.068918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5648.068925] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5648.068929] ? watchdog+0x40/0x40 [ 5648.068934] ? hrtimer_interrupt+0x128/0x2c0 [ 5648.068938] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5648.068949] ? apic_timer_interrupt+0xf/0x20 [ 5648.068955] [ 5648.068957] ? on_each_cpu_cond_mask+0x46/0x150 [ 5648.068961] ? __x64_sys_fspick+0x210/0x210 [ 5648.068971] ? smp_call_function_many_cond+0x3db/0x540 [ 5648.068975] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5648.068979] ? __x64_sys_fspick+0x210/0x210 [ 5648.068983] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5648.068985] on_each_cpu_cond_mask+0x46/0x150 [ 5648.068990] on_each_cpu_cond+0x1c/0x30 [ 5648.068993] invalidate_bh_lrus+0x27/0x30 [ 5648.068997] invalidate_bdev+0x5a/0x90 [ 5648.069004] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5648.069856] generic_shutdown_super+0xb1/0x1b0 [ 5648.069869] kill_block_super+0x29/0x80 [ 5648.069875] deactivate_locked_super+0x52/0xd0 [ 5648.069879] deactivate_super+0x83/0x90 [ 5648.069884] cleanup_mnt+0x5f/0xe0 [ 5648.069890] __cleanup_mnt+0x16/0x20 [ 5648.069894] task_work_run+0xc6/0x110 [ 5648.069904] exit_to_usermode_loop+0x1dd/0x1f0 [ 5648.069914] do_syscall_64+0x3d6/0x440 [ 5648.069920] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5648.069928] RIP: 0033:0x7fbb019648fb [ 5648.069937] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5648.069941] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5648.069946] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5648.069949] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5648.069951] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5648.069953] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5648.069956] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5648.328168] ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5648.346840] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5648.352654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5648.356754] Workqueue: events free_work [libcfs] [ 5648.358847] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5648.361542] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5648.373514] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5648.379153] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5648.382988] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5648.387736] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5648.393853] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5648.399225] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5648.401718] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5648.406923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5648.412032] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5648.415583] Call Trace: [ 5648.416700] [ 5648.417584] ? show_regs.cold.9+0x22/0x2f [ 5648.419317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5648.421462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5648.422849] ? watchdog+0x40/0x40 [ 5648.424339] ? hrtimer_interrupt+0x128/0x2c0 [ 5648.427362] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5648.429734] ? apic_timer_interrupt+0xf/0x20 [ 5648.432351] [ 5648.433369] ? on_each_cpu+0x51/0x120 [ 5648.435278] ? smp_call_function_many_cond+0x3d9/0x540 [ 5648.438042] ? smp_call_function_many_cond+0x3a8/0x540 [ 5648.440785] ? load_new_mm_cr3+0x120/0x120 [ 5648.442384] ? load_new_mm_cr3+0x120/0x120 [ 5648.445750] on_each_cpu+0x51/0x120 [ 5648.447767] flush_tlb_kernel_range+0x93/0xb0 [ 5648.450623] free_unmap_vmap_area+0x63/0x70 [ 5648.453066] remove_vm_area+0xe7/0x110 [ 5648.455101] __vunmap+0x9c/0x400 [ 5648.457123] vfree+0x4d/0x100 [ 5648.458722] free_work+0x34/0x50 [libcfs] [ 5648.461030] process_one_work+0x2c8/0x700 [ 5648.463617] worker_thread+0x296/0x6e0 [ 5648.465774] ? rescuer_thread+0x570/0x570 [ 5648.467556] kthread+0x1d1/0x200 [ 5648.469582] ? set_kthread_struct+0x70/0x70 [ 5648.471555] ret_from_fork+0x1f/0x30 [ 5652.453362] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3294s! [ 5652.462165] Showing busy workqueues and worker pools: [ 5652.470074] workqueue events: flags=0x0 [ 5652.473663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5652.485610] in-flight: 56293:free_work [libcfs] [ 5652.491112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5652.497878] in-flight: 22277:free_work [libcfs] [ 5652.501721] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5652.509604] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5652.516750] pending: nfs4_renew_state [nfsv4] [ 5652.522018] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5652.524527] in-flight: 47596:free_work [libcfs] [ 5652.531729] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5652.540909] workqueue events_unbound: flags=0x2 [ 5652.544138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5652.546243] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5652.550588] workqueue events_power_efficient: flags=0x80 [ 5652.553896] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5652.563653] pending: neigh_periodic_work, do_cache_clean [ 5652.568365] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5652.573594] pending: check_lifetime [ 5652.578224] workqueue mm_percpu_wq: flags=0x8 [ 5652.584149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5652.592655] pending: vmstat_update [ 5652.595937] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5652.598714] pending: vmstat_update [ 5652.601188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5652.605862] pending: vmstat_update [ 5652.607867] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5652.617809] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5652.622538] pending: cgroup_pidlist_destroy_work_fn [ 5652.626356] workqueue writeback: flags=0x4a [ 5652.630927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5652.636856] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5652.643596] workqueue xprtiod: flags=0xa [ 5652.654229] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5652.657488] pending: xprt_autoclose [ 5652.660975] workqueue ipv6_addrconf: flags=0x40008 [ 5652.664994] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5652.673291] pending: addrconf_verify_work [ 5652.677920] workqueue knbd0-recv: flags=0x1a [ 5652.680943] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5652.684964] in-flight: 655:recv_work [ 5652.688692] workqueue ptlrpc_pinger: flags=0x2 [ 5652.692626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5652.696810] pending: ptlrpc_pinger_main [ptlrpc] [ 5652.700109] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3294s workers=5 idle: 53684 27131 54850 46584 [ 5652.708585] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3294s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5652.715156] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3294s workers=6 idle: 53660 37674 46417 53664 46418 [ 5652.722096] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5642s workers=2 idle: 657 [ 5657.828015] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 5657.836914] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 5657.842601] (detected by 0, t=3300112 jiffies, g=626849, q=1398) [ 5657.845339] Sending NMI from CPU 0 to CPUs 3: [ 5657.852725] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 5657.852735] rcu: rcu_sched kthread starved for 3299764 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5657.869965] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5657.879193] rcu: RCU grace-period kthread stack dump: [ 5657.883361] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5657.888941] Call Trace: [ 5657.890297] __schedule+0x351/0xcb0 [ 5657.891721] schedule+0xc0/0x180 [ 5657.893146] schedule_timeout+0xb4/0x190 [ 5657.898460] ? do_raw_spin_unlock+0x75/0x190 [ 5657.900722] ? __next_timer_interrupt+0x160/0x160 [ 5657.903768] ? prepare_to_swait_event+0xf3/0x140 [ 5657.908786] rcu_gp_kthread+0x8dd/0x15b0 [ 5657.912280] ? force_qs_rnp+0x2c0/0x2c0 [ 5657.916667] kthread+0x1d1/0x200 [ 5657.920291] ? set_kthread_struct+0x70/0x70 [ 5657.924847] ret_from_fork+0x1f/0x30 [ 5657.926657] rcu: Stack dump where RCU GP kthread last ran: [ 5657.934770] Sending NMI from CPU 0 to CPUs 2: [ 5657.941012] NMI backtrace for cpu 2 [ 5657.941018] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5657.941020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5657.941022] Workqueue: events free_work [libcfs] [ 5657.941024] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5657.941025] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5657.941028] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 [ 5657.941030] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5657.941031] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5657.941032] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5657.941033] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5657.941035] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5657.941036] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5657.941047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5657.941049] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5657.941050] Call Trace: [ 5657.941051] [ 5657.941052] ? show_regs.cold.9+0x22/0x2f [ 5657.941053] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5657.941054] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5657.941055] ? nmi_handle+0xbd/0x210 [ 5657.941055] ? default_do_nmi+0x4b/0x1b0 [ 5657.941056] ? do_nmi+0x280/0x380 [ 5657.941057] ? end_repeat_nmi+0x16/0x1f [ 5657.941059] ? on_each_cpu+0x51/0x120 [ 5657.941061] ? smp_call_function_many_cond+0x3db/0x540 [ 5657.941062] ? smp_call_function_many_cond+0x3db/0x540 [ 5657.941063] ? smp_call_function_many_cond+0x3db/0x540 [ 5657.941064] [ 5657.941065] ? load_new_mm_cr3+0x120/0x120 [ 5657.941066] ? load_new_mm_cr3+0x120/0x120 [ 5657.941067] on_each_cpu+0x51/0x120 [ 5657.941068] flush_tlb_kernel_range+0x93/0xb0 [ 5657.941069] free_unmap_vmap_area+0x63/0x70 [ 5657.941070] remove_vm_area+0xe7/0x110 [ 5657.941071] __vunmap+0x9c/0x400 [ 5657.941072] vfree+0x4d/0x100 [ 5657.941073] free_work+0x34/0x50 [libcfs] [ 5657.941074] process_one_work+0x2c8/0x700 [ 5657.941075] worker_thread+0x296/0x6e0 [ 5657.941076] ? rescuer_thread+0x570/0x570 [ 5657.941077] kthread+0x1d1/0x200 [ 5657.941078] ? set_kthread_struct+0x70/0x70 [ 5657.941079] ret_from_fork+0x1f/0x30 [ 5676.058575] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5676.063641] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5676.104472] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5676.111282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5676.115095] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5676.118790] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5676.134087] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5676.141563] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5676.147530] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5676.154613] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5676.160924] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5676.165800] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5676.168436] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5676.172432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5676.175944] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5676.182557] Call Trace: [ 5676.184123] [ 5676.185264] ? show_regs.cold.9+0x22/0x2f [ 5676.187383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5676.192103] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5676.194405] ? watchdog+0x40/0x40 [ 5676.196342] ? hrtimer_interrupt+0x128/0x2c0 [ 5676.198438] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5676.200226] ? apic_timer_interrupt+0xf/0x20 [ 5676.202589] [ 5676.203945] ? on_each_cpu_cond_mask+0x46/0x150 [ 5676.206202] ? __x64_sys_fspick+0x210/0x210 [ 5676.208257] ? smp_call_function_many_cond+0x3d9/0x540 [ 5676.211990] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5676.215521] ? __x64_sys_fspick+0x210/0x210 [ 5676.217255] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5676.218805] on_each_cpu_cond_mask+0x46/0x150 [ 5676.222949] on_each_cpu_cond+0x1c/0x30 [ 5676.229386] invalidate_bh_lrus+0x27/0x30 [ 5676.231580] invalidate_bdev+0x5a/0x90 [ 5676.233905] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5676.236533] generic_shutdown_super+0xb1/0x1b0 [ 5676.240144] kill_block_super+0x29/0x80 [ 5676.241569] deactivate_locked_super+0x52/0xd0 [ 5676.246926] deactivate_super+0x83/0x90 [ 5676.248305] cleanup_mnt+0x5f/0xe0 [ 5676.250305] __cleanup_mnt+0x16/0x20 [ 5676.252032] task_work_run+0xc6/0x110 [ 5676.256191] exit_to_usermode_loop+0x1dd/0x1f0 [ 5676.259871] do_syscall_64+0x3d6/0x440 [ 5676.262427] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5676.268096] RIP: 0033:0x7fbb019648fb [ 5676.269933] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5676.280272] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5676.283936] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5676.286982] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5676.290370] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5676.293129] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5676.296200] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5683.168029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3324s! [ 5683.173228] Showing busy workqueues and worker pools: [ 5683.175861] workqueue events: flags=0x0 [ 5683.177834] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5683.181038] in-flight: 56293:free_work [libcfs] [ 5683.182940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5683.185813] in-flight: 22277:free_work [libcfs] [ 5683.187961] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5683.192317] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5683.197239] pending: nfs4_renew_state [nfsv4] [ 5683.201898] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5683.204329] in-flight: 47596:free_work [libcfs] [ 5683.206529] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5683.211539] workqueue events_unbound: flags=0x2 [ 5683.214422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5683.218618] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5683.224322] workqueue events_power_efficient: flags=0x80 [ 5683.227750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5683.230428] pending: neigh_periodic_work, do_cache_clean [ 5683.232791] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5683.235414] pending: check_lifetime [ 5683.238306] workqueue mm_percpu_wq: flags=0x8 [ 5683.240727] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5683.244548] pending: vmstat_update [ 5683.246031] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5683.249396] pending: vmstat_update [ 5683.252195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5683.254943] pending: vmstat_update [ 5683.258280] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5683.262407] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5683.265690] pending: cgroup_pidlist_destroy_work_fn [ 5683.268295] workqueue writeback: flags=0x4a [ 5683.271226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5683.274099] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5683.281378] workqueue xprtiod: flags=0xa [ 5683.284029] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5683.287000] pending: xprt_autoclose [ 5683.289419] workqueue ipv6_addrconf: flags=0x40008 [ 5683.295253] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5683.298326] pending: addrconf_verify_work [ 5683.302731] workqueue knbd0-recv: flags=0x1a [ 5683.304963] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5683.308138] in-flight: 655:recv_work [ 5683.310552] workqueue ptlrpc_pinger: flags=0x2 [ 5683.312308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5683.314933] pending: ptlrpc_pinger_main [ptlrpc] [ 5683.319331] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3325s workers=5 idle: 53684 27131 54850 46584 [ 5683.323982] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3325s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5683.329571] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3325s workers=6 idle: 53660 37674 46417 53664 46418 [ 5683.335698] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5673s workers=2 idle: 657 [ 5684.059020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5684.063872] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5684.085787] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5684.091193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5684.095618] Workqueue: events free_work [libcfs] [ 5684.098125] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5684.100742] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5684.112889] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5684.116613] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5684.119879] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5684.123017] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5684.126306] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5684.130097] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5684.134911] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5684.139335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5684.142418] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5684.147716] Call Trace: [ 5684.148893] [ 5684.150010] ? show_regs.cold.9+0x22/0x2f [ 5684.151945] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5684.155297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5684.157348] ? watchdog+0x40/0x40 [ 5684.158576] ? hrtimer_interrupt+0x128/0x2c0 [ 5684.160932] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5684.164299] ? apic_timer_interrupt+0xf/0x20 [ 5684.166032] [ 5684.166877] ? on_each_cpu+0x51/0x120 [ 5684.168453] ? smp_call_function_many_cond+0x3d9/0x540 [ 5684.172265] ? smp_call_function_many_cond+0x3a8/0x540 [ 5684.175581] ? load_new_mm_cr3+0x120/0x120 [ 5684.178066] ? load_new_mm_cr3+0x120/0x120 [ 5684.182065] on_each_cpu+0x51/0x120 [ 5684.184421] flush_tlb_kernel_range+0x93/0xb0 [ 5684.187298] free_unmap_vmap_area+0x63/0x70 [ 5684.189866] remove_vm_area+0xe7/0x110 [ 5684.193682] __vunmap+0x9c/0x400 [ 5684.194966] vfree+0x4d/0x100 [ 5684.196330] free_work+0x34/0x50 [libcfs] [ 5684.198366] process_one_work+0x2c8/0x700 [ 5684.200257] worker_thread+0x296/0x6e0 [ 5684.202085] ? rescuer_thread+0x570/0x570 [ 5684.203980] kthread+0x1d1/0x200 [ 5684.205355] ? set_kthread_struct+0x70/0x70 [ 5684.207632] ret_from_fork+0x1f/0x30 [ 5704.058019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [umount:59839] [ 5704.064915] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5704.104959] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5704.115363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5704.121389] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5704.124560] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5704.141081] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5704.146400] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5704.150735] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5704.156090] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5704.160502] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5704.169622] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5704.174579] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5704.180072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5704.190582] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5704.198575] Call Trace: [ 5704.201297] [ 5704.202563] ? show_regs.cold.9+0x22/0x2f [ 5704.205392] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5704.208173] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5704.211737] ? watchdog+0x40/0x40 [ 5704.213123] ? hrtimer_interrupt+0x128/0x2c0 [ 5704.215903] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5704.221550] ? apic_timer_interrupt+0xf/0x20 [ 5704.224161] [ 5704.225731] ? on_each_cpu_cond_mask+0x46/0x150 [ 5704.228968] ? __x64_sys_fspick+0x210/0x210 [ 5704.231696] ? smp_call_function_many_cond+0x3d9/0x540 [ 5704.233906] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5704.236408] ? __x64_sys_fspick+0x210/0x210 [ 5704.239641] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5704.242138] on_each_cpu_cond_mask+0x46/0x150 [ 5704.243928] on_each_cpu_cond+0x1c/0x30 [ 5704.245978] invalidate_bh_lrus+0x27/0x30 [ 5704.247577] invalidate_bdev+0x5a/0x90 [ 5704.254503] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5704.263201] generic_shutdown_super+0xb1/0x1b0 [ 5704.267759] kill_block_super+0x29/0x80 [ 5704.269181] deactivate_locked_super+0x52/0xd0 [ 5704.272460] deactivate_super+0x83/0x90 [ 5704.275364] cleanup_mnt+0x5f/0xe0 [ 5704.276550] __cleanup_mnt+0x16/0x20 [ 5704.280666] task_work_run+0xc6/0x110 [ 5704.284376] exit_to_usermode_loop+0x1dd/0x1f0 [ 5704.287456] do_syscall_64+0x3d6/0x440 [ 5704.289796] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5704.292959] RIP: 0033:0x7fbb019648fb [ 5704.295025] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5704.305333] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5704.310968] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5704.316973] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5704.320574] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5704.324348] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5704.332885] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5708.056443] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 5708.069167] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5708.110519] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5708.118616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5708.122777] Workqueue: events free_work [libcfs] [ 5708.126310] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5708.129256] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5708.142806] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5708.148231] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5708.152648] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5708.156696] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5708.161634] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5708.167441] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5708.175235] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5708.179533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5708.187271] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5708.191181] Call Trace: [ 5708.192191] [ 5708.193005] ? show_regs.cold.9+0x22/0x2f [ 5708.196130] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5708.199115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5708.201597] ? watchdog+0x40/0x40 [ 5708.204040] ? hrtimer_interrupt+0x128/0x2c0 [ 5708.208804] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5708.213691] ? apic_timer_interrupt+0xf/0x20 [ 5708.217309] [ 5708.218680] ? on_each_cpu+0x51/0x120 [ 5708.221592] ? smp_call_function_many_cond+0x3db/0x540 [ 5708.224164] ? smp_call_function_many_cond+0x3a8/0x540 [ 5708.227640] ? load_new_mm_cr3+0x120/0x120 [ 5708.229875] ? load_new_mm_cr3+0x120/0x120 [ 5708.232675] on_each_cpu+0x51/0x120 [ 5708.234283] flush_tlb_kernel_range+0x93/0xb0 [ 5708.236600] free_unmap_vmap_area+0x63/0x70 [ 5708.238747] remove_vm_area+0xe7/0x110 [ 5708.241618] __vunmap+0x9c/0x400 [ 5708.245334] vfree+0x4d/0x100 [ 5708.247437] free_work+0x34/0x50 [libcfs] [ 5708.250826] process_one_work+0x2c8/0x700 [ 5708.253585] worker_thread+0x296/0x6e0 [ 5708.258754] ? rescuer_thread+0x570/0x570 [ 5708.262799] kthread+0x1d1/0x200 [ 5708.264429] ? set_kthread_struct+0x70/0x70 [ 5708.267664] ret_from_fork+0x1f/0x30 [ 5712.059873] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5712.073695] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5712.103182] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5712.107986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5712.113434] Workqueue: events free_work [libcfs] [ 5712.116657] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5712.119345] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5712.127171] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5712.138674] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5712.147461] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5712.153916] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5712.160515] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5712.169383] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5712.173534] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5712.178567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5712.181630] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5712.185951] Call Trace: [ 5712.187488] [ 5712.189200] ? show_regs.cold.9+0x22/0x2f [ 5712.191494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5712.197494] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5712.201461] ? watchdog+0x40/0x40 [ 5712.203036] ? hrtimer_interrupt+0x128/0x2c0 [ 5712.210581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5712.215410] ? apic_timer_interrupt+0xf/0x20 [ 5712.222641] [ 5712.224905] ? on_each_cpu+0x51/0x120 [ 5712.226404] ? smp_call_function_many_cond+0x3db/0x540 [ 5712.233638] ? smp_call_function_many_cond+0x3a8/0x540 [ 5712.236720] ? load_new_mm_cr3+0x120/0x120 [ 5712.242903] ? load_new_mm_cr3+0x120/0x120 [ 5712.247159] on_each_cpu+0x51/0x120 [ 5712.249921] flush_tlb_kernel_range+0x93/0xb0 [ 5712.255689] free_unmap_vmap_area+0x63/0x70 [ 5712.258501] remove_vm_area+0xe7/0x110 [ 5712.261753] __vunmap+0x9c/0x400 [ 5712.268529] vfree+0x4d/0x100 [ 5712.270937] free_work+0x34/0x50 [libcfs] [ 5712.276745] process_one_work+0x2c8/0x700 [ 5712.280707] worker_thread+0x296/0x6e0 [ 5712.283395] ? rescuer_thread+0x570/0x570 [ 5712.285684] kthread+0x1d1/0x200 [ 5712.292260] ? set_kthread_struct+0x70/0x70 [ 5712.294509] ret_from_fork+0x1f/0x30 [ 5713.376038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3355s! [ 5713.379571] Showing busy workqueues and worker pools: [ 5713.381325] workqueue events: flags=0x0 [ 5713.382797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5713.389174] in-flight: 56293:free_work [libcfs] [ 5713.392071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5713.396081] in-flight: 22277:free_work [libcfs] [ 5713.399400] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5713.406264] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5713.409660] pending: nfs4_renew_state [nfsv4] [ 5713.413352] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5713.417202] in-flight: 47596:free_work [libcfs] [ 5713.422487] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5713.430606] workqueue events_unbound: flags=0x2 [ 5713.438173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5713.444778] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5713.448980] workqueue events_power_efficient: flags=0x80 [ 5713.451521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5713.455364] pending: neigh_periodic_work, do_cache_clean [ 5713.458362] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5713.461511] pending: check_lifetime [ 5713.463776] workqueue mm_percpu_wq: flags=0x8 [ 5713.468773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5713.476645] pending: vmstat_update [ 5713.480547] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5713.487628] pending: vmstat_update [ 5713.489108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5713.492035] pending: vmstat_update [ 5713.494121] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5713.497050] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5713.500372] pending: cgroup_pidlist_destroy_work_fn [ 5713.503529] workqueue writeback: flags=0x4a [ 5713.505472] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5713.507588] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5713.513710] workqueue xprtiod: flags=0xa [ 5713.521089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5713.524582] pending: xprt_autoclose [ 5713.529514] workqueue ipv6_addrconf: flags=0x40008 [ 5713.531524] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5713.536398] pending: addrconf_verify_work [ 5713.538705] workqueue knbd0-recv: flags=0x1a [ 5713.543539] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5713.547581] in-flight: 655:recv_work [ 5713.551083] workqueue ptlrpc_pinger: flags=0x2 [ 5713.553535] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5713.559219] pending: ptlrpc_pinger_main [ptlrpc] [ 5713.562225] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3355s workers=5 idle: 53684 27131 54850 46584 [ 5713.572710] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3355s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5713.578055] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3355s workers=6 idle: 53660 37674 46417 53664 46418 [ 5713.584705] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5703s workers=2 idle: 657 [ 5732.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5732.061373] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5732.090976] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5732.097284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5732.101011] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5732.103282] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5732.115571] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5732.118697] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5732.121517] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5732.124816] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5732.128254] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5732.131282] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5732.134299] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5732.137614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5732.139646] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5732.142534] Call Trace: [ 5732.143923] [ 5732.145066] ? show_regs.cold.9+0x22/0x2f [ 5732.147678] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5732.150264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5732.152840] ? watchdog+0x40/0x40 [ 5732.154696] ? hrtimer_interrupt+0x128/0x2c0 [ 5732.156771] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5732.158535] ? apic_timer_interrupt+0xf/0x20 [ 5732.160757] [ 5732.161966] ? on_each_cpu_cond_mask+0x46/0x150 [ 5732.164031] ? __x64_sys_fspick+0x210/0x210 [ 5732.167165] ? smp_call_function_many_cond+0x3db/0x540 [ 5732.173631] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5732.178015] ? __x64_sys_fspick+0x210/0x210 [ 5732.183606] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5732.187829] on_each_cpu_cond_mask+0x46/0x150 [ 5732.191849] on_each_cpu_cond+0x1c/0x30 [ 5732.193447] invalidate_bh_lrus+0x27/0x30 [ 5732.199904] invalidate_bdev+0x5a/0x90 [ 5732.204958] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5732.208957] generic_shutdown_super+0xb1/0x1b0 [ 5732.213815] kill_block_super+0x29/0x80 [ 5732.217925] deactivate_locked_super+0x52/0xd0 [ 5732.222021] deactivate_super+0x83/0x90 [ 5732.225597] cleanup_mnt+0x5f/0xe0 [ 5732.227480] __cleanup_mnt+0x16/0x20 [ 5732.229103] task_work_run+0xc6/0x110 [ 5732.231003] exit_to_usermode_loop+0x1dd/0x1f0 [ 5732.232495] do_syscall_64+0x3d6/0x440 [ 5732.241001] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5732.244003] RIP: 0033:0x7fbb019648fb [ 5732.246370] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5732.267103] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5732.275556] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5732.278609] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5732.289676] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5732.297112] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5732.300994] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5740.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5740.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5740.059123] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 5740.062876] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 5740.080059] intel_rapl_common [ 5740.080062] dm_flakey rpcsec_gss_krb5 auth_rpcgss [ 5740.095884] sb_edac [ 5740.097538] nfsv4 [ 5740.099867] rapl [ 5740.100872] dns_resolver [ 5740.101731] pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5740.101804] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5740.101810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5740.101816] Workqueue: events free_work [libcfs] [ 5740.101856] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5740.101865] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5740.101870] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5740.101875] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5740.101877] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5740.101879] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5740.101881] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5740.101883] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5740.101886] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5740.101889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5740.101892] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5740.101900] Call Trace: [ 5740.101906] [ 5740.101910] ? show_regs.cold.9+0x22/0x2f [ 5740.101918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5740.101924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5740.101927] ? watchdog+0x40/0x40 [ 5740.101931] ? hrtimer_interrupt+0x128/0x2c0 [ 5740.101935] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5740.101942] ? apic_timer_interrupt+0xf/0x20 [ 5740.101946] [ 5740.101947] ? on_each_cpu+0x51/0x120 [ 5740.101951] ? smp_call_function_many_cond+0x3db/0x540 [ 5740.101954] ? smp_call_function_many_cond+0x3a8/0x540 [ 5740.101957] ? load_new_mm_cr3+0x120/0x120 [ 5740.101962] ? load_new_mm_cr3+0x120/0x120 [ 5740.101965] on_each_cpu+0x51/0x120 [ 5740.101969] flush_tlb_kernel_range+0x93/0xb0 [ 5740.101975] free_unmap_vmap_area+0x63/0x70 [ 5740.101979] remove_vm_area+0xe7/0x110 [ 5740.101983] __vunmap+0x9c/0x400 [ 5740.101986] vfree+0x4d/0x100 [ 5740.101990] free_work+0x34/0x50 [libcfs] [ 5740.102016] process_one_work+0x2c8/0x700 [ 5740.102023] worker_thread+0x296/0x6e0 [ 5740.102027] ? rescuer_thread+0x570/0x570 [ 5740.102031] kthread+0x1d1/0x200 [ 5740.102035] ? set_kthread_struct+0x70/0x70 [ 5740.102038] ret_from_fork+0x1f/0x30 [ 5740.272630] intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5740.295069] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5740.304335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5740.313921] Workqueue: events free_work [libcfs] [ 5740.321210] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5740.325683] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5740.336314] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5740.342884] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5740.346570] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5740.350184] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5740.358754] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5740.364889] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5740.372403] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5740.375338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5740.389101] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5740.394383] Call Trace: [ 5740.396055] [ 5740.397513] ? show_regs.cold.9+0x22/0x2f [ 5740.400073] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5740.406826] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5740.409908] ? watchdog+0x40/0x40 [ 5740.411120] ? hrtimer_interrupt+0x128/0x2c0 [ 5740.419639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5740.423798] ? apic_timer_interrupt+0xf/0x20 [ 5740.426943] [ 5740.429303] ? on_each_cpu+0x51/0x120 [ 5740.430622] ? smp_call_function_many_cond+0x3db/0x540 [ 5740.432353] ? smp_call_function_many_cond+0x3a8/0x540 [ 5740.436463] ? load_new_mm_cr3+0x120/0x120 [ 5740.438986] ? load_new_mm_cr3+0x120/0x120 [ 5740.442711] on_each_cpu+0x51/0x120 [ 5740.446389] flush_tlb_kernel_range+0x93/0xb0 [ 5740.451359] free_unmap_vmap_area+0x63/0x70 [ 5740.453668] remove_vm_area+0xe7/0x110 [ 5740.456176] __vunmap+0x9c/0x400 [ 5740.458114] vfree+0x4d/0x100 [ 5740.459907] free_work+0x34/0x50 [libcfs] [ 5740.462134] process_one_work+0x2c8/0x700 [ 5740.464666] worker_thread+0x296/0x6e0 [ 5740.466939] ? rescuer_thread+0x570/0x570 [ 5740.469156] kthread+0x1d1/0x200 [ 5740.471311] ? set_kthread_struct+0x70/0x70 [ 5740.474021] ret_from_fork+0x1f/0x30 [ 5744.096077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3385s! [ 5744.099906] Showing busy workqueues and worker pools: [ 5744.105714] workqueue events: flags=0x0 [ 5744.107625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5744.110551] in-flight: 56293:free_work [libcfs] [ 5744.112664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5744.115563] in-flight: 22277:free_work [libcfs] [ 5744.117590] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5744.121286] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5744.126978] pending: nfs4_renew_state [nfsv4] [ 5744.131671] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5744.134302] in-flight: 47596:free_work [libcfs] [ 5744.136083] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5744.142670] workqueue events_unbound: flags=0x2 [ 5744.146662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5744.152670] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5744.157766] workqueue events_power_efficient: flags=0x80 [ 5744.160874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5744.164980] pending: neigh_periodic_work, do_cache_clean [ 5744.168849] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5744.173718] pending: check_lifetime [ 5744.177150] workqueue mm_percpu_wq: flags=0x8 [ 5744.180807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5744.183594] pending: vmstat_update [ 5744.187091] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5744.190781] pending: vmstat_update [ 5744.194887] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5744.203076] pending: vmstat_update [ 5744.208292] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5744.213754] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5744.224676] pending: cgroup_pidlist_destroy_work_fn [ 5744.230153] workqueue writeback: flags=0x4a [ 5744.233670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5744.238707] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5744.245764] workqueue xprtiod: flags=0xa [ 5744.248390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5744.252905] pending: xprt_autoclose [ 5744.256256] workqueue ipv6_addrconf: flags=0x40008 [ 5744.263111] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5744.269095] pending: addrconf_verify_work [ 5744.274672] workqueue knbd0-recv: flags=0x1a [ 5744.280088] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5744.286553] in-flight: 655:recv_work [ 5744.287788] workqueue ptlrpc_pinger: flags=0x2 [ 5744.291536] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5744.296420] pending: ptlrpc_pinger_main [ptlrpc] [ 5744.300517] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3386s workers=5 idle: 53684 27131 54850 46584 [ 5744.304658] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3386s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5744.316560] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3386s workers=6 idle: 53660 37674 46417 53664 46418 [ 5744.325589] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5734s workers=2 idle: 657 [ 5760.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5760.066197] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5760.112909] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5760.119630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5760.125360] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5760.128894] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5760.150447] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5760.161102] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5760.170667] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5760.178372] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5760.188230] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5760.194561] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5760.199554] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5760.205742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5760.210932] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5760.220009] Call Trace: [ 5760.221713] [ 5760.223476] ? show_regs.cold.9+0x22/0x2f [ 5760.226537] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5760.229096] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5760.230610] ? watchdog+0x40/0x40 [ 5760.231796] ? hrtimer_interrupt+0x128/0x2c0 [ 5760.233951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5760.236246] ? apic_timer_interrupt+0xf/0x20 [ 5760.238511] [ 5760.239670] ? on_each_cpu_cond_mask+0x46/0x150 [ 5760.242056] ? __x64_sys_fspick+0x210/0x210 [ 5760.244180] ? smp_call_function_many_cond+0x3d9/0x540 [ 5760.248048] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5760.251490] ? __x64_sys_fspick+0x210/0x210 [ 5760.253859] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5760.256103] on_each_cpu_cond_mask+0x46/0x150 [ 5760.258593] on_each_cpu_cond+0x1c/0x30 [ 5760.260202] invalidate_bh_lrus+0x27/0x30 [ 5760.261515] invalidate_bdev+0x5a/0x90 [ 5760.263441] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5760.266512] generic_shutdown_super+0xb1/0x1b0 [ 5760.269438] kill_block_super+0x29/0x80 [ 5760.272046] deactivate_locked_super+0x52/0xd0 [ 5760.275563] deactivate_super+0x83/0x90 [ 5760.278310] cleanup_mnt+0x5f/0xe0 [ 5760.279812] __cleanup_mnt+0x16/0x20 [ 5760.281366] task_work_run+0xc6/0x110 [ 5760.283267] exit_to_usermode_loop+0x1dd/0x1f0 [ 5760.286060] do_syscall_64+0x3d6/0x440 [ 5760.288220] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5760.290892] RIP: 0033:0x7fbb019648fb [ 5760.292869] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5760.301481] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5760.305995] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5760.310262] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5760.314169] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5760.318500] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5760.322202] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5768.061143] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5768.071090] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5768.114881] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5768.125594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5768.134237] Workqueue: events free_work [libcfs] [ 5768.138087] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5768.143225] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5768.157236] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5768.162268] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5768.166235] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5768.171684] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5768.176745] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5768.181906] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5768.190526] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5768.196226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5768.205687] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5768.209472] Call Trace: [ 5768.210823] [ 5768.211810] ? show_regs.cold.9+0x22/0x2f [ 5768.213864] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5768.217152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5768.220866] ? watchdog+0x40/0x40 [ 5768.222969] ? hrtimer_interrupt+0x128/0x2c0 [ 5768.225960] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5768.227647] ? apic_timer_interrupt+0xf/0x20 [ 5768.230264] [ 5768.232023] ? on_each_cpu+0x51/0x120 [ 5768.234025] ? smp_call_function_many_cond+0x3db/0x540 [ 5768.242024] ? smp_call_function_many_cond+0x3a8/0x540 [ 5768.243624] ? load_new_mm_cr3+0x120/0x120 [ 5768.248494] ? load_new_mm_cr3+0x120/0x120 [ 5768.250963] on_each_cpu+0x51/0x120 [ 5768.252791] flush_tlb_kernel_range+0x93/0xb0 [ 5768.254787] free_unmap_vmap_area+0x63/0x70 [ 5768.256574] remove_vm_area+0xe7/0x110 [ 5768.258418] __vunmap+0x9c/0x400 [ 5768.260114] vfree+0x4d/0x100 [ 5768.261906] free_work+0x34/0x50 [libcfs] [ 5768.263833] process_one_work+0x2c8/0x700 [ 5768.265918] worker_thread+0x296/0x6e0 [ 5768.267559] ? rescuer_thread+0x570/0x570 [ 5768.269597] kthread+0x1d1/0x200 [ 5768.271382] ? set_kthread_struct+0x70/0x70 [ 5768.272752] ret_from_fork+0x1f/0x30 [ 5772.055044] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5772.062124] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5772.096753] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5772.101297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5772.105433] Workqueue: events free_work [libcfs] [ 5772.108209] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5772.111416] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5772.121181] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5772.125000] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5772.128729] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5772.132211] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5772.135796] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5772.139070] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5772.142958] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5772.148781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5772.152709] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5772.156600] Call Trace: [ 5772.159653] [ 5772.160970] ? show_regs.cold.9+0x22/0x2f [ 5772.162275] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5772.164855] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5772.167877] ? watchdog+0x40/0x40 [ 5772.169329] ? hrtimer_interrupt+0x128/0x2c0 [ 5772.171162] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5772.173965] ? apic_timer_interrupt+0xf/0x20 [ 5772.176253] [ 5772.176969] ? on_each_cpu+0x51/0x120 [ 5772.179007] ? smp_call_function_many_cond+0x3db/0x540 [ 5772.181333] ? smp_call_function_many_cond+0x3a8/0x540 [ 5772.184272] ? load_new_mm_cr3+0x120/0x120 [ 5772.186192] ? load_new_mm_cr3+0x120/0x120 [ 5772.187922] on_each_cpu+0x51/0x120 [ 5772.189656] flush_tlb_kernel_range+0x93/0xb0 [ 5772.191651] free_unmap_vmap_area+0x63/0x70 [ 5772.193674] remove_vm_area+0xe7/0x110 [ 5772.195828] __vunmap+0x9c/0x400 [ 5772.197283] vfree+0x4d/0x100 [ 5772.198737] free_work+0x34/0x50 [libcfs] [ 5772.200764] process_one_work+0x2c8/0x700 [ 5772.202665] worker_thread+0x296/0x6e0 [ 5772.204408] ? rescuer_thread+0x570/0x570 [ 5772.206315] kthread+0x1d1/0x200 [ 5772.207802] ? set_kthread_struct+0x70/0x70 [ 5772.209835] ret_from_fork+0x1f/0x30 [ 5774.820723] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3416s! [ 5774.828038] Showing busy workqueues and worker pools: [ 5774.831258] workqueue events: flags=0x0 [ 5774.835965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5774.838220] in-flight: 56293:free_work [libcfs] [ 5774.842172] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5774.851830] in-flight: 22277:free_work [libcfs] [ 5774.853490] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5774.861344] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5774.868182] pending: nfs4_renew_state [nfsv4] [ 5774.873160] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5774.879896] in-flight: 47596:free_work [libcfs] [ 5774.882132] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5774.895898] workqueue events_unbound: flags=0x2 [ 5774.898463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5774.900753] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5774.910432] workqueue events_power_efficient: flags=0x80 [ 5774.920483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5774.924793] pending: neigh_periodic_work, do_cache_clean [ 5774.929001] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5774.931973] pending: check_lifetime [ 5774.934793] workqueue mm_percpu_wq: flags=0x8 [ 5774.936683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5774.939613] pending: vmstat_update [ 5774.946937] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5774.951161] pending: vmstat_update [ 5774.952667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5774.955263] pending: vmstat_update [ 5774.958715] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5774.962138] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5774.964898] pending: cgroup_pidlist_destroy_work_fn [ 5774.966901] workqueue writeback: flags=0x4a [ 5774.969831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5774.972607] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5774.978113] workqueue xprtiod: flags=0xa [ 5774.980067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5774.982938] pending: xprt_autoclose [ 5774.984680] workqueue ipv6_addrconf: flags=0x40008 [ 5774.986959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5774.989699] pending: addrconf_verify_work [ 5774.991690] workqueue knbd0-recv: flags=0x1a [ 5774.993185] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5774.995865] in-flight: 655:recv_work [ 5774.997794] workqueue ptlrpc_pinger: flags=0x2 [ 5774.999802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5775.002186] pending: ptlrpc_pinger_main [ptlrpc] [ 5775.004870] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3417s workers=5 idle: 53684 27131 54850 46584 [ 5775.008438] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3417s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5775.013058] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3417s workers=6 idle: 53660 37674 46417 53664 46418 [ 5775.016875] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5764s workers=2 idle: 657 [ 5788.058019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5788.062971] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5788.116926] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5788.122931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5788.129431] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5788.132933] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5788.150382] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5788.155859] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5788.160824] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5788.165045] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5788.173627] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5788.177991] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5788.181860] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5788.186775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5788.190703] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5788.194887] Call Trace: [ 5788.198826] [ 5788.199779] ? show_regs.cold.9+0x22/0x2f [ 5788.201969] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5788.203884] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5788.205676] ? watchdog+0x40/0x40 [ 5788.207620] ? hrtimer_interrupt+0x128/0x2c0 [ 5788.209377] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5788.210749] ? apic_timer_interrupt+0xf/0x20 [ 5788.212564] [ 5788.213070] ? on_each_cpu_cond_mask+0x46/0x150 [ 5788.214823] ? __x64_sys_fspick+0x210/0x210 [ 5788.216508] ? smp_call_function_many_cond+0x3db/0x540 [ 5788.218027] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5788.219746] ? __x64_sys_fspick+0x210/0x210 [ 5788.221045] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5788.222738] on_each_cpu_cond_mask+0x46/0x150 [ 5788.224359] on_each_cpu_cond+0x1c/0x30 [ 5788.225846] invalidate_bh_lrus+0x27/0x30 [ 5788.227508] invalidate_bdev+0x5a/0x90 [ 5788.228864] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5788.230898] generic_shutdown_super+0xb1/0x1b0 [ 5788.232763] kill_block_super+0x29/0x80 [ 5788.234533] deactivate_locked_super+0x52/0xd0 [ 5788.235995] deactivate_super+0x83/0x90 [ 5788.237732] cleanup_mnt+0x5f/0xe0 [ 5788.238972] __cleanup_mnt+0x16/0x20 [ 5788.240446] task_work_run+0xc6/0x110 [ 5788.241826] exit_to_usermode_loop+0x1dd/0x1f0 [ 5788.243713] do_syscall_64+0x3d6/0x440 [ 5788.244912] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5788.246816] RIP: 0033:0x7fbb019648fb [ 5788.247963] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5788.254876] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5788.257919] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5788.260709] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5788.263026] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5788.265324] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5788.267994] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5796.061913] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 5796.075595] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5796.111230] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5796.118132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5796.123077] Workqueue: events free_work [libcfs] [ 5796.125191] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5796.127137] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5796.145251] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5796.153702] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5796.158950] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5796.164604] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5796.170917] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5796.174210] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5796.177499] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5796.181275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5796.183985] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5796.187478] Call Trace: [ 5796.188346] [ 5796.189348] ? show_regs.cold.9+0x22/0x2f [ 5796.191328] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5796.193413] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5796.195206] ? watchdog+0x40/0x40 [ 5796.196526] ? hrtimer_interrupt+0x128/0x2c0 [ 5796.198423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5796.200444] ? apic_timer_interrupt+0xf/0x20 [ 5796.202251] [ 5796.203179] ? on_each_cpu+0x51/0x120 [ 5796.204674] ? smp_call_function_many_cond+0x3db/0x540 [ 5796.206994] ? smp_call_function_many_cond+0x3a8/0x540 [ 5796.209343] ? load_new_mm_cr3+0x120/0x120 [ 5796.211150] ? load_new_mm_cr3+0x120/0x120 [ 5796.212798] on_each_cpu+0x51/0x120 [ 5796.214483] flush_tlb_kernel_range+0x93/0xb0 [ 5796.216462] free_unmap_vmap_area+0x63/0x70 [ 5796.217963] remove_vm_area+0xe7/0x110 [ 5796.219543] __vunmap+0x9c/0x400 [ 5796.220986] vfree+0x4d/0x100 [ 5796.222549] free_work+0x34/0x50 [libcfs] [ 5796.226099] process_one_work+0x2c8/0x700 [ 5796.229938] worker_thread+0x296/0x6e0 [ 5796.232253] ? rescuer_thread+0x570/0x570 [ 5796.237303] kthread+0x1d1/0x200 [ 5796.240562] ? set_kthread_struct+0x70/0x70 [ 5796.245600] ret_from_fork+0x1f/0x30 [ 5800.055015] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5800.058666] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5800.088186] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5800.093784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5800.097474] Workqueue: events free_work [libcfs] [ 5800.099528] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5800.102330] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5800.109724] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5800.113191] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5800.115988] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5800.118848] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5800.121856] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5800.124620] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5800.128830] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5800.131894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5800.135281] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5800.139877] Call Trace: [ 5800.141296] [ 5800.142391] ? show_regs.cold.9+0x22/0x2f [ 5800.144536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5800.152028] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5800.155889] ? watchdog+0x40/0x40 [ 5800.158087] ? hrtimer_interrupt+0x128/0x2c0 [ 5800.162371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5800.166365] ? apic_timer_interrupt+0xf/0x20 [ 5800.173406] [ 5800.175307] ? on_each_cpu+0x51/0x120 [ 5800.178280] ? smp_call_function_many_cond+0x3db/0x540 [ 5800.182782] ? smp_call_function_many_cond+0x3a8/0x540 [ 5800.185941] ? load_new_mm_cr3+0x120/0x120 [ 5800.188312] ? load_new_mm_cr3+0x120/0x120 [ 5800.190461] on_each_cpu+0x51/0x120 [ 5800.196474] flush_tlb_kernel_range+0x93/0xb0 [ 5800.198588] free_unmap_vmap_area+0x63/0x70 [ 5800.202554] remove_vm_area+0xe7/0x110 [ 5800.204045] __vunmap+0x9c/0x400 [ 5800.206549] vfree+0x4d/0x100 [ 5800.208252] free_work+0x34/0x50 [libcfs] [ 5800.218523] process_one_work+0x2c8/0x700 [ 5800.222176] worker_thread+0x296/0x6e0 [ 5800.224297] ? rescuer_thread+0x570/0x570 [ 5800.226702] kthread+0x1d1/0x200 [ 5800.231358] ? set_kthread_struct+0x70/0x70 [ 5800.234995] ret_from_fork+0x1f/0x30 [ 5805.024023] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3446s! [ 5805.026239] Showing busy workqueues and worker pools: [ 5805.027659] workqueue events: flags=0x0 [ 5805.028817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5805.030745] in-flight: 56293:free_work [libcfs] [ 5805.032226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5805.034172] in-flight: 22277:free_work [libcfs] [ 5805.035592] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5805.038179] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5805.040134] pending: nfs4_renew_state [nfsv4] [ 5805.041566] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5805.043544] in-flight: 47596:free_work [libcfs] [ 5805.045025] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5805.047690] workqueue events_unbound: flags=0x2 [ 5805.049064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5805.050752] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5805.053199] workqueue events_power_efficient: flags=0x80 [ 5805.054727] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5805.056682] pending: neigh_periodic_work, do_cache_clean [ 5805.058397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5805.060436] pending: check_lifetime [ 5805.061562] workqueue mm_percpu_wq: flags=0x8 [ 5805.062902] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5805.064819] pending: vmstat_update [ 5805.066023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5805.067946] pending: vmstat_update [ 5805.069029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5805.070966] pending: vmstat_update [ 5805.072089] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5805.073535] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5805.075678] pending: cgroup_pidlist_destroy_work_fn [ 5805.077780] workqueue writeback: flags=0x4a [ 5805.079545] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5805.082240] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5805.084901] workqueue xprtiod: flags=0xa [ 5805.086570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5805.089125] pending: xprt_autoclose [ 5805.090624] workqueue ipv6_addrconf: flags=0x40008 [ 5805.092623] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5805.095283] pending: addrconf_verify_work [ 5805.096591] workqueue knbd0-recv: flags=0x1a [ 5805.097730] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5805.100325] in-flight: 655:recv_work [ 5805.102098] workqueue ptlrpc_pinger: flags=0x2 [ 5805.103819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5805.106287] pending: ptlrpc_pinger_main [ptlrpc] [ 5805.108651] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3447s workers=5 idle: 53684 27131 54850 46584 [ 5805.112976] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3447s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5805.122737] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3447s workers=6 idle: 53660 37674 46417 53664 46418 [ 5805.132141] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5794s workers=2 idle: 657 [ 5816.060431] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5816.064650] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5816.094830] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5816.105464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5816.110200] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5816.113166] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5816.122526] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5816.125847] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5816.129041] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5816.134424] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5816.137452] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5816.140872] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5816.143767] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5816.146864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5816.148923] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5816.151919] Call Trace: [ 5816.153044] [ 5816.153879] ? show_regs.cold.9+0x22/0x2f [ 5816.155705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5816.159530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5816.161966] ? watchdog+0x40/0x40 [ 5816.163858] ? hrtimer_interrupt+0x128/0x2c0 [ 5816.168085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5816.170245] ? apic_timer_interrupt+0xf/0x20 [ 5816.172599] [ 5816.174182] ? on_each_cpu_cond_mask+0x46/0x150 [ 5816.178869] ? __x64_sys_fspick+0x210/0x210 [ 5816.182893] ? smp_call_function_many_cond+0x3d9/0x540 [ 5816.188397] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5816.191868] ? __x64_sys_fspick+0x210/0x210 [ 5816.195503] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5816.199798] on_each_cpu_cond_mask+0x46/0x150 [ 5816.203518] on_each_cpu_cond+0x1c/0x30 [ 5816.206505] invalidate_bh_lrus+0x27/0x30 [ 5816.209910] invalidate_bdev+0x5a/0x90 [ 5816.213249] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5816.217360] generic_shutdown_super+0xb1/0x1b0 [ 5816.221756] kill_block_super+0x29/0x80 [ 5816.224822] deactivate_locked_super+0x52/0xd0 [ 5816.228263] deactivate_super+0x83/0x90 [ 5816.231133] cleanup_mnt+0x5f/0xe0 [ 5816.232533] __cleanup_mnt+0x16/0x20 [ 5816.235087] task_work_run+0xc6/0x110 [ 5816.239777] exit_to_usermode_loop+0x1dd/0x1f0 [ 5816.243870] do_syscall_64+0x3d6/0x440 [ 5816.249552] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5816.254144] RIP: 0033:0x7fbb019648fb [ 5816.256717] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5816.273113] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5816.278933] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5816.285030] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5816.290927] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5816.296187] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5816.302212] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5824.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 5824.062902] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5824.083893] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5824.087577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5824.090932] Workqueue: events free_work [libcfs] [ 5824.093256] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5824.096004] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5824.103540] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5824.110231] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5824.115909] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5824.121514] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5824.126107] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5824.130956] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5824.139019] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5824.144882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5824.151905] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5824.157699] Call Trace: [ 5824.158678] [ 5824.162580] ? show_regs.cold.9+0x22/0x2f [ 5824.167840] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5824.172427] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5824.174852] ? watchdog+0x40/0x40 [ 5824.176621] ? hrtimer_interrupt+0x128/0x2c0 [ 5824.178363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5824.191975] ? apic_timer_interrupt+0xf/0x20 [ 5824.196914] [ 5824.198954] ? on_each_cpu+0x51/0x120 [ 5824.202922] ? smp_call_function_many_cond+0x3db/0x540 [ 5824.207620] ? smp_call_function_many_cond+0x3a8/0x540 [ 5824.215260] ? load_new_mm_cr3+0x120/0x120 [ 5824.221091] ? load_new_mm_cr3+0x120/0x120 [ 5824.225690] on_each_cpu+0x51/0x120 [ 5824.228375] flush_tlb_kernel_range+0x93/0xb0 [ 5824.233344] free_unmap_vmap_area+0x63/0x70 [ 5824.236923] remove_vm_area+0xe7/0x110 [ 5824.240356] __vunmap+0x9c/0x400 [ 5824.243987] vfree+0x4d/0x100 [ 5824.248115] free_work+0x34/0x50 [libcfs] [ 5824.252792] process_one_work+0x2c8/0x700 [ 5824.254965] worker_thread+0x296/0x6e0 [ 5824.257178] ? rescuer_thread+0x570/0x570 [ 5824.261346] kthread+0x1d1/0x200 [ 5824.263336] ? set_kthread_struct+0x70/0x70 [ 5824.266003] ret_from_fork+0x1f/0x30 [ 5832.055045] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5832.058433] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5832.082686] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5832.088220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5832.092044] Workqueue: events free_work [libcfs] [ 5832.093646] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5832.096083] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5832.106295] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5832.110112] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5832.113961] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5832.117343] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5832.120761] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5832.125291] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5832.129085] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5832.133166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5832.137307] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5832.140683] Call Trace: [ 5832.142025] [ 5832.142770] ? show_regs.cold.9+0x22/0x2f [ 5832.144798] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5832.147235] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5832.150714] ? watchdog+0x40/0x40 [ 5832.154323] ? hrtimer_interrupt+0x128/0x2c0 [ 5832.156372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5832.158531] ? apic_timer_interrupt+0xf/0x20 [ 5832.159861] [ 5832.160618] ? on_each_cpu+0x51/0x120 [ 5832.162024] ? smp_call_function_many_cond+0x3db/0x540 [ 5832.164985] ? smp_call_function_many_cond+0x3a8/0x540 [ 5832.167194] ? load_new_mm_cr3+0x120/0x120 [ 5832.169140] ? load_new_mm_cr3+0x120/0x120 [ 5832.172841] on_each_cpu+0x51/0x120 [ 5832.177420] flush_tlb_kernel_range+0x93/0xb0 [ 5832.180803] free_unmap_vmap_area+0x63/0x70 [ 5832.185661] remove_vm_area+0xe7/0x110 [ 5832.190152] __vunmap+0x9c/0x400 [ 5832.192346] vfree+0x4d/0x100 [ 5832.194732] free_work+0x34/0x50 [libcfs] [ 5832.198789] process_one_work+0x2c8/0x700 [ 5832.203618] worker_thread+0x296/0x6e0 [ 5832.206710] ? rescuer_thread+0x570/0x570 [ 5832.209288] kthread+0x1d1/0x200 [ 5832.210424] ? set_kthread_struct+0x70/0x70 [ 5832.212174] ret_from_fork+0x1f/0x30 [ 5835.232028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3476s! [ 5835.235172] Showing busy workqueues and worker pools: [ 5835.237674] workqueue events: flags=0x0 [ 5835.239483] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5835.245667] in-flight: 56293:free_work [libcfs] [ 5835.248145] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5835.251524] in-flight: 22277:free_work [libcfs] [ 5835.253738] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5835.259279] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5835.262967] pending: nfs4_renew_state [nfsv4] [ 5835.268822] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5835.272932] in-flight: 47596:free_work [libcfs] [ 5835.275692] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5835.281811] workqueue events_unbound: flags=0x2 [ 5835.284405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5835.287125] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5835.294780] workqueue events_power_efficient: flags=0x80 [ 5835.301375] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5835.304921] pending: neigh_periodic_work, do_cache_clean [ 5835.308453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5835.312807] pending: check_lifetime [ 5835.314961] workqueue mm_percpu_wq: flags=0x8 [ 5835.317435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5835.320736] pending: vmstat_update [ 5835.323438] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5835.326931] pending: vmstat_update [ 5835.329534] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5835.333144] pending: vmstat_update [ 5835.335283] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5835.338810] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5835.349759] pending: cgroup_pidlist_destroy_work_fn [ 5835.354024] workqueue writeback: flags=0x4a [ 5835.357492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5835.362434] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5835.367484] workqueue xprtiod: flags=0xa [ 5835.369992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5835.373672] pending: xprt_autoclose [ 5835.376476] workqueue ipv6_addrconf: flags=0x40008 [ 5835.380498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5835.384977] pending: addrconf_verify_work [ 5835.388283] workqueue knbd0-recv: flags=0x1a [ 5835.389945] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5835.395466] in-flight: 655:recv_work [ 5835.399631] workqueue ptlrpc_pinger: flags=0x2 [ 5835.407830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5835.412371] pending: ptlrpc_pinger_main [ptlrpc] [ 5835.421489] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3477s workers=5 idle: 53684 27131 54850 46584 [ 5835.436024] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3477s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5835.447995] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3477s workers=6 idle: 53660 37674 46417 53664 46418 [ 5835.463056] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5825s workers=2 idle: 657 [ 5837.842405] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 5837.853883] rcu: 3-...!: (1 GPs behind) idle=1b6/1/0x4000000000000002 softirq=303161/303162 fqs=49 [ 5837.867438] (detected by 2, t=3480112 jiffies, g=626849, q=1398) [ 5837.871737] Sending NMI from CPU 2 to CPUs 3: [ 5837.874756] NMI backtrace for cpu 3 skipped: idling at native_halt+0x2d/0x40 [ 5837.874764] rcu: rcu_sched kthread starved for 3479754 jiffies! g626849 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5837.892888] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5837.898892] rcu: RCU grace-period kthread stack dump: [ 5837.906905] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5837.914907] Call Trace: [ 5837.915903] __schedule+0x351/0xcb0 [ 5837.919641] schedule+0xc0/0x180 [ 5837.921182] schedule_timeout+0xb4/0x190 [ 5837.923965] ? do_raw_spin_unlock+0x75/0x190 [ 5837.926714] ? __next_timer_interrupt+0x160/0x160 [ 5837.929766] ? prepare_to_swait_event+0xf3/0x140 [ 5837.933469] rcu_gp_kthread+0x8dd/0x15b0 [ 5837.935705] ? force_qs_rnp+0x2c0/0x2c0 [ 5837.937583] kthread+0x1d1/0x200 [ 5837.938902] ? set_kthread_struct+0x70/0x70 [ 5837.940880] ret_from_fork+0x1f/0x30 [ 5837.942468] rcu: Stack dump where RCU GP kthread last ran: [ 5837.944607] NMI backtrace for cpu 2 [ 5837.945845] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5837.950908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5837.953918] Workqueue: events free_work [libcfs] [ 5837.959877] Call Trace: [ 5837.961695] [ 5837.962698] ? dump_stack+0xbb/0x10e [ 5837.964909] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5837.976875] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5837.980055] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5837.983766] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5837.986558] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5837.989844] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5837.994711] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 5837.999898] ? update_process_times+0x6d/0xc0 [ 5838.004546] ? tick_sched_handle.isra.18+0x27/0x90 [ 5838.007950] ? tick_sched_timer+0x53/0xb0 [ 5838.010055] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5838.015375] ? tick_sched_handle.isra.18+0x90/0x90 [ 5838.018978] ? hrtimer_interrupt+0x128/0x2c0 [ 5838.024194] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5838.030966] ? apic_timer_interrupt+0xf/0x20 [ 5838.035830] [ 5838.042504] ? on_each_cpu+0x51/0x120 [ 5838.043692] ? smp_call_function_many_cond+0x3db/0x540 [ 5838.045146] ? smp_call_function_many_cond+0x3a8/0x540 [ 5838.058864] ? load_new_mm_cr3+0x120/0x120 [ 5838.061598] ? load_new_mm_cr3+0x120/0x120 [ 5838.063166] ? on_each_cpu+0x51/0x120 [ 5838.066489] ? flush_tlb_kernel_range+0x93/0xb0 [ 5838.070808] ? free_unmap_vmap_area+0x63/0x70 [ 5838.075419] ? remove_vm_area+0xe7/0x110 [ 5838.076890] ? __vunmap+0x9c/0x400 [ 5838.078212] ? vfree+0x4d/0x100 [ 5838.082757] ? free_work+0x34/0x50 [libcfs] [ 5838.085932] ? process_one_work+0x2c8/0x700 [ 5838.088944] ? worker_thread+0x296/0x6e0 [ 5838.095797] ? rescuer_thread+0x570/0x570 [ 5838.098860] ? kthread+0x1d1/0x200 [ 5838.105474] ? set_kthread_struct+0x70/0x70 [ 5838.107093] ? ret_from_fork+0x1f/0x30 [ 5844.062392] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5844.073617] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5844.144816] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5844.159668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5844.165410] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5844.171431] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5844.202717] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5844.206831] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5844.215445] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5844.222370] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5844.228058] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5844.240301] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5844.247514] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5844.252567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5844.255937] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5844.264899] Call Trace: [ 5844.268096] [ 5844.268704] ? show_regs.cold.9+0x22/0x2f [ 5844.278849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5844.284306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5844.299007] ? watchdog+0x40/0x40 [ 5844.300225] ? hrtimer_interrupt+0x128/0x2c0 [ 5844.311522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5844.315741] ? apic_timer_interrupt+0xf/0x20 [ 5844.320403] [ 5844.324376] ? on_each_cpu_cond_mask+0x46/0x150 [ 5844.332349] ? __x64_sys_fspick+0x210/0x210 [ 5844.333849] ? smp_call_function_many_cond+0x3db/0x540 [ 5844.353775] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5844.361603] ? __x64_sys_fspick+0x210/0x210 [ 5844.370660] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5844.386310] on_each_cpu_cond_mask+0x46/0x150 [ 5844.391567] on_each_cpu_cond+0x1c/0x30 [ 5844.395746] invalidate_bh_lrus+0x27/0x30 [ 5844.397161] invalidate_bdev+0x5a/0x90 [ 5844.402517] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5844.414892] generic_shutdown_super+0xb1/0x1b0 [ 5844.417172] kill_block_super+0x29/0x80 [ 5844.425140] deactivate_locked_super+0x52/0xd0 [ 5844.428904] deactivate_super+0x83/0x90 [ 5844.437898] cleanup_mnt+0x5f/0xe0 [ 5844.440952] __cleanup_mnt+0x16/0x20 [ 5844.449961] task_work_run+0xc6/0x110 [ 5844.451221] exit_to_usermode_loop+0x1dd/0x1f0 [ 5844.464664] do_syscall_64+0x3d6/0x440 [ 5844.465995] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5844.478279] RIP: 0033:0x7fbb019648fb [ 5844.479547] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5844.498381] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5844.515975] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5844.526826] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5844.540408] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5844.553005] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5844.565776] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5860.055955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5860.059483] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5860.091248] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5860.098514] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5860.102185] Workqueue: events free_work [libcfs] [ 5860.104158] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5860.106507] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5860.114823] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5860.117724] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5860.120531] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5860.123480] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5860.126591] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5860.129474] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5860.132966] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5860.137450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5860.139983] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5860.145755] Call Trace: [ 5860.147474] [ 5860.149785] ? show_regs.cold.9+0x22/0x2f [ 5860.151780] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5860.154889] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5860.157197] ? watchdog+0x40/0x40 [ 5860.162236] ? hrtimer_interrupt+0x128/0x2c0 [ 5860.164866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5860.166811] ? apic_timer_interrupt+0xf/0x20 [ 5860.168863] [ 5860.169843] ? on_each_cpu+0x51/0x120 [ 5860.171399] ? smp_call_function_many_cond+0x3db/0x540 [ 5860.173209] ? smp_call_function_many_cond+0x3a8/0x540 [ 5860.178138] ? load_new_mm_cr3+0x120/0x120 [ 5860.179796] ? load_new_mm_cr3+0x120/0x120 [ 5860.181305] on_each_cpu+0x51/0x120 [ 5860.182284] flush_tlb_kernel_range+0x93/0xb0 [ 5860.186780] free_unmap_vmap_area+0x63/0x70 [ 5860.189103] remove_vm_area+0xe7/0x110 [ 5860.193224] __vunmap+0x9c/0x400 [ 5860.194603] vfree+0x4d/0x100 [ 5860.198765] free_work+0x34/0x50 [libcfs] [ 5860.200838] process_one_work+0x2c8/0x700 [ 5860.206283] worker_thread+0x296/0x6e0 [ 5860.210991] ? rescuer_thread+0x570/0x570 [ 5860.215562] kthread+0x1d1/0x200 [ 5860.219745] ? set_kthread_struct+0x70/0x70 [ 5860.222178] ret_from_fork+0x1f/0x30 [ 5864.059065] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:6:22277] [ 5864.064084] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.102823] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.120510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5864.127058] Workqueue: events free_work [libcfs] [ 5864.136072] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5864.138885] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5864.147981] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5864.152555] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5864.158728] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5864.162179] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5864.165621] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5864.175106] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5864.190081] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5864.199827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.202426] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5864.214194] Call Trace: [ 5864.216862] [ 5864.219882] ? show_regs.cold.9+0x22/0x2f [ 5864.222888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.228097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.231058] ? watchdog+0x40/0x40 [ 5864.234033] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.236109] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.242077] ? apic_timer_interrupt+0xf/0x20 [ 5864.247944] [ 5864.250034] ? on_each_cpu+0x51/0x120 [ 5864.252281] ? smp_call_function_many_cond+0x3d9/0x540 [ 5864.255558] ? smp_call_function_many_cond+0x3a8/0x540 [ 5864.260638] ? load_new_mm_cr3+0x120/0x120 [ 5864.264048] ? load_new_mm_cr3+0x120/0x120 [ 5864.267184] on_each_cpu+0x51/0x120 [ 5864.268942] flush_tlb_kernel_range+0x93/0xb0 [ 5864.273413] free_unmap_vmap_area+0x63/0x70 [ 5864.276240] remove_vm_area+0xe7/0x110 [ 5864.281292] __vunmap+0x9c/0x400 [ 5864.285235] vfree+0x4d/0x100 [ 5864.286368] free_work+0x34/0x50 [libcfs] [ 5864.290401] process_one_work+0x2c8/0x700 [ 5864.293322] worker_thread+0x296/0x6e0 [ 5864.296206] ? rescuer_thread+0x570/0x570 [ 5864.299004] kthread+0x1d1/0x200 [ 5864.300390] ? set_kthread_struct+0x70/0x70 [ 5864.303446] ret_from_fork+0x1f/0x30 [ 5865.952025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3507s! [ 5865.956120] Showing busy workqueues and worker pools: [ 5865.958414] workqueue events: flags=0x0 [ 5865.960068] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5865.962900] in-flight: 56293:free_work [libcfs] [ 5865.965164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5865.967932] in-flight: 22277:free_work [libcfs] [ 5865.970012] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5865.973975] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5865.976301] pending: nfs4_renew_state [nfsv4] [ 5865.978516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5865.982306] in-flight: 47596:free_work [libcfs] [ 5865.985440] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5865.990277] workqueue events_unbound: flags=0x2 [ 5865.993233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5865.996815] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5866.000872] workqueue events_power_efficient: flags=0x80 [ 5866.003750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5866.007296] pending: neigh_periodic_work, do_cache_clean [ 5866.010224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5866.013075] pending: check_lifetime [ 5866.014919] workqueue mm_percpu_wq: flags=0x8 [ 5866.017784] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5866.021199] pending: vmstat_update [ 5866.023037] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5866.025987] pending: vmstat_update [ 5866.028132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5866.032049] pending: vmstat_update [ 5866.034109] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5866.041752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5866.044313] pending: cgroup_pidlist_destroy_work_fn [ 5866.053361] workqueue writeback: flags=0x4a [ 5866.058973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5866.066046] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5866.072783] workqueue xprtiod: flags=0xa [ 5866.075363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5866.078707] pending: xprt_autoclose [ 5866.081064] workqueue ipv6_addrconf: flags=0x40008 [ 5866.083544] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5866.087870] pending: addrconf_verify_work [ 5866.089869] workqueue knbd0-recv: flags=0x1a [ 5866.091902] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5866.095713] in-flight: 655:recv_work [ 5866.097766] workqueue ptlrpc_pinger: flags=0x2 [ 5866.100259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5866.104473] pending: ptlrpc_pinger_main [ptlrpc] [ 5866.108778] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3508s workers=5 idle: 53684 27131 54850 46584 [ 5866.114131] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3508s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5866.130125] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3508s workers=6 idle: 53660 37674 46417 53664 46418 [ 5866.141782] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5855s workers=2 idle: 657 [ 5872.058008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5872.063323] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5872.106867] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5872.111794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5872.116298] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5872.119037] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5872.127283] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5872.130843] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5872.137765] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5872.141623] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5872.144543] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5872.147850] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5872.151115] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5872.155132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5872.157146] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5872.161132] Call Trace: [ 5872.162839] [ 5872.163880] ? show_regs.cold.9+0x22/0x2f [ 5872.165702] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5872.169248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5872.174113] ? watchdog+0x40/0x40 [ 5872.181346] ? hrtimer_interrupt+0x128/0x2c0 [ 5872.182885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5872.186087] ? apic_timer_interrupt+0xf/0x20 [ 5872.188660] [ 5872.189737] ? on_each_cpu_cond_mask+0x46/0x150 [ 5872.193078] ? __x64_sys_fspick+0x210/0x210 [ 5872.195980] ? smp_call_function_many_cond+0x3d9/0x540 [ 5872.198662] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5872.203244] ? __x64_sys_fspick+0x210/0x210 [ 5872.204588] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5872.213749] on_each_cpu_cond_mask+0x46/0x150 [ 5872.217092] on_each_cpu_cond+0x1c/0x30 [ 5872.219521] invalidate_bh_lrus+0x27/0x30 [ 5872.223316] invalidate_bdev+0x5a/0x90 [ 5872.227114] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5872.237310] generic_shutdown_super+0xb1/0x1b0 [ 5872.243876] kill_block_super+0x29/0x80 [ 5872.246900] deactivate_locked_super+0x52/0xd0 [ 5872.250788] deactivate_super+0x83/0x90 [ 5872.254117] cleanup_mnt+0x5f/0xe0 [ 5872.256825] __cleanup_mnt+0x16/0x20 [ 5872.259000] task_work_run+0xc6/0x110 [ 5872.261832] exit_to_usermode_loop+0x1dd/0x1f0 [ 5872.268041] do_syscall_64+0x3d6/0x440 [ 5872.269487] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5872.272591] RIP: 0033:0x7fbb019648fb [ 5872.274765] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5872.289056] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5872.296130] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5872.302786] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5872.313119] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5872.317787] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5872.322300] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5892.055379] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:47596] [ 5892.060346] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5892.060371] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.060445] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.060451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5892.060456] Workqueue: events free_work [libcfs] [ 5892.060497] RIP: 0010:smp_call_function_many_cond+0x3d9/0x540 [ 5892.060505] Code: 83 fc 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 90 8b 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 [ 5892.060510] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5892.060514] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5892.060517] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5892.060519] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5892.060521] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5892.060523] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5892.060525] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5892.060528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.060530] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5892.060540] Call Trace: [ 5892.060545] [ 5892.060550] ? show_regs.cold.9+0x22/0x2f [ 5892.060559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.060566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.060570] ? watchdog+0x40/0x40 [ 5892.060574] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.060578] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.060588] ? apic_timer_interrupt+0xf/0x20 [ 5892.060593] [ 5892.060595] ? on_each_cpu+0x51/0x120 [ 5892.060600] ? smp_call_function_many_cond+0x3d9/0x540 [ 5892.060603] ? smp_call_function_many_cond+0x3a8/0x540 [ 5892.060607] ? load_new_mm_cr3+0x120/0x120 [ 5892.060612] ? load_new_mm_cr3+0x120/0x120 [ 5892.060615] on_each_cpu+0x51/0x120 [ 5892.060619] flush_tlb_kernel_range+0x93/0xb0 [ 5892.060625] free_unmap_vmap_area+0x63/0x70 [ 5892.060630] remove_vm_area+0xe7/0x110 [ 5892.060634] __vunmap+0x9c/0x400 [ 5892.060638] vfree+0x4d/0x100 [ 5892.060643] free_work+0x34/0x50 [libcfs] [ 5892.060667] process_one_work+0x2c8/0x700 [ 5892.060674] worker_thread+0x296/0x6e0 [ 5892.060679] ? rescuer_thread+0x570/0x570 [ 5892.060683] kthread+0x1d1/0x200 [ 5892.060688] ? set_kthread_struct+0x70/0x70 [ 5892.060692] ret_from_fork+0x1f/0x30 [ 5892.267019] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.298788] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.310528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5892.317450] Workqueue: events free_work [libcfs] [ 5892.322229] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5892.325803] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5892.348894] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5892.355566] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5892.358155] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5892.372018] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5892.380331] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5892.385982] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5892.388708] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5892.394706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.398330] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5892.405261] Call Trace: [ 5892.406529] [ 5892.409400] ? show_regs.cold.9+0x22/0x2f [ 5892.410907] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.413789] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.416169] ? watchdog+0x40/0x40 [ 5892.418201] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.420794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.424251] ? apic_timer_interrupt+0xf/0x20 [ 5892.428801] [ 5892.430314] ? on_each_cpu+0x51/0x120 [ 5892.433087] ? smp_call_function_many_cond+0x3db/0x540 [ 5892.438781] ? smp_call_function_many_cond+0x3a8/0x540 [ 5892.443408] ? load_new_mm_cr3+0x120/0x120 [ 5892.447813] ? load_new_mm_cr3+0x120/0x120 [ 5892.450610] on_each_cpu+0x51/0x120 [ 5892.452012] flush_tlb_kernel_range+0x93/0xb0 [ 5892.453612] free_unmap_vmap_area+0x63/0x70 [ 5892.456329] remove_vm_area+0xe7/0x110 [ 5892.458515] __vunmap+0x9c/0x400 [ 5892.460821] vfree+0x4d/0x100 [ 5892.462799] free_work+0x34/0x50 [libcfs] [ 5892.467344] process_one_work+0x2c8/0x700 [ 5892.472821] worker_thread+0x296/0x6e0 [ 5892.475538] ? rescuer_thread+0x570/0x570 [ 5892.479694] kthread+0x1d1/0x200 [ 5892.482257] ? set_kthread_struct+0x70/0x70 [ 5892.484831] ret_from_fork+0x1f/0x30 [ 5896.160028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3537s! [ 5896.163767] Showing busy workqueues and worker pools: [ 5896.169185] workqueue events: flags=0x0 [ 5896.171178] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5896.176065] in-flight: 56293:free_work [libcfs] [ 5896.179424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5896.182066] in-flight: 22277:free_work [libcfs] [ 5896.184506] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5896.193976] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5896.196407] pending: nfs4_renew_state [nfsv4] [ 5896.200752] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5896.204678] in-flight: 47596:free_work [libcfs] [ 5896.213107] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5896.218262] workqueue events_unbound: flags=0x2 [ 5896.221894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5896.227994] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5896.236638] workqueue events_power_efficient: flags=0x80 [ 5896.238646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5896.249232] pending: neigh_periodic_work, do_cache_clean [ 5896.253768] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5896.259585] pending: check_lifetime [ 5896.262571] workqueue mm_percpu_wq: flags=0x8 [ 5896.266266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5896.272652] pending: vmstat_update [ 5896.276221] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5896.284990] pending: vmstat_update [ 5896.287889] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5896.292168] pending: vmstat_update [ 5896.297767] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5896.305664] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5896.311759] pending: cgroup_pidlist_destroy_work_fn [ 5896.316353] workqueue writeback: flags=0x4a [ 5896.317987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5896.328816] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5896.336373] workqueue xprtiod: flags=0xa [ 5896.337892] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5896.348092] pending: xprt_autoclose [ 5896.351089] workqueue ipv6_addrconf: flags=0x40008 [ 5896.354016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5896.358562] pending: addrconf_verify_work [ 5896.360839] workqueue knbd0-recv: flags=0x1a [ 5896.363784] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5896.369356] in-flight: 655:recv_work [ 5896.373420] workqueue ptlrpc_pinger: flags=0x2 [ 5896.375134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5896.386685] pending: ptlrpc_pinger_main [ptlrpc] [ 5896.388441] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3538s workers=5 idle: 53684 27131 54850 46584 [ 5896.402016] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3538s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5896.417437] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3538s workers=6 idle: 53660 37674 46417 53664 46418 [ 5896.429715] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5886s workers=2 idle: 657 [ 5900.058015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5900.060658] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5900.090911] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5900.097026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5900.103092] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5900.106138] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5900.118516] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5900.123698] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5900.127793] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5900.130601] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5900.136921] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5900.141704] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5900.145631] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5900.150539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5900.154200] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5900.158534] Call Trace: [ 5900.160179] [ 5900.161603] ? show_regs.cold.9+0x22/0x2f [ 5900.163712] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5900.166883] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5900.169697] ? watchdog+0x40/0x40 [ 5900.171639] ? hrtimer_interrupt+0x128/0x2c0 [ 5900.174186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5900.177050] ? apic_timer_interrupt+0xf/0x20 [ 5900.179638] [ 5900.180906] ? on_each_cpu_cond_mask+0x46/0x150 [ 5900.184022] ? __x64_sys_fspick+0x210/0x210 [ 5900.186714] ? smp_call_function_many_cond+0x3db/0x540 [ 5900.188752] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5900.191257] ? __x64_sys_fspick+0x210/0x210 [ 5900.193931] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5900.196591] on_each_cpu_cond_mask+0x46/0x150 [ 5900.201144] on_each_cpu_cond+0x1c/0x30 [ 5900.204685] invalidate_bh_lrus+0x27/0x30 [ 5900.208923] invalidate_bdev+0x5a/0x90 [ 5900.211563] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5900.216213] generic_shutdown_super+0xb1/0x1b0 [ 5900.219249] kill_block_super+0x29/0x80 [ 5900.222002] deactivate_locked_super+0x52/0xd0 [ 5900.224148] deactivate_super+0x83/0x90 [ 5900.226047] cleanup_mnt+0x5f/0xe0 [ 5900.228221] __cleanup_mnt+0x16/0x20 [ 5900.230591] task_work_run+0xc6/0x110 [ 5900.232718] exit_to_usermode_loop+0x1dd/0x1f0 [ 5900.235122] do_syscall_64+0x3d6/0x440 [ 5900.237392] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5900.240530] RIP: 0033:0x7fbb019648fb [ 5900.241886] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5900.253186] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5900.257864] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5900.261999] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5900.265897] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5900.270075] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5900.274283] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5920.060958] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5920.070792] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5920.104471] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5920.110755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5920.117828] Workqueue: events free_work [libcfs] [ 5920.122430] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5920.125569] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5920.141060] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5920.149012] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5920.154712] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5920.158484] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5920.162943] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5920.167852] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5920.172284] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5920.181728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5920.185957] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5920.190877] Call Trace: [ 5920.192949] [ 5920.194051] ? show_regs.cold.9+0x22/0x2f [ 5920.199273] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5920.203891] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5920.210805] ? watchdog+0x40/0x40 [ 5920.212932] ? hrtimer_interrupt+0x128/0x2c0 [ 5920.215916] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5920.218063] ? apic_timer_interrupt+0xf/0x20 [ 5920.221882] [ 5920.222509] ? on_each_cpu+0x51/0x120 [ 5920.226158] ? smp_call_function_many_cond+0x3db/0x540 [ 5920.229299] ? smp_call_function_many_cond+0x3a8/0x540 [ 5920.235553] ? load_new_mm_cr3+0x120/0x120 [ 5920.241250] ? load_new_mm_cr3+0x120/0x120 [ 5920.242855] on_each_cpu+0x51/0x120 [ 5920.245592] flush_tlb_kernel_range+0x93/0xb0 [ 5920.248089] free_unmap_vmap_area+0x63/0x70 [ 5920.253175] remove_vm_area+0xe7/0x110 [ 5920.255742] __vunmap+0x9c/0x400 [ 5920.260757] vfree+0x4d/0x100 [ 5920.261974] free_work+0x34/0x50 [libcfs] [ 5920.263394] process_one_work+0x2c8/0x700 [ 5920.264654] worker_thread+0x296/0x6e0 [ 5920.265818] ? rescuer_thread+0x570/0x570 [ 5920.269025] kthread+0x1d1/0x200 [ 5920.270291] ? set_kthread_struct+0x70/0x70 [ 5920.273892] ret_from_fork+0x1f/0x30 [ 5924.055076] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:2:47596] [ 5924.060669] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5924.099755] CPU: 0 PID: 47596 Comm: kworker/0:2 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5924.106233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5924.113969] Workqueue: events free_work [libcfs] [ 5924.120182] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5924.124148] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5924.138501] RSP: 0018:ffffa1ca016afd68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5924.142773] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821b8920 [ 5924.146838] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400236c0 [ 5924.154061] RBP: ffff89da400234d0 R08: 0000000000000004 R09: 0000000000000000 [ 5924.160220] R10: ffff89da51ac3cf0 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5924.167159] R13: 0000000000000004 R14: ffff89da82034600 R15: 0000000000000003 [ 5924.171802] FS: 0000000000000000(0000) GS:ffff89da82000000(0000) knlGS:0000000000000000 [ 5924.175124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5924.178987] CR2: 000055aff5e13058 CR3: 000000005e816001 CR4: 0000000000170ef0 [ 5924.183599] Call Trace: [ 5924.184785] [ 5924.187823] ? show_regs.cold.9+0x22/0x2f [ 5924.190080] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5924.192712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5924.194952] ? watchdog+0x40/0x40 [ 5924.196709] ? hrtimer_interrupt+0x128/0x2c0 [ 5924.198750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5924.201077] ? apic_timer_interrupt+0xf/0x20 [ 5924.202712] [ 5924.203700] ? on_each_cpu+0x51/0x120 [ 5924.205358] ? smp_call_function_many_cond+0x3db/0x540 [ 5924.207942] ? smp_call_function_many_cond+0x3a8/0x540 [ 5924.210193] ? load_new_mm_cr3+0x120/0x120 [ 5924.212339] ? load_new_mm_cr3+0x120/0x120 [ 5924.214400] on_each_cpu+0x51/0x120 [ 5924.215838] flush_tlb_kernel_range+0x93/0xb0 [ 5924.217639] free_unmap_vmap_area+0x63/0x70 [ 5924.219027] remove_vm_area+0xe7/0x110 [ 5924.220600] __vunmap+0x9c/0x400 [ 5924.222011] vfree+0x4d/0x100 [ 5924.223244] free_work+0x34/0x50 [libcfs] [ 5924.225038] process_one_work+0x2c8/0x700 [ 5924.227099] worker_thread+0x296/0x6e0 [ 5924.228936] ? rescuer_thread+0x570/0x570 [ 5924.230852] kthread+0x1d1/0x200 [ 5924.231764] ? set_kthread_struct+0x70/0x70 [ 5924.233258] ret_from_fork+0x1f/0x30 [ 5926.880069] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3568s! [ 5926.884084] Showing busy workqueues and worker pools: [ 5926.886643] workqueue events: flags=0x0 [ 5926.888342] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5926.891111] in-flight: 56293:free_work [libcfs] [ 5926.893144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5926.895855] in-flight: 22277:free_work [libcfs] [ 5926.897891] pending: kfree_rcu_monitor, key_garbage_collector, osc_grant_work_handler [osc] [ 5926.901578] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5926.904474] pending: nfs4_renew_state [nfsv4] [ 5926.907995] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5926.911116] in-flight: 47596:free_work [libcfs] [ 5926.914041] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5926.922011] workqueue events_unbound: flags=0x2 [ 5926.926682] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 5926.930140] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5926.936683] workqueue events_power_efficient: flags=0x80 [ 5926.940016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5926.945889] pending: neigh_periodic_work, do_cache_clean [ 5926.949462] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5926.951566] pending: check_lifetime [ 5926.952482] workqueue mm_percpu_wq: flags=0x8 [ 5926.955137] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5926.959974] pending: vmstat_update [ 5926.962062] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5926.965423] pending: vmstat_update [ 5926.967466] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5926.971078] pending: vmstat_update [ 5926.973097] workqueue cgroup_pidlist_destroy: flags=0x0 [ 5926.977320] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5926.982579] pending: cgroup_pidlist_destroy_work_fn [ 5926.985564] workqueue writeback: flags=0x4a [ 5926.987266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5926.991064] pending: wb_update_bandwidth_workfn, wb_workfn, wb_workfn [ 5926.994758] workqueue xprtiod: flags=0xa [ 5926.997065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5927.002354] pending: xprt_autoclose [ 5927.003995] workqueue ipv6_addrconf: flags=0x40008 [ 5927.006685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5927.009866] pending: addrconf_verify_work [ 5927.015520] workqueue knbd0-recv: flags=0x1a [ 5927.017421] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5927.024025] in-flight: 655:recv_work [ 5927.026542] workqueue ptlrpc_pinger: flags=0x2 [ 5927.030955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5927.035959] pending: ptlrpc_pinger_main [ptlrpc] [ 5927.040967] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=3569s workers=5 idle: 53684 27131 54850 46584 [ 5927.049514] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=3569s workers=9 idle: 54853 49633 54852 54851 54849 37676 43546 54854 [ 5927.061207] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=3569s workers=6 idle: 53660 37674 46417 53664 46418 [ 5927.065983] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5916s workers=2 idle: 657 [ 5928.059360] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [umount:59839] [ 5928.062954] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5928.094050] CPU: 1 PID: 59839 Comm: umount Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5928.101946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5928.107172] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5928.109715] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5928.119512] RSP: 0018:ffffa1ca01d33da0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5928.123482] RAX: 0000000000000003 RBX: ffffffffb2337560 RCX: ffff89da821bae40 [ 5928.128185] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da400231f0 [ 5928.130897] RBP: ffff89da400231d8 R08: 0000000000000004 R09: 0000000000000001 [ 5928.134186] R10: ffff89d9446db4e8 R11: 0000000000000230 R12: ffffffffb202df26 [ 5928.137327] R13: 0000000000000004 R14: ffff89da820b4600 R15: 0000000000000003 [ 5928.140309] FS: 00007fbb02a0e080(0000) GS:ffff89da82080000(0000) knlGS:0000000000000000 [ 5928.144135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5928.146613] CR2: 00007f084bf8a2f8 CR3: 000000010674b006 CR4: 0000000000170ee0 [ 5928.149365] Call Trace: [ 5928.151439] [ 5928.152138] ? show_regs.cold.9+0x22/0x2f [ 5928.156723] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5928.161304] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5928.163120] ? watchdog+0x40/0x40 [ 5928.164629] ? hrtimer_interrupt+0x128/0x2c0 [ 5928.166486] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5928.168525] ? apic_timer_interrupt+0xf/0x20 [ 5928.170584] [ 5928.171511] ? on_each_cpu_cond_mask+0x46/0x150 [ 5928.173475] ? __x64_sys_fspick+0x210/0x210 [ 5928.174951] ? smp_call_function_many_cond+0x3db/0x540 [ 5928.176933] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5928.178697] ? __x64_sys_fspick+0x210/0x210 [ 5928.184462] ? buffer_exit_cpu_dead+0xb0/0xb0 [ 5928.187334] on_each_cpu_cond_mask+0x46/0x150 [ 5928.190131] on_each_cpu_cond+0x1c/0x30 [ 5928.195540] invalidate_bh_lrus+0x27/0x30 [ 5928.203795] invalidate_bdev+0x5a/0x90 [ 5928.206222] ldiskfs_put_super+0x36c/0x5c0 [ldiskfs] [ 5928.208908] generic_shutdown_super+0xb1/0x1b0 [ 5928.212592] kill_block_super+0x29/0x80 [ 5928.214164] deactivate_locked_super+0x52/0xd0 [ 5928.219790] deactivate_super+0x83/0x90 [ 5928.221397] cleanup_mnt+0x5f/0xe0 [ 5928.225522] __cleanup_mnt+0x16/0x20 [ 5928.231034] task_work_run+0xc6/0x110 [ 5928.234247] exit_to_usermode_loop+0x1dd/0x1f0 [ 5928.236963] do_syscall_64+0x3d6/0x440 [ 5928.238410] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5928.240998] RIP: 0033:0x7fbb019648fb [ 5928.245121] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 5928.259616] RSP: 002b:00007ffd55f21468 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 5928.265851] RAX: 0000000000000000 RBX: 000055aff5dffa40 RCX: 00007fbb019648fb [ 5928.268342] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055aff5e138b0 [ 5928.272568] RBP: 0000000000000000 R08: 000055aff5e05fa0 R09: 000055aff5dfe010 [ 5928.280546] R10: 0000000000000000 R11: 0000000000000246 R12: 000055aff5e138b0 [ 5928.283578] R13: 00007fbb027e8184 R14: 000055aff5dffc20 R15: 00000000ffffffff [ 5948.059378] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:6:22277] [ 5948.065748] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5948.104080] CPU: 2 PID: 22277 Comm: kworker/2:6 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5948.112873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 5948.120431] Workqueue: events free_work [libcfs] [ 5948.123091] RIP: 0010:smp_call_function_many_cond+0x3db/0x540 [ 5948.126065] Code: 00 00 00 48 63 d0 49 8b 0e 48 83 05 4e 97 ea 02 01 48 03 0c d5 80 58 88 b3 8b 11 83 e2 01 74 11 48 83 05 47 97 ea 02 01 f3 90 <8b> 11 83 e2 01 75 ef 48 83 05 2e 97 ea 02 01 48 83 05 36 97 ea 02 [ 5948.135495] RSP: 0018:ffffa1ca01c93d68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5948.139250] RAX: 0000000000000003 RBX: 0000000000000000 RCX: ffff89da821bae60 [ 5948.142973] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff89da40023310 [ 5948.148108] RBP: ffff89da40023190 R08: 0000000000000000 R09: 00000000ffffffff [ 5948.152114] R10: ffff89d944dc3558 R11: 0000000000000007 R12: ffffffffb202e1d1 [ 5948.156115] R13: 0000000000000004 R14: ffff89da82134600 R15: 0000000000000003 [ 5948.160178] FS: 0000000000000000(0000) GS:ffff89da82100000(0000) knlGS:0000000000000000 [ 5948.166562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5948.173483] CR2: 0000563447f1bab8 CR3: 000000005e816001 CR4: 0000000000170ee0 [ 5948.183209] Call Trace: [ 5948.185203] [ 5948.188642] ? show_regs.cold.9+0x22/0x2f [ 5948.191429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5948.194897] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5948.197164] ? watchdog+0x40/0x40 [ 5948.203876] ? hrtimer_interrupt+0x128/0x2c0 [ 5948.205828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5948.208407] ? apic_timer_interrupt+0xf/0x20 [ 5948.210795] [ 5948.211684] ? on_each_cpu+0x51/0x120 [ 5948.213932] ? smp_call_function_many_cond+0x3db/0x540 [ 5948.217896] ? smp_call_function_many_cond+0x3a8/0x540 [ 5948.222404] ? load_new_mm_cr3+0x120/0x120 [ 5948.226132] ? load_new_mm_cr3+0x120/0x120 [ 5948.228317] on_each_cpu+0x51/0x120 [ 5948.230111] flush_tlb_kernel_range+0x93/0xb0 [ 5948.232226] free_unmap_vmap_area+0x63/0x70 [ 5948.236442] remove_vm_area+0xe7/0x110 [ 5948.238529] __vunmap+0x9c/0x400 [ 5948.239892] vfree+0x4d/0x100 [ 5948.240970] free_work+0x34/0x50 [libcfs] [ 5948.243190] process_one_work+0x2c8/0x700 [ 5948.245608] worker_thread+0x296/0x6e0 [ 5948.248817] ? rescuer_thread+0x570/0x570 [ 5948.252106] kthread+0x1d1/0x200 [ 5948.254523] ? set_kthread_struct+0x70/0x70 [ 5948.256694] ret_from_fork+0x1f/0x30