************************ crashinfo ************************* /exports/testreports/54378/testresults/sanity3-zfs-rocky8.10_x86_64-rocky8.10_x86_64/oleg128-client-vmcore (4.18.0rh8.10-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.KMUC8/vmlinux [TAINTED] DUMPFILE: /exports/testreports/54378/testresults/sanity3-zfs-rocky8.10_x86_64-rocky8.10_x86_64/oleg128-client-vmcore [PARTIAL DUMP] CPUS: 4 DATE: Tue Aug 19 17:22:38 EDT 2025 UPTIME: 00:28:11 LOAD AVERAGE: 1.05, 0.88, 0.79 TASKS: 196 NODENAME: oleg128-client.virtnet RELEASE: 4.18.0rh8.10-debug VERSION: #2 SMP Mon Jul 14 01:24:22 EDT 2025 MACHINE: x86_64 (2399 Mhz) MEMORY: 4.1 GB PANIC: "Kernel panic - not syncing: LBUG" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=138688 CPU=0 CMD=ldlm_bl_02 #0 crash_nmi_callback+0x6d #1 nmi_handle+0xbd #2 default_do_nmi+0x4b #3 do_nmi+0x280 #4 end_repeat_nmi+0x16 #-1 __raw_callee_save___pv_queued_spin_unlock+0xc, 507 bytes of data #5 __raw_callee_save___pv_queued_spin_unlock+0xc #6 do_raw_spin_unlock+0x75 #7 _raw_spin_unlock+0x12 #8 cfs_trace_unlock_tcd+0x28 #9 libcfs_debug_msg+0xcf4 #10 osc_page_gang_lookup+0x261 #11 osc_lock_discard_pages+0x20a #12 osc_dlm_blocking_ast0+0x541 #13 osc_ldlm_blocking_ast+0x1ba #14 ldlm_cancel_callback+0x9b #15 ldlm_cli_cancel_local+0xd2 #16 ldlm_cli_cancel+0xd9 #17 osc_ldlm_blocking_ast+0x186 #18 ldlm_handle_bl_callback+0xfa #19 ldlm_bl_thread_main+0x916 #20 kthread+0x1d1 #21 ret_from_fork+0x1f -- CPU#1 -- PID=151703 CPU=1 CMD=ldlm_bl_03 #0 crash_nmi_callback+0x6d #1 nmi_handle+0xbd #2 default_do_nmi+0x4b #3 do_nmi+0x280 #4 end_repeat_nmi+0x16 #-1 do_raw_spin_lock+0x66, 507 bytes of data #5 do_raw_spin_lock+0x66 #6 _raw_spin_lock+0x21 #7 cfs_trace_lock_tcd+0x2c #8 libcfs_debug_msg+0x14c #9 osc_page_gang_lookup+0x261 #10 osc_lock_discard_pages+0x20a #11 osc_dlm_blocking_ast0+0x541 #12 osc_ldlm_blocking_ast+0x1ba #13 ldlm_cancel_callback+0x9b #14 ldlm_cli_cancel_local+0xd2 #15 ldlm_cli_cancel+0xd9 #16 osc_ldlm_blocking_ast+0x186 #17 ldlm_handle_bl_callback+0xfa #18 ldlm_bl_thread_main+0x916 #19 kthread+0x1d1 #20 ret_from_fork+0x1f -- CPU#2 -- PID=138687 CPU=2 CMD=ldlm_bl_01 #0 machine_kexec+0x243 #1 __crash_kexec+0xaa #2 panic+0x160 #3 lbug_with_loc.cold.4+0x32 #4 osc_page_delete+0x788 #5 __cl_page_delete+0xa8 #6 cl_page_delete+0x3e #7 ll_invalidatepage+0xca #8 truncate_cleanup_page+0xe6 #9 truncate_inode_pages_range+0x57d #10 vvp_inode_ops+0x18a #11 cl_object_inode_ops+0x167 #12 osc_page_gang_lookup+0x306 #13 osc_lock_discard_pages+0x20a #14 osc_dlm_blocking_ast0+0x541 #15 osc_ldlm_blocking_ast+0x1ba #16 ldlm_cancel_callback+0x9b #17 ldlm_cli_cancel_local+0xd2 #18 ldlm_cli_cancel+0xd9 #19 osc_ldlm_blocking_ast+0x186 #20 ldlm_handle_bl_callback+0xfa #21 ldlm_bl_thread_main+0x916 #22 kthread+0x1d1 #23 ret_from_fork+0x1f -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #0 crash_nmi_callback+0x6d #1 nmi_handle+0xbd #2 default_do_nmi+0x4b #3 do_nmi+0x280 #4 end_repeat_nmi+0x16 #-1 native_safe_halt+0x2e, 507 bytes of data #5 native_safe_halt+0x2e #6 default_idle+0x1a #7 arch_cpu_idle+0x13 #8 default_idle_call+0xb8 #9 do_idle+0x3c9 #10 cpu_startup_entry+0x9f #11 start_secondary+0x2b8 #12 secondary_startup_64_no_verify+0xd1 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ *** Panic *** Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task +++ERROR+++ corrupted task Number of Threads That Ran Recently ----------------------------------- last second 46 last 5s 56 last 60s 101 ----- Total Numbers of Threads per State ------ ?? 193 +++WARNING+++ There are 6 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 138688 ldlm_bl_02 1 ms (no user stack) 157578 fio 1 ms (no user stack) 13 rcu_sched 2 ms (no user stack) 157579 fio 5 ms (no user stack) 793 systemd-journal 42 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 945455 3.6 GB ---- FREE 265229 1 GB 28% of TOTAL MEM USED 680226 2.6 GB 71% of TOTAL MEM SHARED 588353 2.2 GB 62% of TOTAL MEM BUFFERS 2651 10.4 MB 0% of TOTAL MEM CACHED 602035 2.3 GB 63% of TOTAL MEM SLAB 9176 35.8 MB 0% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 734870 2.8 GB ---- COMMITTED 1100646 4.2 GB 149% of TOTAL LIMIT print_blkreq is not supported on this kernel Traceback (most recent call last): File "/usr/local/crash/mpykdump.so/progs/crashinfo.py", line 1673, in File "/home/green/git/pykdump/LinuxDump/Dev.py", line 500, in print_gendisk # Get the list of (dev, gendisk) sorted by dev File "/home/green/git/pykdump/LinuxDump/Dev.py", line 473, in get_gendisks ttype = target.type File "/home/green/git/pykdump/pykdump/lowlevel.py", line 391, in __getattr__ raise KeyError(msg) KeyError: ' does not have a field ' ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 6 threads running in their own namespaces Use 'taskinfo --ns' to get more details --------------------- A list of all +++ERROR+++ messages --------------------- corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task corrupted task ------------------------------------------------------------------------------ ** Execution took 20.16s (real) 7.66s (CPU), Child processes: 12.43s