************************ crashinfo ************************* /exports/testreports/45618/testresults/sanity2-zfs-centos7_x86_64-centos7_x86_64/oleg217-server-vmcore (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.5VNB5/vmlinux [TAINTED] DUMPFILE: /exports/testreports/45618/testresults/sanity2-zfs-centos7_x86_64-centos7_x86_64/oleg217-server-vmcore [PARTIAL DUMP] CPUS: 4 DATE: Thu Sep 12 01:31:11 EDT 2024 UPTIME: 00:52:42 LOAD AVERAGE: 5.24, 2.68, 1.29 TASKS: 445 NODENAME: oleg217-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "Kernel panic - not syncing: LBUG" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 poll_idle+0x92, 507 bytes of data #4 poll_idle+0x92 #5 cpuidle_enter_state+0x40 #6 cpuidle_idle_call+0xd8 #7 arch_cpu_idle+0xe #8 cpu_startup_entry+0x14a #9 rest_init+0x8e #10 start_kernel+0x456 #11 x86_64_start_reservations+0x2a #12 x86_64_start_kernel+0x152 #13 start_cpu+0x5 -- CPU#1 -- PID=3293 CPU=1 CMD=socknal_sd00_02 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e, 547 bytes of data #4 __getnstimeofday64+0x0 #5 getnstimeofday64+0xe #6 request_in_callback+0x2aa #7 ptlrpc_master_callback+0x38 #8 lnet_finalize+0x2e0 #9 ksocknal_process_receive+0x761 #10 ksocknal_scheduler+0x1ca #11 kthread+0xe4 #12 ret_from_fork_nospec_begin+0x7 -- CPU#2 -- PID=25515 CPU=2 CMD=mdt_out00_003 #0 machine_kexec+0x19e #1 __crash_kexec+0x72 #2 panic+0xf3 #3 lbug_with_loc+0xab #4 ptl_send_buf+0x5a7 #5 ptlrpc_send_reply+0x2f3 #6 target_send_reply_msg+0x63 #7 target_send_reply+0x31e #8 tgt_request_handle+0x36d #9 ptlrpc_server_handle_request+0x281 #10 ptlrpc_main+0xc7e #11 kthread+0xe4 #12 ret_from_fork_nospec_begin+0x7 -- CPU#3 -- PID=3291 CPU=3 CMD=socknal_sd00_00 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 memcpy+0x6, 507 bytes of data #4 memcpy+0x6 #5 vsnprintf+0x234 #6 libcfs_debug_msg+0xb13 #7 ksocknal_free_tx+0xc6 #8 ksocknal_tx_done+0x67 #9 ksocknal_scheduler+0x309 #10 kthread+0xe4 #11 ret_from_fork_nospec_begin+0x7 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ *** Panic *** BUG: 73 of 73 active objects replaced +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 139 last 5s 183 last 60s 216 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 437 TASK_RUNNING 5 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 3291 socknal_sd00_00 0 ms (no user stack) 3256 ll_ost00_036 0 ms (no user stack) 3305 ll_ost00_043 0 ms (no user stack) 4681 ll_ost00_088 0 ms (no user stack) 3838 ll_ost00_062 0 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 581267 2.2 GB 60% of TOTAL MEM USED 373800 1.4 GB 39% of TOTAL MEM SHARED 11612 45.4 MB 1% of TOTAL MEM BUFFERS 5123 20 MB 0% of TOTAL MEM CACHED 76112 297.3 MB 7% of TOTAL MEM SLAB 114518 447.3 MB 11% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 64490 251.9 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=socknal_sd00_02 ---+ CPU=2 ---- | CURRENT TASK , CMD=mdt_out00_003 ---+ CPU=3 ---- | CURRENT TASK , CMD=socknal_sd00_00 3256 ll_ost00_036 1.58286 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 9 LISTEN 3 NAGLE disabled (TCP_NODELAY): 7 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 3160.0 eth0 n/a 0.0 RSS_TOTAL=51716 pages, %mem= 0.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012a2f0000 ffff88012a318000 sysfs sysfs /sys ffff88012a2f01c0 ffff880139944000 proc proc /proc ffff88012a2f0380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012a2f0540 ffff8800b5203800 securityfs securityfs /sys/kernel/security ffff88012a2f0700 ffff88012a318800 tmpfs tmpfs /dev/shm ffff88012a2f08c0 ffff880137161800 devpts devpts /dev/pts ffff88012a2f0a80 ffff88012a319000 tmpfs tmpfs /run ffff88012a2f0c40 ffff88012a319800 tmpfs tmpfs /sys/fs/cgroup ffff88012a2f0e00 ffff88012a31a000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012a2f0fc0 ffff88012a31a800 pstore pstore /sys/fs/pstore ffff88012a2f1180 ffff88012a31c800 cgroup cgroup /sys/fs/cgroup/devices ffff88012a2f1340 ffff88012a31c000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a2f1500 ffff88012a31b800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a2f16c0 ffff88012a31b000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a2f1880 ffff88012a31d000 cgroup cgroup /sys/fs/cgroup/pids ffff88012a2f1a40 ffff88012a31d800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a2f1c00 ffff88012a31e000 cgroup cgroup /sys/fs/cgroup/memory ffff88012a2f1dc0 ffff88012a31e800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012abfa000 ffff88012a31f000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012abfa1c0 ffff88012a31f800 cgroup cgroup /sys/fs/cgroup/perf_event ffff880137668a80 ffff8800b5207800 configfs configfs /sys/kernel/config ffff88012abfa540 ffff8800b5250800 ext4 /dev/nbd0 / ffff88012abfa700 ffff8800b5254800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012abfa8c0 ffff88000006b000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880137669180 ffff8800b4df7000 hugetlbfs hugetlbfs /dev/hugepages ffff8800b4e26000 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b4e261c0 ffff880137679000 mqueue mqueue /dev/mqueue ffff88012abfaa80 ffff88000006c000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b4e26380 ffff8800b5276800 ramfs none /mnt ffff8800b4e26540 ffff8800b5251800 tmpfs none /var/lib/stateless/writable ffff880137669340 ffff8800b4df1000 squashfs /dev/vda /home/green/git/lustre-release ffff88012abfac40 ffff8800b5251800 tmpfs none /var/cache/man ffff8800b4e26700 ffff8800b5251800 tmpfs none /var/log ffff880137669500 ffff8800b5251800 tmpfs none /var/lib/dbus ffff88012abfae00 ffff8800b5251800 tmpfs none /tmp ffff8801376696c0 ffff8800b5251800 tmpfs none /var/lib/dhclient ffff880137669880 ffff8800b5251800 tmpfs none /var/tmp ffff880137669a40 ffff8800b5251800 tmpfs none /var/lib/NetworkManager ffff88012abfafc0 ffff8800b5251800 tmpfs none /var/lib/systemd/random-seed ffff88012abfb180 ffff8800b5251800 tmpfs none /var/spool ffff88012abfb340 ffff8800b5251800 tmpfs none /var/lib/nfs ffff88012abfb500 ffff8800b5251800 tmpfs none /var/lib/gssproxy ffff880137669c00 ffff8800b5251800 tmpfs none /var/lib/logrotate ffff880137669dc0 ffff8800b5251800 tmpfs none /etc ffff880137668fc0 ffff8800b5251800 tmpfs none /var/lib/rsyslog ffff88012abfb6c0 ffff8800b5251800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012abfb880 ffff8800b51ca800 nfs4 192.168.200.253:/exports/state/oleg217-server.virtnet /var/lib/stateless/state ffff88012abfbc00 ffff8800b51ca800 nfs4 192.168.200.253:/exports/state/oleg217-server.virtnet /boot ffff88012abfbdc0 ffff8800b51ca800 nfs4 192.168.200.253:/exports/state/oleg217-server.virtnet /etc/etc/kdump.conf ffff88012abfba40 ffff8800b5254800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b25f5880 ffff8800b4df1000 squashfs /dev/vda /usr/sbin/mount.lustre ffff880129e63a40 ffff8800b199a000 lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 ffff8800b25f4700 ffff880094471000 lustre lustre-ost2/ost2 /mnt/lustre-ost2 ffff8800b24f8700 ffff88012f458000 lustre lustre-ost1/ost1 /mnt/lustre-ost1 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 3156.492107] CPU: 0 PID: 25515 Comm: mdt_out00_003 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1 [ 3156.493993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 3156.495749] Call Trace: [ 3156.496305] [] dump_stack+0x19/0x1b [ 3156.497331] [] lbug_with_loc+0x4d/0xb0 [libcfs] [ 3156.498434] [] ptl_send_buf+0x5a7/0x5b0 [ptlrpc] [ 3156.499831] [] ? ptlrpc_ni_fini+0x60/0x60 [ptlrpc] [ 3156.501046] [] ptlrpc_send_reply+0x2f3/0x9d0 [ptlrpc] [ 3156.502331] [] target_send_reply_msg+0x63/0x1e0 [ptlrpc] [ 3156.503640] [] target_send_reply+0x31e/0x780 [ptlrpc] [ 3156.505078] [] tgt_request_handle+0x36d/0x1a60 [ptlrpc] [ 3156.506327] [] ptlrpc_server_handle_request+0x281/0xce0 [ptlrpc] [ 3156.507869] [] ptlrpc_main+0xc7e/0x1690 [ptlrpc] [ 3156.508960] [] ? do_raw_spin_unlock+0x49/0x90 [ 3156.510267] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 3156.511463] [] kthread+0xe4/0xf0 [ 3156.512419] [] ? kthread_create_on_node+0x140/0x140 [ 3156.513601] [] ret_from_fork_nospec_begin+0x7/0x21 [ 3156.514859] [] ? kthread_create_on_node+0x140/0x140 [ 3162.517861] Kernel panic - not syncing: LBUG [ 3162.518776] CPU: 2 PID: 25515 Comm: mdt_out00_003 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1 [ 3162.520662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 3162.522599] Call Trace: [ 3162.523113] [] dump_stack+0x19/0x1b [ 3162.524160] [] panic+0xe8/0x20d [ 3162.525303] [] lbug_with_loc+0xab/0xb0 [libcfs] [ 3162.526918] [] ptl_send_buf+0x5a7/0x5b0 [ptlrpc] [ 3162.527948] [] ? ptlrpc_ni_fini+0x60/0x60 [ptlrpc] [ 3162.529013] [] ptlrpc_send_reply+0x2f3/0x9d0 [ptlrpc] [ 3162.530149] [] target_send_reply_msg+0x63/0x1e0 [ptlrpc] [ 3162.531786] [] target_send_reply+0x31e/0x780 [ptlrpc] [ 3162.533404] [] tgt_request_handle+0x36d/0x1a60 [ptlrpc] [ 3162.534520] [] ptlrpc_server_handle_request+0x281/0xce0 [ptlrpc] [ 3162.535839] [] ptlrpc_main+0xc7e/0x1690 [ptlrpc] [ 3162.536985] [] ? do_raw_spin_unlock+0x49/0x90 [ 3162.537957] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 3162.539114] [] kthread+0xe4/0xf0 [ 3162.539984] [] ? kthread_create_on_node+0x140/0x140 [ 3162.541057] [] ret_from_fork_nospec_begin+0x7/0x21 [ 3162.542097] [] ? kthread_create_on_node+0x140/0x140 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.95s (real) 6.42s (CPU), Child processes: 4.52s