************************ crashinfo ************************* /exports/testreports/52899/testresults/sanity3-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg439-client-vmcore (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.EKCXL/vmlinux [TAINTED] DUMPFILE: /exports/testreports/52899/testresults/sanity3-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg439-client-vmcore [PARTIAL DUMP] CPUS: 4 DATE: Sun Jun 22 07:43:02 EDT 2025 UPTIME: 00:38:28 LOAD AVERAGE: 0.85, 0.89, 0.74 TASKS: 154 NODENAME: oleg439-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "Kernel panic - not syncing: LBUG" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=19076 CPU=0 CMD=sendfile_groupl #0 machine_kexec+0x19e #1 __crash_kexec+0x72 #2 panic+0xf3 #3 lbug_with_loc+0x95 #4 cl_env_percpu_put+0x7e #5 do_read_folio+0x6fc #6 ll_readpage+0xe #7 do_generic_file_read.constprop.52+0x225 #8 generic_file_aio_read+0x1c9 #9 do_file_read_iter+0x96a #10 do_loop_readv_writev+0x1fe32864 #11 ll_file_read_iter+0x22 #12 ll_file_aio_read+0x1d6 #13 do_sync_readv_writev+0x79 #14 do_readv_writev+0xce #15 vfs_readv+0x32 #16 default_file_splice_read+0x2a5 #17 pcc_file_splice_read+0x195 #18 do_splice_to+0x72 #19 splice_direct_to_actor+0xb7 #20 do_splice_direct+0x62 #21 do_sendfile+0x1d8 #22 sys_sendfile64+0xda #23 system_call_fastpath+0x1f, 477 bytes of data -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ *** Panic *** BUG: 73 of 73 active objects replaced +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 38 last 5s 42 last 60s 67 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 147 TASK_RUNNING 4 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 9 rcu_sched 0 ms (no user stack) 907 rs:main Q:Reg 1 ms (no user stack) 19076 sendfile_groupl 7 ms (no user stack) 905 in:imjournal 84 ms (no user stack) 566 systemd-journal 84 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 862825 3.3 GB 90% of TOTAL MEM USED 92254 360.4 MB 9% of TOTAL MEM SHARED 24799 96.9 MB 2% of TOTAL MEM BUFFERS 476 1.9 MB 0% of TOTAL MEM CACHED 40420 157.9 MB 4% of TOTAL MEM SLAB 12484 48.8 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 69387 271 MB 9% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=sendfile_groupl 6 ksoftirqd/0 3.70279 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 10 TIME_WAIT 8 LISTEN 3 NAGLE disabled (TCP_NODELAY): 8 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 2306.6 eth0 n/a 0.1 RSS_TOTAL=96432 pages, %mem= 1.6 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aaea000 ffff88012aaf0000 sysfs sysfs /sys ffff88012aaea1c0 ffff880139944000 proc proc /proc ffff88012aaea380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aaea540 ffff8800b6cc2800 securityfs securityfs /sys/kernel/security ffff88012aaea700 ffff88012aaf0800 tmpfs tmpfs /dev/shm ffff88012aaea8c0 ffff880136bc9000 devpts devpts /dev/pts ffff88012aaeaa80 ffff88012aaf1000 tmpfs tmpfs /run ffff88012aaeac40 ffff88012aaf1800 tmpfs tmpfs /sys/fs/cgroup ffff88012aaeae00 ffff88012aaf2000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aaeafc0 ffff88012aaf2800 pstore pstore /sys/fs/pstore ffff88012aaeb180 ffff88012aaf4800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aaeb340 ffff88012aaf4000 cgroup cgroup /sys/fs/cgroup/memory ffff88012aaeb500 ffff88012aaf3800 cgroup cgroup /sys/fs/cgroup/pids ffff88012aaeb6c0 ffff88012aaf3000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aaeb880 ffff88012aaf5000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aaeba40 ffff88012aaf5800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aaebc00 ffff88012aaf6000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aaebdc0 ffff88012aaf6800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aafe000 ffff88012aaf7000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aafe1c0 ffff88012aaf7800 cgroup cgroup /sys/fs/cgroup/devices ffff880137668540 ffff8800b6cc6800 configfs configfs /sys/kernel/config ffff88012aafe540 ffff88012ab88800 ext4 /dev/nbd0 / ffff88012aafe700 ffff88012ab8f000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012a51c1c0 ffff8800b5c8c800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8801376688c0 ffff8800b64ab800 hugetlbfs hugetlbfs /dev/hugepages ffff88012a51c540 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff88012a51c380 ffff880136bca800 mqueue mqueue /dev/mqueue ffff88012aafee00 ffff8800b6ee3800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b67a0c40 ffff8800b646e800 ramfs none /mnt ffff88012aafefc0 ffff8800b6ee5000 tmpfs none /var/lib/stateless/writable ffff88012a51c8c0 ffff8800b5e6f000 squashfs /dev/vda /home/green/git/lustre-release ffff880137668a80 ffff8800b6ee5000 tmpfs none /var/cache/man ffff880137668c40 ffff8800b6ee5000 tmpfs none /var/log ffff880137668e00 ffff8800b6ee5000 tmpfs none /var/lib/dbus ffff8800b67a0fc0 ffff8800b6ee5000 tmpfs none /tmp ffff880137668fc0 ffff8800b6ee5000 tmpfs none /var/lib/dhclient ffff880137669180 ffff8800b6ee5000 tmpfs none /var/tmp ffff880137669340 ffff8800b6ee5000 tmpfs none /var/lib/NetworkManager ffff88012aaff180 ffff8800b6ee5000 tmpfs none /var/lib/systemd/random-seed ffff8800b67a1180 ffff8800b6ee5000 tmpfs none /var/spool ffff880137669500 ffff8800b6ee5000 tmpfs none /var/lib/nfs ffff8801376696c0 ffff8800b6ee5000 tmpfs none /var/lib/gssproxy ffff880137669880 ffff8800b6ee5000 tmpfs none /var/lib/logrotate ffff880137669a40 ffff8800b6ee5000 tmpfs none /etc ffff8800b67a1340 ffff8800b6ee5000 tmpfs none /var/lib/rsyslog ffff880137669c00 ffff8800b6ee5000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012aaff340 ffff8800b5c88000 nfs4 192.168.200.253:/exports/state/oleg439-client.virtnet /var/lib/stateless/state ffff880137669dc0 ffff8800b5c88000 nfs4 192.168.200.253:/exports/state/oleg439-client.virtnet /boot ffff8800b67a1500 ffff8800b5c88000 nfs4 192.168.200.253:/exports/state/oleg439-client.virtnet /etc/etc/kdump.conf ffff880138ccb880 ffff88012ab8f000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b041d880 ffff88012ab8a800 nfs4 192.168.200.253://exports/testreports/52899/testresults/sanity3-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800b041da40 ffff88012ab8e000 tmpfs tmpfs /run/user/0 ffff8800b04ee1c0 ffff8800b5e6f000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b04eefc0 ffff8800b060d800 lustre 192.168.204.139@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 2309.435337] [] do_sendfile+0x1d8/0x3c0 [ 2309.438649] [] SyS_sendfile64+0xda/0xf0 [ 2309.443330] [] system_call_fastpath+0x1f/0x24 [ 2309.445599] Kernel panic - not syncing: LBUG [ 2309.446949] CPU: 0 PID: 19076 Comm: sendfile_groupl Kdump: loaded Tainted: G W OE ------------ 3.10.0-7.9-debug #1 [ 2309.450867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 2309.453379] Call Trace: [ 2309.453993] [] dump_stack+0x19/0x1b [ 2309.455514] [] panic+0xe8/0x20d [ 2309.457267] [] lbug_with_loc+0x95/0xa0 [libcfs] [ 2309.459661] [] cl_env_percpu_put+0x7e/0xb0 [obdclass] [ 2309.461803] [] do_read_folio+0x6fc/0x1290 [lustre] [ 2309.463690] [] ll_readpage+0xe/0x10 [lustre] [ 2309.465700] [] do_generic_file_read.constprop.52+0x225/0x540 [ 2309.469013] [] generic_file_aio_read+0x1c9/0x290 [ 2309.471383] [] do_file_read_iter+0x96a/0x1370 [lustre] [ 2309.473917] [] ? libcfs_debug_msg+0x4c9/0xc60 [libcfs] [ 2309.476066] [] ? ll_file_write+0x1e0/0x1e0 [lustre] [ 2309.477894] [] do_loop_readv_writev.isra.65+0x84/0xc0 [lustre] [ 2309.480502] [] ll_file_read_iter+0x22/0x40 [lustre] [ 2309.482996] [] ll_file_aio_read+0x1d6/0x210 [lustre] [ 2309.485643] [] do_sync_readv_writev+0x79/0xd0 [ 2309.487234] [] do_readv_writev+0xce/0x260 [ 2309.488935] [] ? alloc_pages_current+0x98/0x110 [ 2309.491109] [] vfs_readv+0x32/0x60 [ 2309.492550] [] default_file_splice_read+0x2a5/0x3c0 [ 2309.494955] [] ? page_cache_pipe_buf_release+0x20/0x20 [ 2309.497368] [] ? string.isra.7+0x3b/0xf0 [ 2309.499700] [] ? vsnprintf+0x201/0x6a0 [ 2309.501805] [] ? snprintf+0x49/0x70 [ 2309.503330] [] ? do_raw_spin_unlock+0x49/0x90 [ 2309.507241] [] ? _raw_spin_unlock+0xe/0x20 [ 2309.509689] [] pcc_file_splice_read+0x195/0x270 [lustre] [ 2309.511672] [] do_splice_to+0x72/0x90 [ 2309.515454] [] splice_direct_to_actor+0xb7/0x200 [ 2309.520324] [] ? do_splice_from+0xf0/0xf0 [ 2309.522138] [] do_splice_direct+0x62/0x90 [ 2309.524649] [] do_sendfile+0x1d8/0x3c0 [ 2309.526802] [] SyS_sendfile64+0xda/0xf0 [ 2309.528366] [] system_call_fastpath+0x1f/0x24 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 12.31s (real) 6.22s (CPU), Child processes: 5.74s