************************ crashinfo ************************* /exports/testreports/42551/testresults/sanity-benchmark-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg440-client-vmcore (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.0wU2E/vmlinux [TAINTED] DUMPFILE: /exports/testreports/42551/testresults/sanity-benchmark-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg440-client-vmcore [PARTIAL DUMP] CPUS: 4 DATE: Wed May 8 18:08:14 EDT 2024 UPTIME: 00:07:23 LOAD AVERAGE: 0.77, 1.34, 0.69 TASKS: 159 NODENAME: oleg440-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "BUG: unable to handle kernel NULL pointer dereference at (null)" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=12004 CPU=0 CMD=fsx #0 machine_kexec+0x19e #1 __crash_kexec+0x72 #2 crash_kexec+0x30 #3 oops_end+0xa8 #4 no_context+0x144 #5 __bad_area_nosemaphore+0x112 #6 bad_area+0x49 #7 __do_page_fault+0x494 #8 trace_do_page_fault+0x56 #9 do_async_page_fault+0x22 #10 async_page_fault+0x28 #-1 osc_discard_cb+0x1e3, 477 bytes of data #11 radix_tree_gang_lookup+0xcd #12 osc_page_gang_lookup+0x3bf #13 osc_punch_start+0x8e #14 osc_io_setattr_start+0x57d #15 cl_io_start+0x6d #16 lov_io_call+0x5a #17 lov_io_start+0x56 #18 lov_io_setattr_start+0x8b #19 cl_io_start+0x6d #20 cl_io_loop+0x9f #21 ll_fallocate+0x575 #22 vfs_fallocate+0x13c #23 sys_fallocate+0x5b #24 system_call_fastpath+0x1f, 477 bytes of data -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ *** Dump was triggered by kernel *** BUG: 73 of 73 active objects replaced +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 55 last 5s 69 last 60s 73 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 154 TASK_RUNNING 2 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 12004 fsx 0 ms (no user stack) 2008 ptlrpcd_01_00 0 ms (no user stack) 1999 socknal_sd00_01 0 ms (no user stack) 1998 socknal_sd00_00 0 ms (no user stack) 2009 ptlrpcd_01_01 0 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 845151 3.2 GB 88% of TOTAL MEM USED 109928 429.4 MB 11% of TOTAL MEM SHARED 9851 38.5 MB 1% of TOTAL MEM BUFFERS 5155 20.1 MB 0% of TOTAL MEM CACHED 52550 205.3 MB 5% of TOTAL MEM SLAB 14448 56.4 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 65189 254.6 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=fsx ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 TIME_WAIT 17 LISTEN 3 NAGLE disabled (TCP_NODELAY): 5 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 441.5 eth0 n/a 0.0 RSS_TOTAL=86756 pages, %mem= 1.5 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff880137668700 ffff88013767c000 sysfs sysfs /sys ffff8801376688c0 ffff880139944000 proc proc /proc ffff880137668a80 ffff880137678000 devtmpfs devtmpfs /dev ffff880137668c40 ffff8800b6cb5000 securityfs securityfs /sys/kernel/security ffff880137668e00 ffff88013767c800 tmpfs tmpfs /dev/shm ffff880137668fc0 ffff880137314000 devpts devpts /dev/pts ffff880137669180 ffff88013767d000 tmpfs tmpfs /run ffff880137669340 ffff88013767d800 tmpfs tmpfs /sys/fs/cgroup ffff880137669500 ffff88013767e000 cgroup cgroup /sys/fs/cgroup/systemd ffff8801376696c0 ffff88013767e800 pstore pstore /sys/fs/pstore ffff880137669880 ffff88012aaf0800 cgroup cgroup /sys/fs/cgroup/pids ffff880137669a40 ffff88012aaf0000 cgroup cgroup /sys/fs/cgroup/perf_event ffff880137669c00 ffff88012aaf1000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff880137669dc0 ffff88012aaf1800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012ab20000 ffff88012aaf2000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012ab201c0 ffff88012aaf2800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012ab20380 ffff88012aaf3000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012ab20540 ffff88012aaf3800 cgroup cgroup /sys/fs/cgroup/devices ffff88012ab20700 ffff88012aaf4000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012ab208c0 ffff88012aaf4800 cgroup cgroup /sys/fs/cgroup/memory ffff880137018380 ffff8800b6d95000 configfs configfs /sys/kernel/config ffff88012ab21c00 ffff8800b60c4000 ext4 /dev/nbd0 / ffff88012ab21dc0 ffff8800b6d93000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff880137018a80 ffff8800b6089800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8800b6e42000 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b6e421c0 ffff8800b6057000 hugetlbfs hugetlbfs /dev/hugepages ffff8800b6e42380 ffff880137679000 mqueue mqueue /dev/mqueue ffff88012ab1a380 ffff8800b6d96000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff880137018e00 ffff88012a49a000 ramfs none /mnt ffff8800b6e42540 ffff88012abd0800 tmpfs none /var/lib/stateless/writable ffff880137018fc0 ffff88012e1f1000 squashfs /dev/vda /home/green/git/lustre-release ffff8800b6e428c0 ffff88012abd0800 tmpfs none /var/cache/man ffff880138ccb500 ffff88012abd0800 tmpfs none /var/log ffff8800b6e42a80 ffff88012abd0800 tmpfs none /var/lib/dbus ffff8800b6e42c40 ffff88012abd0800 tmpfs none /tmp ffff880138ccb6c0 ffff88012abd0800 tmpfs none /var/lib/dhclient ffff880137019180 ffff88012abd0800 tmpfs none /var/tmp ffff880138ccb880 ffff88012abd0800 tmpfs none /var/lib/NetworkManager ffff88012ab1a540 ffff88012abd0800 tmpfs none /var/lib/systemd/random-seed ffff880137019340 ffff88012abd0800 tmpfs none /var/spool ffff8800b6e42e00 ffff88012abd0800 tmpfs none /var/lib/nfs ffff880137019500 ffff88012abd0800 tmpfs none /var/lib/gssproxy ffff88012ab1a700 ffff88012abd0800 tmpfs none /var/lib/logrotate ffff88012ab1a8c0 ffff88012abd0800 tmpfs none /etc ffff8801370196c0 ffff88012abd0800 tmpfs none /var/lib/rsyslog ffff880137019880 ffff88012abd0800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b6e42fc0 ffff8800b5878000 nfs4 192.168.200.253:/exports/state/oleg440-client.virtnet /var/lib/stateless/state ffff880138ccba40 ffff8800b5878000 nfs4 192.168.200.253:/exports/state/oleg440-client.virtnet /boot ffff8800b6e43180 ffff8800b5878000 nfs4 192.168.200.253:/exports/state/oleg440-client.virtnet /etc/etc/kdump.conf ffff88012ab1ae00 ffff8800b6d93000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b3f44540 ffff8800b6088800 nfs4 192.168.200.253://exports/testreports/42551/testresults/sanity-benchmark-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800b3f87180 ffff8800b587f800 tmpfs tmpfs /run/user/0 ffff880137010700 ffff88012e1f1000 squashfs /dev/vda /usr/sbin/mount.lustre ffff880137668540 ffff88012a49c800 lustre 192.168.204.140@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 443.908728] Lustre: DEBUG MARKER: == sanity-benchmark test fsx: fsx ======================== 18:08:13 (1715206093) [ 444.524762] BUG: unable to handle kernel NULL pointer dereference at (null) [ 444.528106] IP: [] osc_discard_cb+0x1e3/0x2a0 [osc] [ 444.531135] PGD 770a5067 PUD a3038067 PMD 0 [ 444.533786] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [ 444.536365] Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic rpcsec_gss_krb5 sb_edac edac_core iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel squashfs aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 i2c_core pcspkr binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata serio_raw [ 444.558303] CPU: 0 PID: 12004 Comm: fsx Kdump: loaded Tainted: G OE ------------ 3.10.0-7.9-debug #1 [ 444.561073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 444.563398] task: ffff8800b639c440 ti: ffff88009ba14000 task.ti: ffff88009ba14000 [ 444.565288] RIP: 0010:[] [] osc_discard_cb+0x1e3/0x2a0 [osc] [ 444.567354] RSP: 0018:ffff88009ba17b70 EFLAGS: 00010246 [ 444.568689] RAX: 0000000000000000 RBX: ffff880096db1b48 RCX: 0000000000000000 [ 444.570348] RDX: 0000000000000000 RSI: ffffffffa0a86e80 RDI: ffff880096db1b48 [ 444.571957] RBP: ffff88009ba17bc0 R08: ffff8800a8e4caf0 R09: 0000000000000001 [ 444.573441] R10: 00000000663bf7ce R11: 0000000000000000 R12: ffff8800b031f690 [ 444.574906] R13: ffff880096fc4e18 R14: 0000000000000000 R15: ffff880096fc5088 [ 444.576510] FS: 00007fb34e21d740(0000) GS:ffff88013e200000(0000) knlGS:0000000000000000 [ 444.578127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 444.579209] CR2: 0000000000000000 CR3: 0000000098f4e000 CR4: 0000000000160ff0 [ 444.580550] Call Trace: [ 444.581083] [] ? radix_tree_gang_lookup+0xcd/0x150 [ 444.582356] [] osc_page_gang_lookup+0x3bf/0x4e0 [osc] [ 444.583655] [] ? osc_consume_write_grant+0xe0/0xe0 [osc] [ 444.584897] [] osc_punch_start+0x8e/0x190 [osc] [ 444.586058] [] osc_io_setattr_start+0x57d/0x940 [osc] [ 444.587342] [] ? cl_io_iter_init+0x140/0x140 [obdclass] [ 444.588580] [] cl_io_start+0x6d/0x150 [obdclass] [ 444.589703] [] lov_io_call.isra.8+0x5a/0x170 [lov] [ 444.590875] [] lov_io_start+0x56/0x160 [lov] [ 444.591993] [] lov_io_setattr_start+0x8b/0x1e0 [lov] [ 444.593152] [] cl_io_start+0x6d/0x150 [obdclass] [ 444.594318] [] cl_io_loop+0x9f/0x230 [obdclass] [ 444.595518] [] ll_fallocate+0x575/0x620 [lustre] [ 444.596609] [] vfs_fallocate+0x13c/0x1e0 [ 444.597654] [] SyS_fallocate+0x5b/0xa0 [ 444.598754] [] system_call_fastpath+0x1f/0x24 [ 444.599933] Code: c7 05 3e ff 02 00 00 00 00 00 e8 49 b0 7b ff e9 af fe ff ff 0f 1f 40 00 44 8b 75 c8 4c 8b 7d c0 49 63 d6 49 8b 07 49 8b 54 d7 f8 <48> 8b 00 48 8b 12 48 8b 40 18 48 8b 52 18 48 8b 70 10 48 8b 78 [ 444.604813] RIP [] osc_discard_cb+0x1e3/0x2a0 [osc] [ 444.606135] RSP [ 444.606771] CR2: 0000000000000000 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.76s (real) 5.58s (CPU), Child processes: 5.06s