************************ crashinfo ************************* /exports/testreports/41705/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg330-client-vmcore (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.j70of/vmlinux [TAINTED] DUMPFILE: /exports/testreports/41705/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg330-client-vmcore [PARTIAL DUMP] CPUS: 4 DATE: Sun Apr 7 05:26:02 EDT 2024 UPTIME: 01:33:07 LOAD AVERAGE: 0.70, 0.48, 0.69 TASKS: 158 NODENAME: oleg330-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "BUG: unable to handle kernel paging request at ffff8800a4aebed8" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 rest_init+0x8e #12 start_kernel+0x456 #13 x86_64_start_reservations+0x2a #14 x86_64_start_kernel+0x152 #15 start_cpu+0x5 -- CPU#1 -- PID=1044 CPU=1 CMD=bash #0 machine_kexec+0x19e #1 __crash_kexec+0x72 #2 crash_kexec+0x30 #3 oops_end+0xa8 #4 no_context+0x144 #5 __bad_area_nosemaphore+0x112 #6 bad_area_nosemaphore+0x14 #7 __do_page_fault+0x348 #8 trace_do_page_fault+0x56 #9 do_async_page_fault+0x22 #10 async_page_fault+0x28 #-1 ll_dir_read+0x131, 477 bytes of data #11 ll_readdir+0x171 #12 iterate_dir+0x91 #13 sys_getdents+0xc4 #14 system_call_fastpath+0x1f, 477 bytes of data -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ *** Dump was triggered by kernel *** BUG: 73 of 73 active objects replaced +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 44 last 5s 60 last 60s 69 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 153 TASK_RUNNING 2 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 27 rcuos/2 0 ms (no user stack) 1044 bash 0 ms (no user stack) 29854 socknal_sd01_00 0 ms (no user stack) 29855 socknal_sd01_01 0 ms (no user stack) 34 rcuos/3 3 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 842826 3.2 GB 88% of TOTAL MEM USED 112253 438.5 MB 11% of TOTAL MEM SHARED 11996 46.9 MB 1% of TOTAL MEM BUFFERS 5615 21.9 MB 0% of TOTAL MEM CACHED 55953 218.6 MB 5% of TOTAL MEM SLAB 13086 51.1 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 76611 299.3 MB 10% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=bash ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 TIME_WAIT 177 LISTEN 3 NAGLE disabled (TCP_NODELAY): 5 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5585.4 eth0 n/a 0.0 RSS_TOTAL=104096 pages, %mem= 1.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff8801376688c0 ffff8800b6c9a800 sysfs sysfs /sys ffff880137668a80 ffff880139944000 proc proc /proc ffff880137668c40 ffff880137678000 devtmpfs devtmpfs /dev ffff880137668e00 ffff880137324800 securityfs securityfs /sys/kernel/security ffff880137668fc0 ffff8800b6c9b800 tmpfs tmpfs /dev/shm ffff880137669180 ffff880137321000 devpts devpts /dev/pts ffff880137669340 ffff8800b6c9c000 tmpfs tmpfs /run ffff880137669500 ffff8800b6c9c800 tmpfs tmpfs /sys/fs/cgroup ffff8801376696c0 ffff8800b6c9d000 cgroup cgroup /sys/fs/cgroup/systemd ffff880137669880 ffff8800b6c9d800 pstore pstore /sys/fs/pstore ffff880137669a40 ffff8800b6c9f800 cgroup cgroup /sys/fs/cgroup/devices ffff880137669c00 ffff8800b6c9f000 cgroup cgroup /sys/fs/cgroup/pids ffff880137669dc0 ffff8800b6c9e800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aa50000 ffff8800b6c9e000 cgroup cgroup /sys/fs/cgroup/memory ffff88012aa501c0 ffff88012aa80000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aa50380 ffff88012aa80800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aa50540 ffff88012aa81000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aa50700 ffff88012aa81800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aa508c0 ffff88012aa82000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aa50a80 ffff88012aa82800 cgroup cgroup /sys/fs/cgroup/freezer ffff8800b6ec81c0 ffff8800b6d3b800 configfs configfs /sys/kernel/config ffff8800b6ec8380 ffff8800b6d38000 ext4 /dev/nbd0 / ffff8800b6ec8540 ffff8800b6d3b000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012aa50c40 ffff88012aa86000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8800b6ec8c40 ffff88012a652000 hugetlbfs hugetlbfs /dev/hugepages ffff8800b6ec8e00 ffff88013771f000 mqueue mqueue /dev/mqueue ffff880138ccac40 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b6ec8fc0 ffff88012a654800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b6ec9180 ffff8800b5cc7000 ramfs none /mnt ffff88012aa50fc0 ffff8800b5c7d000 tmpfs none /var/lib/stateless/writable ffff88012b260c40 ffff8800b64e9000 squashfs /dev/vda /home/green/git/lustre-release ffff88012aa51180 ffff8800b5c7d000 tmpfs none /var/cache/man ffff88012b260e00 ffff8800b5c7d000 tmpfs none /var/log ffff8800b6ec9340 ffff8800b5c7d000 tmpfs none /var/lib/dbus ffff8800b6ec9500 ffff8800b5c7d000 tmpfs none /tmp ffff8800b6ec96c0 ffff8800b5c7d000 tmpfs none /var/lib/dhclient ffff88012aa51340 ffff8800b5c7d000 tmpfs none /var/tmp ffff8800b6ec9880 ffff8800b5c7d000 tmpfs none /var/lib/NetworkManager ffff8800b6ec9a40 ffff8800b5c7d000 tmpfs none /var/lib/systemd/random-seed ffff8800b5df4000 ffff8800b5c7d000 tmpfs none /var/spool ffff88012aa51500 ffff8800b5c7d000 tmpfs none /var/lib/nfs ffff88012aa516c0 ffff8800b5c7d000 tmpfs none /var/lib/gssproxy ffff88012aa51880 ffff8800b5c7d000 tmpfs none /var/lib/logrotate ffff8800b6ec9c00 ffff8800b5c7d000 tmpfs none /etc ffff88012aa51a40 ffff8800b5c7d000 tmpfs none /var/lib/rsyslog ffff88012aa51c00 ffff8800b5c7d000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b6ec9dc0 ffff8800b6fad800 nfs4 192.168.200.253:/exports/state/oleg330-client.virtnet /var/lib/stateless/state ffff8800b6ec88c0 ffff8800b6fad800 nfs4 192.168.200.253:/exports/state/oleg330-client.virtnet /boot ffff88012b261180 ffff8800b6fad800 nfs4 192.168.200.253:/exports/state/oleg330-client.virtnet /etc/etc/kdump.conf ffff88012b261340 ffff8800b6d3b000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8801369d2fc0 ffff8800b5cc4800 nfs4 192.168.200.253://exports/testreports/41705/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800b5df4540 ffff88012b2b2000 tmpfs tmpfs /run/user/0 ffff88012d0bae00 ffff8800b64e9000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800af7b5340 ffff8800909fc800 lustre 192.168.203.130@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 5554.988485] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 05:25:29 (1712481929) [ 5555.611863] Lustre: DEBUG MARKER: SKIP: sanity test_150h Test must be statx() syscall supported [ 5557.628293] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 05:25:32 (1712481932) [ 5566.507086] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 05:25:41 (1712481941) [ 5569.809826] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 05:25:44 (1712481944) [ 5572.984049] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 05:25:47 (1712481947) [ 5576.239032] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 05:25:51 (1712481951) [ 5579.722009] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 05:25:54 (1712481954) [ 5583.774391] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 05:25:58 (1712481958) [ 5587.658461] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 05:26:02 (1712481962) [ 5587.768310] BUG: unable to handle kernel paging request at ffff8800a4aebed8 [ 5587.769831] IP: [] ll_dir_read+0x131/0x340 [lustre] [ 5587.771273] PGD 23c1067 PUD 13e80a067 PMD 13e6e4067 PTE 80000000a4aeb060 [ 5587.772914] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [ 5587.774002] Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) tcp_diag inet_diag crc_t10dif crct10dif_generic rpcsec_gss_krb5 sb_edac edac_core iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel squashfs aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i2c_piix4 i2c_core binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel ata_piix serio_raw libata [last unloaded: libcfs] [ 5587.786931] CPU: 1 PID: 1044 Comm: bash Kdump: loaded Tainted: G W OE ------------ 3.10.0-7.9-debug #1 [ 5587.788740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 5587.790345] task: ffff880136914440 ti: ffff8800aa4b0000 task.ti: ffff8800aa4b0000 [ 5587.791736] RIP: 0010:[] [] ll_dir_read+0x131/0x340 [lustre] [ 5587.793409] RSP: 0018:ffff8800aa4b3d68 EFLAGS: 00010282 [ 5587.794405] RAX: 0000000000002e80 RBX: 7fc3ce8a92dbfd72 RCX: 0000000000000000 [ 5587.795825] RDX: 0000000000000200 RSI: ffff8800a4ae933a RDI: 0000000001aa2aa4 [ 5587.797085] RBP: ffff8800aa4b3df0 R08: 0002d1d30205d680 R09: 0000000000000000 [ 5587.798768] R10: 0000000000000000 R11: 000000000000000f R12: ffff8800a4ae9000 [ 5587.800212] R13: ffffea000292ba40 R14: ffffea0002d29d80 R15: ffff8800a4aebec8 [ 5587.801697] FS: 00007f917ec2e740(0000) GS:ffff88013e280000(0000) knlGS:0000000000000000 [ 5587.803448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5587.804538] CR2: ffff8800a4aebed8 CR3: 00000000aa584000 CR4: 0000000000160fe0 [ 5587.805864] Call Trace: [ 5587.806326] [] ? filldir64+0x130/0x130 [ 5587.807390] [] ll_readdir+0x171/0x6f0 [lustre] [ 5587.808600] [] ? filldir64+0x130/0x130 [ 5587.809643] [] iterate_dir+0x91/0x120 [ 5587.810692] [] SyS_getdents+0xc4/0x190 [ 5587.811634] [] ? filldir64+0x130/0x130 [ 5587.812538] [] system_call_fastpath+0x1f/0x24 [ 5587.813661] Code: 06 49 c1 e4 0c 49 01 c4 41 f6 44 24 10 01 4d 8d 54 24 18 4d 0f 44 fa 31 d2 90 84 d2 0f 85 bd 00 00 00 4d 85 ff 0f 84 b4 00 00 00 <4d> 8b 77 10 49 39 de 0f 82 f2 00 00 00 41 0f b7 57 1a 85 d2 0f [ 5587.818370] RIP [] ll_dir_read+0x131/0x340 [lustre] [ 5587.819564] RSP [ 5587.820226] CR2: ffff8800a4aebed8 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 9.86s (real) 5.11s (CPU), Child processes: 4.71s