************************ crashinfo ************************* /exports/testreports/50556/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg217-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.BJCB0/vmlinux [TAINTED] DUMPFILE: /exports/testreports/50556/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg217-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Mon Apr 7 13:25:46 EDT 2025 UPTIME: 00:17:03 LOAD AVERAGE: 0.02, 35.02, 60.27 TASKS: 163 NODENAME: oleg217-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +++WARNING+++ High Load Averages: 0.02, 35.02, 60.27 +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 14 last 5s 42 last 60s 43 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 159 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 27 rcuos/2 0 ms (no user stack) 2000 monitor_thread 0 ms (no user stack) 9 rcu_sched 0 ms (no user stack) 902 in:imjournal 0 ms /usr/sbin/rsyslogd -n 11 rcuos/0 8 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 800728 3.1 GB 83% of TOTAL MEM USED 154351 602.9 MB 16% of TOTAL MEM SHARED 10125 39.6 MB 1% of TOTAL MEM BUFFERS 5206 20.3 MB 0% of TOTAL MEM CACHED 46644 182.2 MB 4% of TOTAL MEM SLAB 24447 95.5 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 63134 246.6 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 5 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 1020.9 eth0 n/a 1.6 RSS_TOTAL=83544 pages, %mem= 1.3 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aacc000 ffff88012aa78000 sysfs sysfs /sys ffff88012aacc1c0 ffff880139944000 proc proc /proc ffff88012aacc380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aacc540 ffff8800b6cf2000 securityfs securityfs /sys/kernel/security ffff88012aacc700 ffff88012aa78800 tmpfs tmpfs /dev/shm ffff88012aacc8c0 ffff8801372f6000 devpts devpts /dev/pts ffff88012aacca80 ffff88012aa79000 tmpfs tmpfs /run ffff88012aaccc40 ffff88012aa79800 tmpfs tmpfs /sys/fs/cgroup ffff88012aacce00 ffff88012aa7a000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aaccfc0 ffff88012aa7a800 pstore pstore /sys/fs/pstore ffff88012aacd180 ffff88012aa7c800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aacd340 ffff88012aa7c000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aacd500 ffff88012aa7b800 cgroup cgroup /sys/fs/cgroup/memory ffff88012aacd6c0 ffff88012aa7b000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aacd880 ffff88012aa7d000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aacda40 ffff88012aa7d800 cgroup cgroup /sys/fs/cgroup/pids ffff88012aacdc00 ffff88012aa7e000 cgroup cgroup /sys/fs/cgroup/devices ffff88012aacddc0 ffff88012aa7e800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aaa8000 ffff88012aa7f000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aaa81c0 ffff88012aa7f800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012b266a80 ffff8800b7891000 configfs configfs /sys/kernel/config ffff88012aaa96c0 ffff88012abb9000 ext4 /dev/nbd0 / ffff880138ccb500 ffff8800b6cf5800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012aaa9880 ffff8800b62ea000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff88012b266c40 ffff88012abe9800 hugetlbfs hugetlbfs /dev/hugepages ffff8801376688c0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff880138ccb340 ffff8801372f7800 mqueue mqueue /dev/mqueue ffff88012b266e00 ffff8800b6cf3000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012b267180 ffff8800b6cf7000 ramfs none /mnt ffff88012b267340 ffff88012abad000 tmpfs none /var/lib/stateless/writable ffff88012aaa9a40 ffff8800b58ea000 squashfs /dev/vda /home/green/git/lustre-release ffff88012aaa9c00 ffff88012abad000 tmpfs none /var/cache/man ffff880137668a80 ffff88012abad000 tmpfs none /var/log ffff880137668c40 ffff88012abad000 tmpfs none /var/lib/dbus ffff88012aaa9dc0 ffff88012abad000 tmpfs none /tmp ffff880137668e00 ffff88012abad000 tmpfs none /var/lib/dhclient ffff880138ccb6c0 ffff88012abad000 tmpfs none /var/tmp ffff880137668fc0 ffff88012abad000 tmpfs none /var/lib/NetworkManager ffff880138ccb880 ffff88012abad000 tmpfs none /var/lib/systemd/random-seed ffff8800b34fa000 ffff88012abad000 tmpfs none /var/spool ffff880137669180 ffff88012abad000 tmpfs none /var/lib/nfs ffff880137669340 ffff88012abad000 tmpfs none /var/lib/gssproxy ffff880137669500 ffff88012abad000 tmpfs none /var/lib/logrotate ffff880138ccba40 ffff88012abad000 tmpfs none /etc ffff8801376696c0 ffff88012abad000 tmpfs none /var/lib/rsyslog ffff880138ccbc00 ffff88012abad000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b34fa1c0 ffff8800b62ed000 nfs4 192.168.200.253:/exports/state/oleg217-client.virtnet /var/lib/stateless/state ffff8800b34fa540 ffff8800b62ed000 nfs4 192.168.200.253:/exports/state/oleg217-client.virtnet /boot ffff880137669880 ffff8800b62ed000 nfs4 192.168.200.253:/exports/state/oleg217-client.virtnet /etc/etc/kdump.conf ffff8800b34fa700 ffff8800b6cf5800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b37688c0 ffff8800ae97b000 nfs4 192.168.200.253://exports/testreports/50556/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff880137668380 ffff88012bf4e800 tmpfs tmpfs /run/user/0 ffff880137668540 ffff8800b58ea000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b37681c0 ffff8800ae97b800 lustre 192.168.202.117@tcp:/lustre /mnt/lustre ffff8800aa084540 ffff8800aa2d2000 lustre 192.168.202.117@tcp:/lustre /mnt/lustre2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 317.033932] LustreError: 26305:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=2 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool= [ 317.040373] LustreError: 26311:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 317.046055] LustreError: 26311:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=2 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool= [ 317.055428] LustreError: 26275:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 317.062389] LustreError: 26275:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=2 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool= [ 317.660626] LustreError: 26025:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 317.667815] LustreError: 26025:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool= [ 317.674866] LustreError: 26066:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=4 count=2 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 317.682814] LustreError: 26066:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool= [ 318.153936] LustreError: 27527:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8800ae97b800: cannot apply new layout on [0x200000403:0x206a:0x0] : rc = -5 [ 318.159503] LustreError: 27527:0:(lov_object.c:1341:lov_layout_change()) Skipped 15 previous similar messages [ 318.163258] LustreError: 27527:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x200000403:0x206a:0x0] error -5. [ 318.167061] LustreError: 27527:0:(vvp_io.c:1903:vvp_io_init()) Skipped 6 previous similar messages [ 323.440878] 2[1670]: segfault at 8 ip 00007f1d97aa47e8 sp 00007ffcb5cad9f0 error 4 in ld-2.17.so[7f1d97a99000+22000] [ 323.781003] traps: 1[1923] general protection ip:40576f sp:7fff6d3e2c00 error:0 in 1[400000+6000] [ 326.792060] 15[4324]: segfault at 8 ip 00007fe529ce47e8 sp 00007ffef74a7730 error 4 in ld-2.17.so[7fe529cd9000+22000] [ 328.634864] 17[6234]: segfault at 8 ip 00007f8a1eada7e8 sp 00007fff291f7910 error 4 in ld-2.17.so[7f8a1eacf000+22000] [ 330.084773] traps: 2[7265] trap invalid opcode ip:40596d sp:7fff85225308 error:0 in 2[400000+6000] [ 344.327713] chmod (20003) used greatest stack depth: 10464 bytes left [ 344.643732] 18[20261]: segfault at 8 ip 00007f1abadb47e8 sp 00007fff02d8c020 error 4 in ld-2.17.so[7f1abada9000+22000] [ 351.656997] 8[26310]: segfault at 0 ip 0000000000403e5f sp 00007ffc14aca470 error 6 in 10[400000+6000] [ 353.847825] 17[28491]: segfault at 0 ip 0000000000403e5f sp 00007fffbe85ba50 error 6 in 17[400000+6000] [ 363.755725] 13[4461]: segfault at 8 ip 00007f767aba37e8 sp 00007fff2963dec0 error 4 in ld-2.17.so[7f767ab98000+22000] [ 366.282779] LustreError: 46:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 16 [0x0:0x0:0x0] inode@0000000000000000: rc = -5 [ 368.426831] 13[7932]: segfault at 406000 ip 0000000000406000 sp 00007fff14d31b38 error 14 in 13[606000+1000] [ 376.509483] Lustre: dir [0x240000404:0x11ff:0x0] stripe 2 readdir failed: -2, directory is partially accessed! [ 376.512451] Lustre: Skipped 214 previous similar messages [ 383.023296] LustreError: 19541:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8800ae97b800: cannot apply new layout on [0x240000404:0x128e:0x0] : rc = -5 [ 383.029766] LustreError: 19541:0:(lov_object.c:1341:lov_layout_change()) Skipped 60 previous similar messages [ 383.032716] LustreError: 19541:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000404:0x128e:0x0] error -5. [ 383.037863] LustreError: 19541:0:(vvp_io.c:1903:vvp_io_init()) Skipped 20 previous similar messages [ 385.377366] LustreError: 283:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 13 [0x0:0x0:0x0] inode@0000000000000000: rc = -5 [ 385.382173] LustreError: 283:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 3 previous similar messages [ 386.631304] 14[23525]: segfault at 8 ip 00007f832540a7e8 sp 00007ffe2b48fee0 error 4 in ld-2.17.so[7f83253ff000+22000] [ 395.300256] 11[31124]: segfault at 0 ip (null) sp 00007ffc93b10978 error 14 in 11[400000+6000] [ 399.633626] LustreError: 17:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x0:0x0:0x0] inode@0000000000000000: rc = -5 [ 399.638622] LustreError: 17:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 7 previous similar messages [ 412.512301] 9[15220]: segfault at 0 ip 0000000000403e5f sp 00007ffcaead4690 error 6 in 9[400000+6000] [ 412.569075] 15[15730]: segfault at 0 ip (null) sp 00007fff58429bf8 error 14 in 15[400000+6000] [ 420.334852] traps: 5[22088] trap invalid opcode ip:405a26 sp:7ffefc9953e8 error:0 in 5[400000+6000] ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) High Load Averages: 0.02, 35.02, 60.27 There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.98s (real) 5.52s (CPU), Child processes: 5.40s