************************ crashinfo ************************* /exports/testreports/48496/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg233-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.vA9L1/vmlinux [TAINTED] DUMPFILE: /exports/testreports/48496/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg233-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Mon Jan 13 11:41:23 EST 2025 UPTIME: 00:16:59 LOAD AVERAGE: 0.02, 35.02, 59.95 TASKS: 160 NODENAME: oleg233-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +++WARNING+++ High Load Averages: 0.02, 35.02, 59.95 +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 13 last 5s 33 last 60s 39 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 156 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 2004 monitor_thread 0 ms (no user stack) 1 systemd 0 ms /usr/lib/systemd/systemd --switched-root --system --deserialize 21 1998 socknal_reaper 6 ms (no user stack) 2005 lnet_discovery 21 ms (no user stack) 1126 sshd 38 ms sshd: root@pts/0 +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 790171 3 GB 82% of TOTAL MEM USED 164908 644.2 MB 17% of TOTAL MEM SHARED 12202 47.7 MB 1% of TOTAL MEM BUFFERS 5175 20.2 MB 0% of TOTAL MEM CACHED 48626 189.9 MB 5% of TOTAL MEM SLAB 30509 119.2 MB 3% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 62952 245.9 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 5 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 1017.3 eth0 n/a 0.0 RSS_TOTAL=78772 pages, %mem= 1.3 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff880137668380 ffff8800b6cfa000 sysfs sysfs /sys ffff880137668540 ffff880139944000 proc proc /proc ffff880137668700 ffff880137678000 devtmpfs devtmpfs /dev ffff8801376688c0 ffff8800b7c19000 securityfs securityfs /sys/kernel/security ffff880137668a80 ffff8800b6cfa800 tmpfs tmpfs /dev/shm ffff880137668c40 ffff8801372f6000 devpts devpts /dev/pts ffff880137668e00 ffff8800b6cfb000 tmpfs tmpfs /run ffff880137668fc0 ffff8800b6cfb800 tmpfs tmpfs /sys/fs/cgroup ffff880137669180 ffff8800b6cfc000 cgroup cgroup /sys/fs/cgroup/systemd ffff880137669340 ffff8800b6cfc800 pstore pstore /sys/fs/pstore ffff880137669500 ffff8800b6cfe800 cgroup cgroup /sys/fs/cgroup/perf_event ffff8801376696c0 ffff8800b6cfe000 cgroup cgroup /sys/fs/cgroup/freezer ffff880137669880 ffff8800b6cfd800 cgroup cgroup /sys/fs/cgroup/devices ffff880137669a40 ffff8800b6cfd000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff880137669c00 ffff8800b6cff000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff880137669dc0 ffff8800b6cff800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aaca000 ffff88012aad8000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aaca1c0 ffff88012aad8800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aaca380 ffff88012aad9000 cgroup cgroup /sys/fs/cgroup/memory ffff88012aaca540 ffff88012aad9800 cgroup cgroup /sys/fs/cgroup/pids ffff88012aacaa80 ffff88012aadd800 configfs configfs /sys/kernel/config ffff8800b6632000 ffff88012a46b800 ext4 /dev/nbd0 / ffff8800b6e82000 ffff88012aadf000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012b23ae00 ffff8800b67e7800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8800b6e82700 ffff88012a46d800 hugetlbfs hugetlbfs /dev/hugepages ffff8800b6e828c0 ffff8801372f7800 mqueue mqueue /dev/mqueue ffff88012aacae00 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b6e82a80 ffff88012a46e000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b6e82e00 ffff88012a46d000 ramfs none /mnt ffff8800b66321c0 ffff8800b580b000 tmpfs none /var/lib/stateless/writable ffff8800b6e82fc0 ffff8800b5b54000 squashfs /dev/vda /home/green/git/lustre-release ffff88012aacafc0 ffff8800b580b000 tmpfs none /var/cache/man ffff88012aacb180 ffff8800b580b000 tmpfs none /var/log ffff88012aacb340 ffff8800b580b000 tmpfs none /var/lib/dbus ffff8800b6632380 ffff8800b580b000 tmpfs none /tmp ffff88012aacb500 ffff8800b580b000 tmpfs none /var/lib/dhclient ffff8800b6e83180 ffff8800b580b000 tmpfs none /var/tmp ffff88012aacb6c0 ffff8800b580b000 tmpfs none /var/lib/NetworkManager ffff8800b6632540 ffff8800b580b000 tmpfs none /var/lib/systemd/random-seed ffff88012aacb880 ffff8800b580b000 tmpfs none /var/spool ffff88012aacba40 ffff8800b580b000 tmpfs none /var/lib/nfs ffff8800b6632700 ffff8800b580b000 tmpfs none /var/lib/gssproxy ffff88012aacbc00 ffff8800b580b000 tmpfs none /var/lib/logrotate ffff8800b66328c0 ffff8800b580b000 tmpfs none /etc ffff88012aacbdc0 ffff8800b580b000 tmpfs none /var/lib/rsyslog ffff88012aaca8c0 ffff8800b580b000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b3f66000 ffff8800b5b55000 nfs4 192.168.200.253:/exports/state/oleg233-client.virtnet /var/lib/stateless/state ffff8800b6e83500 ffff8800b5b55000 nfs4 192.168.200.253:/exports/state/oleg233-client.virtnet /boot ffff8800b3f66380 ffff8800b5b55000 nfs4 192.168.200.253:/exports/state/oleg233-client.virtnet /etc/etc/kdump.conf ffff8800b3f66540 ffff88012aadf000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b349f6c0 ffff88012a5a7800 nfs4 192.168.200.253://exports/testreports/48496/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800b3f66fc0 ffff8800b58a0000 tmpfs tmpfs /run/user/0 ffff8800b3f67880 ffff8800b5b54000 squashfs /dev/vda /usr/sbin/mount.lustre ffff88012b23b880 ffff8800b7c1e800 lustre 192.168.202.133@tcp:/lustre /mnt/lustre ffff8800b349fa40 ffff8800b5a39000 lustre 192.168.202.133@tcp:/lustre /mnt/lustre2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 350.911429] 16[7386]: segfault at 8 ip 00007f4599b7e7e8 sp 00007fff60d72f20 error 4 in ld-2.17.so[7f4599b73000+22000] [ 351.665808] 6[8239]: segfault at 8 ip 00007f2f2845a7e8 sp 00007ffff0d044d0 error 4 in ld-2.17.so[7f2f2844f000+22000] [ 352.772944] 5[9341]: segfault at 8 ip 00007f9643c7a7e8 sp 00007fff3fa9f1a0 error 4 in ld-2.17.so[7f9643c6f000+22000] [ 354.864178] 3[11737]: segfault at 8 ip 00007f028e4717e8 sp 00007fff5c4cc900 error 4 in ld-2.17.so[7f028e466000+22000] [ 355.268681] 16[11956]: segfault at 8 ip 00007f215752a7e8 sp 00007fff9449d510 error 4 in ld-2.17.so[7f215751f000+22000] [ 358.243726] 18[14861]: segfault at 8 ip 00007f0aa87937e8 sp 00007ffd3424bce0 error 4 in ld-2.17.so[7f0aa8788000+22000] [ 358.431792] LustreError: 15040:0:(llite_lib.c:1881:ll_update_lsm_md()) lustre: [0x240000402:0x5a55:0x0] dir layout mismatch: [ 358.434275] LustreError: 15040:0:(llite_lib.c:1881:ll_update_lsm_md()) Skipped 1 previous similar message [ 358.436226] LustreError: 15040:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=1 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 358.440439] LustreError: 15040:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x240000400:0x1b0:0x0] [ 358.442812] LustreError: 15040:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 7 previous similar messages [ 358.444941] LustreError: 15040:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=crush:2000003 pool= [ 358.449619] LustreError: 15038:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=1 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 358.458000] LustreError: 15038:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=crush:2000003 pool= [ 358.467823] LustreError: 15044:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=1 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 358.472338] LustreError: 15044:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=crush:2000003 pool= [ 361.748442] 13[18007]: segfault at 0 ip 0000000000403e5f sp 00007ffcea2cb1c0 error 6 in 13[400000+6000] [ 371.048724] 7[27191]: segfault at 8 ip 00007f394f8157e8 sp 00007ffdea766fe0 error 4 in ld-2.17.so[7f394f80a000+22000] [ 373.798801] LustreError: 30189:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 373.805642] LustreError: 30189:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:2000003 pool= [ 373.965681] LustreError: 542:0:(statahead.c:836:ll_statahead_interpret_work()) lustre: failed to prep 2 [0x0:0x0:0x0] inode@0000000000000000: rc = -5 [ 373.970118] LustreError: 542:0:(statahead.c:836:ll_statahead_interpret_work()) Skipped 10 previous similar messages [ 377.033195] LustreError: 391:0:(file.c:267:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b7c1e800: inode [0x240000403:0x607e:0x0] mdc close failed: rc = -116 [ 377.036533] LustreError: 391:0:(file.c:267:ll_close_inode_openhandle()) Skipped 196 previous similar messages [ 382.041375] LustreError: 5292:0:(lcommon_cl.c:195:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000402:0x6732:0x0]: rc = -5 [ 382.045281] LustreError: 5292:0:(lcommon_cl.c:195:cl_file_inode_init()) Skipped 644 previous similar messages [ 382.053959] LustreError: 5292:0:(llite_lib.c:3745:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5 [ 382.056900] LustreError: 5292:0:(llite_lib.c:3745:ll_prep_inode()) Skipped 645 previous similar messages [ 385.735373] LustreError: 7241:0:(llite_lib.c:1881:ll_update_lsm_md()) lustre: [0x200000402:0x6c7c:0x0] dir layout mismatch: [ 385.739776] LustreError: 7241:0:(llite_lib.c:1881:ll_update_lsm_md()) Skipped 3 previous similar messages [ 385.743196] LustreError: 7241:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 385.749996] LustreError: 7241:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x200000400:0x1c2:0x0] [ 385.753460] LustreError: 7241:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 17 previous similar messages [ 385.756946] LustreError: 7241:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool= [ 388.503812] 5[11695]: segfault at 8 ip 00007f60efe867e8 sp 00007ffd822cbc10 error 4 in ld-2.17.so[7f60efe7b000+22000] [ 396.569263] 15[18632]: segfault at 0 ip 00007f6ef0fb5070 sp 00007ffc6ef56078 error 4 in ld-2.17.so[7f6ef0faa000+22000] [ 396.691593] 11[18561]: segfault at 8 ip 00007f88dfa2d7e8 sp 00007ffd60603460 error 4 in ld-2.17.so[7f88dfa22000+22000] [ 402.721898] 7[24346]: segfault at 8 ip 00007f394ee1b7e8 sp 00007ffed0cbdac0 error 4 in ld-2.17.so[7f394ee10000+22000] [ 415.163796] Lustre: dir [0x240000403:0x6fbb:0x0] stripe 1 readdir failed: -2, directory is partially accessed! [ 415.167072] Lustre: Skipped 71 previous similar messages ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) High Load Averages: 0.02, 35.02, 59.95 There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.79s (real) 5.40s (CPU), Child processes: 5.34s