************************ crashinfo ************************* /exports/testreports/50545/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg129-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.E94N2/vmlinux [TAINTED] DUMPFILE: /exports/testreports/50545/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg129-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Mon Apr 7 06:11:56 EDT 2025 UPTIME: 00:17:02 LOAD AVERAGE: 0.02, 35.12, 60.90 TASKS: 161 NODENAME: oleg129-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2400 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +++WARNING+++ High Load Averages: 0.02, 35.12, 60.90 +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 15 last 5s 35 last 60s 42 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 157 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 20 rcuos/1 0 ms (no user stack) 11 rcuos/0 0 ms (no user stack) 9 rcu_sched 0 ms (no user stack) 909 in:imjournal 0 ms /usr/sbin/rsyslogd -n 563 systemd-journal 7 ms /usr/lib/systemd/systemd-journald +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 789822 3 GB 82% of TOTAL MEM USED 165257 645.5 MB 17% of TOTAL MEM SHARED 16529 64.6 MB 1% of TOTAL MEM BUFFERS 5202 20.3 MB 0% of TOTAL MEM CACHED 53005 207.1 MB 5% of TOTAL MEM SLAB 28087 109.7 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 62959 245.9 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 5 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 1020.2 eth0 n/a 2.6 RSS_TOTAL=82260 pages, %mem= 1.4 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aa6a000 ffff8800b6ce9000 sysfs sysfs /sys ffff88012aa6a1c0 ffff880139944000 proc proc /proc ffff88012aa6a380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aa6a540 ffff88012a9f8800 securityfs securityfs /sys/kernel/security ffff88012aa6a700 ffff8800b6ce9800 tmpfs tmpfs /dev/shm ffff88012aa6a8c0 ffff880137112800 devpts devpts /dev/pts ffff88012aa6aa80 ffff8800b6cea000 tmpfs tmpfs /run ffff88012aa6ac40 ffff8800b6cea800 tmpfs tmpfs /sys/fs/cgroup ffff88012aa6ae00 ffff8800b6ceb000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aa6afc0 ffff8800b6ceb800 pstore pstore /sys/fs/pstore ffff88012aa6b180 ffff8800b6ced800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aa6b340 ffff8800b6ced000 cgroup cgroup /sys/fs/cgroup/pids ffff88012aa6b500 ffff8800b6cec800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aa6b6c0 ffff8800b6cec000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aa6b880 ffff8800b6cee000 cgroup cgroup /sys/fs/cgroup/memory ffff88012aa6ba40 ffff8800b6cee800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aa6bc00 ffff8800b6cef000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aa6bdc0 ffff8800b6cef800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aa36000 ffff88012aae8000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aa361c0 ffff88012aae8800 cgroup cgroup /sys/fs/cgroup/devices ffff880138ccbdc0 ffff88012a9f9800 configfs configfs /sys/kernel/config ffff8800b626cfc0 ffff88012a58c000 ext4 /dev/nbd0 / ffff8800b6d50000 ffff88013767f000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8800b6d501c0 ffff8800b6278000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8800b6d50380 ffff8800b627b000 hugetlbfs hugetlbfs /dev/hugepages ffff8800b6d50540 ffff880137114000 mqueue mqueue /dev/mqueue ffff8800b626d180 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff88012aa368c0 ffff88012a58a800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b626d340 ffff8800b6396800 ramfs none /mnt ffff8800b626d500 ffff8800b633b000 tmpfs none /var/lib/stateless/writable ffff8800b6d50700 ffff8800b63e6000 squashfs /dev/vda /home/green/git/lustre-release ffff8801376688c0 ffff8800b633b000 tmpfs none /var/cache/man ffff8800b626d880 ffff8800b633b000 tmpfs none /var/log ffff88012aa36a80 ffff8800b633b000 tmpfs none /var/lib/dbus ffff880137668a80 ffff8800b633b000 tmpfs none /tmp ffff880137668c40 ffff8800b633b000 tmpfs none /var/lib/dhclient ffff88012aa36c40 ffff8800b633b000 tmpfs none /var/tmp ffff88012aa36e00 ffff8800b633b000 tmpfs none /var/lib/NetworkManager ffff88012aa36fc0 ffff8800b633b000 tmpfs none /var/lib/systemd/random-seed ffff88012aa37180 ffff8800b633b000 tmpfs none /var/spool ffff88012aa37340 ffff8800b633b000 tmpfs none /var/lib/nfs ffff88012aa37500 ffff8800b633b000 tmpfs none /var/lib/gssproxy ffff88012aa376c0 ffff8800b633b000 tmpfs none /var/lib/logrotate ffff880137668e00 ffff8800b633b000 tmpfs none /etc ffff88012aa37880 ffff8800b633b000 tmpfs none /var/lib/rsyslog ffff88012aa37a40 ffff8800b633b000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012aa37c00 ffff8800b4008800 nfs4 192.168.200.253:/exports/state/oleg129-client.virtnet /var/lib/stateless/state ffff88012aa37dc0 ffff8800b4008800 nfs4 192.168.200.253:/exports/state/oleg129-client.virtnet /boot ffff8800b626da40 ffff8800b4008800 nfs4 192.168.200.253:/exports/state/oleg129-client.virtnet /etc/etc/kdump.conf ffff88012aa36700 ffff88013767f000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b4319c00 ffff88012a61c000 nfs4 192.168.200.253://exports/testreports/50545/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff88012bf81180 ffff8800b41ed800 tmpfs tmpfs /run/user/0 ffff8800b6d51340 ffff8800b63e6000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b6d51c00 ffff88013767e800 lustre 192.168.201.129@tcp:/lustre /mnt/lustre ffff8800b626c540 ffff8800b63e0800 lustre 192.168.201.129@tcp:/lustre /mnt/lustre2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 359.399714] 17[30993]: segfault at 8 ip 00007f21c625e7e8 sp 00007ffe2ba08ae0 error 4 in ld-2.17.so[7f21c6253000+22000] [ 365.886920] 6[4222]: segfault at 8 ip 00007f7e1efcd7e8 sp 00007fff3242b060 error 4 in ld-2.17.so[7f7e1efc2000+22000] [ 372.035932] LustreError: 9013:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b63e0800: inode [0x200000402:0x7ac0:0x0] mdc close failed: rc = -116 [ 372.043506] LustreError: 9013:0:(file.c:247:ll_close_inode_openhandle()) Skipped 390 previous similar messages [ 372.942278] LustreError: 7706:0:(llite_lib.c:1883:ll_update_lsm_md()) lustre: [0x240000403:0x4aab:0x0] dir layout mismatch: [ 372.946491] LustreError: 7706:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=1 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 372.954558] LustreError: 7706:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x240000400:0x179:0x0] [ 372.961873] LustreError: 7706:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 5 previous similar messages [ 372.965344] LustreError: 7706:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=2 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool= [ 372.972861] LustreError: 7704:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=1 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 372.977187] LustreError: 7704:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=2 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool= [ 373.971024] 19[10233]: segfault at 8 ip 00007f5cfdb007e8 sp 00007ffd24485960 error 4 in ld-2.17.so[7f5cfdaf5000+22000] [ 377.155564] 7[13181]: segfault at 0 ip 0000000000403e5f sp 00007fff514ba2c0 error 6 in 7[400000+6000] [ 379.035617] LustreError: 14664:0:(lcommon_cl.c:179:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x4a56:0x0]: rc = -5 [ 379.039116] LustreError: 14664:0:(lcommon_cl.c:179:cl_file_inode_init()) Skipped 835 previous similar messages [ 379.041060] LustreError: 14664:0:(llite_lib.c:3733:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5 [ 379.043916] LustreError: 14664:0:(llite_lib.c:3733:ll_prep_inode()) Skipped 836 previous similar messages [ 382.337129] 14[17441]: segfault at 8 ip 00007fd710fc87e8 sp 00007ffc2142e6d0 error 4 in ld-2.17.so[7fd710fbd000+22000] [ 383.172181] 5[17766]: segfault at 8 ip 00007f39739347e8 sp 00007fff940c0d10 error 4 in ld-2.17.so[7f3973929000+22000] [ 383.242289] LustreError: 17315:0:(llite_lib.c:1883:ll_update_lsm_md()) lustre: [0x240000404:0x1409:0x0] dir layout mismatch: [ 383.246154] LustreError: 17315:0:(llite_lib.c:1883:ll_update_lsm_md()) Skipped 1 previous similar message [ 383.248309] LustreError: 17315:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 383.253944] LustreError: 17315:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x240000400:0x18f:0x0] [ 383.256633] LustreError: 17315:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 5 previous similar messages [ 383.259324] LustreError: 17315:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool= [ 383.270105] LustreError: 17312:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 383.277310] LustreError: 17312:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool= [ 399.128329] 17[29825]: segfault at 8 ip 00007f04e9c2a7e8 sp 00007ffe11980620 error 4 in ld-2.17.so[7f04e9c1f000+22000] [ 400.294260] Lustre: dir [0x240000403:0x52c5:0x0] stripe 2 readdir failed: -2, directory is partially accessed! [ 400.297964] Lustre: Skipped 65 previous similar messages [ 402.578577] 11[31941]: segfault at 406000 ip 0000000000406000 sp 00007ffc883acaa8 error 14 in 11[606000+1000] [ 403.761510] 2[32542]: segfault at 8 ip 00007f8a4545f7e8 sp 00007ffd9c3267c0 error 4 in ld-2.17.so[7f8a45454000+22000] [ 405.866751] 19[1472]: segfault at 0 ip 00000000004016cd sp 00007ffc31bc7cd0 error 6 in 19[400000+6000] [ 409.726241] LustreError: 3957:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8800b63e0800: cannot apply new layout on [0x240000404:0x1a23:0x0] : rc = -5 [ 409.732383] LustreError: 3957:0:(lov_object.c:1341:lov_layout_change()) Skipped 96 previous similar messages [ 410.235108] LustreError: 46:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 14 [0x0:0x0:0x0] inode@0000000000000000: rc = -5 [ 410.240657] LustreError: 46:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 11 previous similar messages [ 413.563152] 7[6913]: segfault at 8 ip 00007f7db6a737e8 sp 00007ffd781e3d70 error 4 in ld-2.17.so[7f7db6a68000+22000] [ 413.697885] dir_create.sh (10620) used greatest stack depth: 10336 bytes left [ 416.232266] 11[8313]: segfault at 8 ip 00007fee988f07e8 sp 00007ffe83447140 error 4 in ld-2.17.so[7fee988e5000+22000] ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) High Load Averages: 0.02, 35.12, 60.90 There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.97s (real) 5.48s (CPU), Child processes: 5.44s