************************ crashinfo ************************* /exports/testreports/48178/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg356-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.eu0dz/vmlinux [TAINTED] DUMPFILE: /exports/testreports/48178/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg356-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Tue Dec 24 23:26:24 EST 2024 UPTIME: 00:33:43 LOAD AVERAGE: 0.00, 1.41, 22.01 TASKS: 161 NODENAME: oleg356-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 12 last 5s 30 last 60s 36 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 157 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 921 in:imjournal 0 ms /usr/sbin/rsyslogd -n 12201 ldlm_bl_04 0 ms (no user stack) 20 rcuos/1 0 ms (no user stack) 2016 lnet_discovery 0 ms (no user stack) 2015 monitor_thread 110 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 807527 3.1 GB 84% of TOTAL MEM USED 147552 576.4 MB 15% of TOTAL MEM SHARED 9949 38.9 MB 1% of TOTAL MEM BUFFERS 5176 20.2 MB 0% of TOTAL MEM CACHED 46356 181.1 MB 4% of TOTAL MEM SLAB 23394 91.4 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 62996 246.1 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 5 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 2020.6 eth0 n/a 0.9 RSS_TOTAL=79440 pages, %mem= 1.3 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff880137668c40 ffff88012b28c000 sysfs sysfs /sys ffff880137668e00 ffff880139944000 proc proc /proc ffff880137668fc0 ffff880137678000 devtmpfs devtmpfs /dev ffff880137669180 ffff88012dbaa800 securityfs securityfs /sys/kernel/security ffff880137669340 ffff88012b28c800 tmpfs tmpfs /dev/shm ffff880137669500 ffff88012dba9000 devpts devpts /dev/pts ffff8801376696c0 ffff88012b28d000 tmpfs tmpfs /run ffff880137669880 ffff88012b28d800 tmpfs tmpfs /sys/fs/cgroup ffff880137669a40 ffff88012b28e000 cgroup cgroup /sys/fs/cgroup/systemd ffff880137669c00 ffff88012b28e800 pstore pstore /sys/fs/pstore ffff880137669dc0 ffff88012aad8800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aa6c000 ffff88012aad8000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aa6c1c0 ffff88012aad9000 cgroup cgroup /sys/fs/cgroup/pids ffff88012aa6c380 ffff88012aad9800 cgroup cgroup /sys/fs/cgroup/memory ffff88012aa6c540 ffff88012aada000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aa6c700 ffff88012aada800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aa6c8c0 ffff88012aadb000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aa6ca80 ffff88012aadb800 cgroup cgroup /sys/fs/cgroup/devices ffff88012aa6cc40 ffff88012aadc000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aa6ce00 ffff88012aadc800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aa6d340 ffff88012b28f000 configfs configfs /sys/kernel/config ffff88012aa6d6c0 ffff8800b6129000 ext4 /dev/nbd0 / ffff8800b62ccfc0 ffff88012aadf800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8800b62cd180 ffff88012dbaf000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880138ccaa80 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff88012aa6ddc0 ffff8801377ff800 mqueue mqueue /dev/mqueue ffff88012aa6dc00 ffff8800b63ba000 hugetlbfs hugetlbfs /dev/hugepages ffff88012b246a80 ffff8800b6025800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012b246c40 ffff8800b6021800 ramfs none /mnt ffff88012aa6d880 ffff88012a5b3000 tmpfs none /var/lib/stateless/writable ffff8800b62cd340 ffff8800b62d5800 squashfs /dev/vda /home/green/git/lustre-release ffff8800b62cd880 ffff88012a5b3000 tmpfs none /var/cache/man ffff88012aa6d180 ffff88012a5b3000 tmpfs none /var/log ffff8800b62cda40 ffff88012a5b3000 tmpfs none /var/lib/dbus ffff88012b246e00 ffff88012a5b3000 tmpfs none /tmp ffff8800b62cdc00 ffff88012a5b3000 tmpfs none /var/lib/dhclient ffff88012b246fc0 ffff88012a5b3000 tmpfs none /var/tmp ffff8800b62cddc0 ffff88012a5b3000 tmpfs none /var/lib/NetworkManager ffff88012b247180 ffff88012a5b3000 tmpfs none /var/lib/systemd/random-seed ffff88012b247340 ffff88012a5b3000 tmpfs none /var/spool ffff88012b247500 ffff88012a5b3000 tmpfs none /var/lib/nfs ffff88012b2476c0 ffff88012a5b3000 tmpfs none /var/lib/gssproxy ffff8800b3c8e000 ffff88012a5b3000 tmpfs none /var/lib/logrotate ffff8800b3c8e1c0 ffff88012a5b3000 tmpfs none /etc ffff88012b247880 ffff88012a5b3000 tmpfs none /var/lib/rsyslog ffff8800b6378000 ffff88012a5b3000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012b247a40 ffff8800b6d07000 nfs4 192.168.200.253:/exports/state/oleg356-client.virtnet /var/lib/stateless/state ffff8800b6378380 ffff8800b6d07000 nfs4 192.168.200.253:/exports/state/oleg356-client.virtnet /boot ffff8800b6378540 ffff8800b6d07000 nfs4 192.168.200.253:/exports/state/oleg356-client.virtnet /etc/etc/kdump.conf ffff8800b6378700 ffff88012aadf800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b3702e00 ffff8800b359d800 nfs4 192.168.200.253://exports/testreports/48178/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800b3c8ee00 ffff8800b5cf4000 tmpfs tmpfs /run/user/0 ffff8800b3c8f500 ffff8800b62d5800 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b3c8fa40 ffff8800b63bf000 lustre 192.168.203.156@tcp:/lustre /mnt/lustre ffff8800b5f6ae00 ffff880131174800 lustre 192.168.203.156@tcp:/lustre /mnt/lustre2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 344.510502] LustreError: 16093:0:(llite_lib.c:3736:ll_prep_inode()) Skipped 80 previous similar messages [ 346.528586] Lustre: dir [0x200000404:0x342:0x0] stripe 1 readdir failed: -2, directory is partially accessed! [ 347.842541] 19[19191]: segfault at 406000 ip 0000000000406000 sp 00007ffdb11d3318 error 14 in 19[606000+1000] [ 349.474357] 7[20574]: segfault at 8 ip 00007f57d3be67e8 sp 00007ffc95613ad0 error 4 in ld-2.17.so[7f57d3bdb000+22000] [ 351.869210] LustreError: 21249:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff880131174800: cannot apply new layout on [0x200000404:0x3b2:0x0] : rc = -5 [ 351.875104] LustreError: 21249:0:(lov_object.c:1341:lov_layout_change()) Skipped 3 previous similar messages [ 351.879104] LustreError: 21249:0:(vvp_io.c:1921:vvp_io_init()) lustre: refresh file layout [0x200000404:0x3b2:0x0] error -5. [ 351.884027] LustreError: 21249:0:(vvp_io.c:1921:vvp_io_init()) Skipped 1 previous similar message [ 354.255284] 12[24659]: segfault at 8 ip 00007f7c4d8597e8 sp 00007ffcf50f37a0 error 4 in ld-2.17.so[7f7c4d84e000+22000] [ 355.764097] LustreError: 46:0:(statahead.c:836:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x0:0x0:0x0] inode@0000000000000000: rc = -5 [ 355.768455] LustreError: 46:0:(statahead.c:836:ll_statahead_interpret_work()) Skipped 2 previous similar messages [ 363.001176] LustreError: 29590:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x200000404:0x885:0x0] dir layout mismatch: [ 363.007625] LustreError: 29590:0:(llite_lib.c:1872:ll_update_lsm_md()) Skipped 1 previous similar message [ 363.009997] LustreError: 29590:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 363.017160] LustreError: 29590:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x200000400:0x8d:0x0] [ 363.021047] LustreError: 29590:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 9 previous similar messages [ 363.025615] LustreError: 29590:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=2 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool= [ 368.904723] 13[3685]: segfault at 8 ip 00007fc3f39697e8 sp 00007ffce5865a60 error 4 in ld-2.17.so[7fc3f395e000+22000] [ 390.978931] 15[22878]: segfault at 8 ip 00007f4db29487e8 sp 00007fffe394f4c0 error 4 in ld-2.17.so[7f4db293d000+22000] [ 406.727970] traps: 17[3400] general protection ip:7f60999a6bbb sp:7ffea9559150 error:0 in ld-2.17.so[7f609999f000+22000] [ 407.659332] LustreError: 32702:0:(statahead.c:836:ll_statahead_interpret_work()) lustre: failed to prep 5 [0x0:0x0:0x0] inode@0000000000000000: rc = -5 [ 408.060689] 0[4619]: segfault at 8 ip 00007fcaca64a7e8 sp 00007ffdf5769890 error 4 in ld-2.17.so[7fcaca63f000+22000] [ 409.280223] 2[5623]: segfault at 8 ip 00007f4c615067e8 sp 00007ffc906a5740 error 4 in ld-2.17.so[7f4c614fb000+22000] [ 409.313684] 2[6007]: segfault at 8 ip 00007ff84c1587e8 sp 00007ffccadf4420 error 4 in ld-2.17.so[7ff84c14d000+22000] [ 410.292606] dd (6205) used greatest stack depth: 10592 bytes left [ 413.129936] Lustre: dir [0x240000404:0x14de:0x0] stripe 2 readdir failed: -2, directory is partially accessed! [ 413.134333] Lustre: Skipped 22 previous similar messages [ 417.892736] LustreError: 590:0:(statahead.c:836:ll_statahead_interpret_work()) lustre: failed to prep sleep [0x240000404:0x15af:0x0] inode@0000000000000000: rc = -5 [ 417.898497] LustreError: 590:0:(statahead.c:836:ll_statahead_interpret_work()) Skipped 2 previous similar messages [ 420.876892] LustreError: 15559:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8800b63bf000: cannot apply new layout on [0x200000405:0x17f7:0x0] : rc = -5 [ 420.883007] LustreError: 15559:0:(lov_object.c:1341:lov_layout_change()) Skipped 14 previous similar messages [ 421.089224] 5[15811]: segfault at 8 ip 00007f3cc072b7e8 sp 00007ffeb4770320 error 4 in ld-2.17.so[7f3cc0720000+22000] [ 427.093350] 15[20957]: segfault at 8 ip 00007ffb81f8c7e8 sp 00007ffc3856dbd0 error 4 in ld-2.17.so[7ffb81f81000+22000] [ 427.960021] 11[21707]: segfault at 8 ip 00007f378e0e57e8 sp 00007ffffbd251e0 error 4 in ld-2.17.so[7f378e0da000+22000] [ 429.145224] 15[22437]: segfault at 8 ip 00007fa989a4b7e8 sp 00007fffd7f934d0 error 4 in ld-2.17.so[7fa989a40000+22000] [ 441.305253] 6[32378]: segfault at 8 ip 00007f9192c4f7e8 sp 00007ffe2d37c2e0 error 4 in ld-2.17.so[7f9192c44000+22000] [ 447.365298] 18[5294]: segfault at 8 ip 00007f28f10c47e8 sp 00007ffc31fbe5f0 error 4 in ld-2.17.so[7f28f10b9000+22000] [ 448.627237] 15[6429]: segfault at 406000 ip 0000000000406000 sp 00007ffcafb83928 error 14 in 15[606000+1000] [ 1079.694283] Lustre: lustre-OST0000-osc-ffff8800b63bf000: disconnect after 24s idle [ 1079.696894] Lustre: Skipped 1 previous similar message ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 11.32s (real) 5.71s (CPU), Child processes: 5.58s