************************ crashinfo ************************* /exports/testreports/47101/testresults/replay-single-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg206-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.ytOnW/vmlinux [TAINTED] DUMPFILE: /exports/testreports/47101/testresults/replay-single-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg206-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Mon Nov 11 15:07:50 EST 2024 UPTIME: 02:41:11 LOAD AVERAGE: 0.00, 0.01, 0.05 TASKS: 149 NODENAME: oleg206-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2400 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 14 last 5s 25 last 60s 38 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 145 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 10363 ldlm_bl_02 0 ms (no user stack) 34 rcuos/3 0 ms (no user stack) 1561 kworker/0:2 0 ms (no user stack) 920 in:imjournal 0 ms /usr/sbin/rsyslogd -n 2005 ptlrpcd_00_01 199 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 843618 3.2 GB 88% of TOTAL MEM USED 111461 435.4 MB 11% of TOTAL MEM SHARED 12271 47.9 MB 1% of TOTAL MEM BUFFERS 5227 20.4 MB 0% of TOTAL MEM CACHED 57382 224.1 MB 6% of TOTAL MEM SLAB 13007 50.8 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 64420 251.6 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 5 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 9668.9 eth0 n/a 0.5 RSS_TOTAL=77992 pages, %mem= 1.2 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aae6000 ffff8800b6cdd800 sysfs sysfs /sys ffff88012aae61c0 ffff880139944000 proc proc /proc ffff88012aae6380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aae6540 ffff8800b6cab000 securityfs securityfs /sys/kernel/security ffff88012aae6700 ffff8800b6cde000 tmpfs tmpfs /dev/shm ffff88012aae68c0 ffff8801372f6000 devpts devpts /dev/pts ffff88012aae6a80 ffff8800b6cde800 tmpfs tmpfs /run ffff88012aae6c40 ffff8800b6cdf000 tmpfs tmpfs /sys/fs/cgroup ffff88012aae6e00 ffff8800b6cdf800 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aae6fc0 ffff88012aaa8000 pstore pstore /sys/fs/pstore ffff88012aae7180 ffff88012aaaa000 cgroup cgroup /sys/fs/cgroup/devices ffff88012aae7340 ffff88012aaa9800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aae7500 ffff88012aaa9000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aae76c0 ffff88012aaa8800 cgroup cgroup /sys/fs/cgroup/memory ffff88012aae7880 ffff88012aaaa800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aae7a40 ffff88012aaab000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aae7c00 ffff88012aaab800 cgroup cgroup /sys/fs/cgroup/pids ffff88012aae7dc0 ffff88012aaac000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aafc000 ffff88012aaac800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aafc1c0 ffff88012aaad000 cgroup cgroup /sys/fs/cgroup/freezer ffff880137668380 ffff88013767a000 configfs configfs /sys/kernel/config ffff88012b2a6a80 ffff8800b6319000 ext4 /dev/nbd0 / ffff88012aafca80 ffff88012aaaf800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012aafcfc0 ffff88012a479000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880137668540 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff880137668700 ffff8801372f7800 mqueue mqueue /dev/mqueue ffff8801376688c0 ffff8800b6cab800 hugetlbfs hugetlbfs /dev/hugepages ffff880137668a80 ffff8800b6cae000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012aafd180 ffff88012a548000 ramfs none /mnt ffff88012b2a68c0 ffff8800b6311800 tmpfs none /var/lib/stateless/writable ffff88012b2a6c40 ffff8800b5bb7000 squashfs /dev/vda /home/green/git/lustre-release ffff88012b2a6e00 ffff8800b6311800 tmpfs none /var/cache/man ffff88012aafd340 ffff8800b6311800 tmpfs none /var/log ffff88012aafd500 ffff8800b6311800 tmpfs none /var/lib/dbus ffff88012aafd6c0 ffff8800b6311800 tmpfs none /tmp ffff88012b2a6fc0 ffff8800b6311800 tmpfs none /var/lib/dhclient ffff88012aafd880 ffff8800b6311800 tmpfs none /var/tmp ffff880137668e00 ffff8800b6311800 tmpfs none /var/lib/NetworkManager ffff88012aafda40 ffff8800b6311800 tmpfs none /var/lib/systemd/random-seed ffff88012b2a7180 ffff8800b6311800 tmpfs none /var/spool ffff880137668fc0 ffff8800b6311800 tmpfs none /var/lib/nfs ffff88012b2a7340 ffff8800b6311800 tmpfs none /var/lib/gssproxy ffff880137669180 ffff8800b6311800 tmpfs none /var/lib/logrotate ffff88012aafdc00 ffff8800b6311800 tmpfs none /etc ffff88012b2a7500 ffff8800b6311800 tmpfs none /var/lib/rsyslog ffff88012b2a76c0 ffff8800b6311800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012b2a7880 ffff8800b5bb9800 nfs4 192.168.200.253:/exports/state/oleg206-client.virtnet /var/lib/stateless/state ffff88012aafddc0 ffff8800b5bb9800 nfs4 192.168.200.253:/exports/state/oleg206-client.virtnet /boot ffff88012aafce00 ffff8800b5bb9800 nfs4 192.168.200.253:/exports/state/oleg206-client.virtnet /etc/etc/kdump.conf ffff88012aafcc40 ffff88012aaaf800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b6f0c540 ffff8800b5bb3800 nfs4 192.168.200.253://exports/testreports/47101/testresults/replay-single-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff88012b176c40 ffff88012a4f4000 tmpfs tmpfs /run/user/0 ffff880138ccb6c0 ffff8800b5bb7000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b6f0d180 ffff8800aa675000 lustre 192.168.202.106@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 6053.343017] Lustre: Unmounted lustre-client [ 6064.074958] Lustre: Mounted lustre-client [ 6066.393216] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 14:07:44 (1731352064) [ 6069.613899] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 6101.828712] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1731352084/real 1731352084] req@ffff88012e32df80 x1815448009999488/t0(0) o4->lustre-OST0000-osc-ffff8800aa675000@192.168.202.106@tcp:6/4 lens 488/448 e 0 to 1 dl 1731352100 ref 2 fl Rpc:XQr/202/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 [ 6101.845383] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 6598.544693] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection to lustre-OST0000 (at 192.168.202.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6598.555549] Lustre: Skipped 34 previous similar messages [ 6598.560822] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection restored to (at 192.168.202.106@tcp) [ 6598.568232] Lustre: Skipped 34 previous similar messages [ 6710.720662] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1731352693/real 1731352693] req@ffff88012e32df80 x1815448009999488/t0(0) o4->lustre-OST0000-osc-ffff8800aa675000@192.168.202.106@tcp:6/4 lens 488/448 e 0 to 1 dl 1731352709 ref 2 fl Rpc:XQr/202/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 [ 6710.737395] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) Skipped 37 previous similar messages [ 7206.560707] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection to lustre-OST0000 (at 192.168.202.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7206.571229] Lustre: Skipped 37 previous similar messages [ 7206.576629] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection restored to (at 192.168.202.106@tcp) [ 7206.580883] Lustre: Skipped 37 previous similar messages [ 7318.880651] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1731353301/real 1731353301] req@ffff88012e32df80 x1815448009999488/t0(0) o4->lustre-OST0000-osc-ffff8800aa675000@192.168.202.106@tcp:6/4 lens 488/448 e 0 to 1 dl 1731353317 ref 2 fl Rpc:XQr/202/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 [ 7318.889498] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) Skipped 37 previous similar messages [ 7814.652618] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection to lustre-OST0000 (at 192.168.202.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7814.660673] Lustre: Skipped 37 previous similar messages [ 7814.666136] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection restored to (at 192.168.202.106@tcp) [ 7814.670857] Lustre: Skipped 37 previous similar messages [ 7926.835702] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1731353909/real 1731353909] req@ffff88012e32df80 x1815448009999488/t0(0) o4->lustre-OST0000-osc-ffff8800aa675000@192.168.202.106@tcp:6/4 lens 488/448 e 0 to 1 dl 1731353925 ref 2 fl Rpc:XQr/202/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 [ 7926.855014] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) Skipped 37 previous similar messages [ 8423.376616] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection to lustre-OST0000 (at 192.168.202.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8423.386543] Lustre: Skipped 37 previous similar messages [ 8423.391880] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection restored to (at 192.168.202.106@tcp) [ 8423.397131] Lustre: Skipped 37 previous similar messages [ 8535.569694] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1731354518/real 1731354518] req@ffff88012e32df80 x1815448009999488/t0(0) o4->lustre-OST0000-osc-ffff8800aa675000@192.168.202.106@tcp:6/4 lens 488/448 e 0 to 1 dl 1731354534 ref 2 fl Rpc:XQr/202/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 [ 8535.577309] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) Skipped 37 previous similar messages [ 9031.880871] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection to lustre-OST0000 (at 192.168.202.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9031.888597] Lustre: Skipped 37 previous similar messages [ 9031.893706] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection restored to (at 192.168.202.106@tcp) [ 9031.897874] Lustre: Skipped 37 previous similar messages [ 9143.915772] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1731355126/real 1731355126] req@ffff88012e32df80 x1815448009999488/t0(0) o4->lustre-OST0000-osc-ffff8800aa675000@192.168.202.106@tcp:6/4 lens 488/448 e 0 to 1 dl 1731355142 ref 2 fl Rpc:XQr/202/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 [ 9143.928828] Lustre: 2004:0:(client.c:2363:ptlrpc_expire_one_request()) Skipped 37 previous similar messages [ 9640.226661] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection to lustre-OST0000 (at 192.168.202.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9640.230732] Lustre: Skipped 37 previous similar messages [ 9640.233559] Lustre: lustre-OST0000-osc-ffff8800aa675000: Connection restored to (at 192.168.202.106@tcp) [ 9640.237306] Lustre: Skipped 37 previous similar messages ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.33s (real) 5.30s (CPU), Child processes: 4.99s