************************ crashinfo ************************* /exports/testreports/40573/testresults/conf-sanity-special7-zfs-centos7_x86_64-centos7_x86_64-retry2/oleg134-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.6DPNr/vmlinux [TAINTED] DUMPFILE: /exports/testreports/40573/testresults/conf-sanity-special7-zfs-centos7_x86_64-centos7_x86_64-retry2/oleg134-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu Mar 7 00:46:27 EST 2024 UPTIME: 01:08:10 LOAD AVERAGE: 0.47, 0.24, 0.21 TASKS: 348 NODENAME: oleg134-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2400 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 35 last 5s 137 last 60s 148 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 344 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 32595 ll_evictor 0 ms (no user stack) 32632 osp-pre-0-0 0 ms (no user stack) 24843 ptlrpcd_00_00 0 ms (no user stack) 1150 ll_ost_create00 0 ms (no user stack) 15472 ll_ost_create00 0 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 694868 2.7 GB 72% of TOTAL MEM USED 260199 1016.4 MB 27% of TOTAL MEM SHARED 8860 34.6 MB 0% of TOTAL MEM BUFFERS 5106 19.9 MB 0% of TOTAL MEM CACHED 66636 260.3 MB 6% of TOTAL MEM SLAB 64503 252 MB 6% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 60033 234.5 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 5 LISTEN 3 NAGLE disabled (TCP_NODELAY): 4 user_data set (NFS etc.): 4 Unusual Situations: Doing Retransmission: 1 (run xportshow --retrans for details) UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 4088.0 eth0 n/a 0.1 RSS_TOTAL=50460 pages, %mem= 0.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012a264000 ffff8800b7e7a000 sysfs sysfs /sys ffff88012a2641c0 ffff880139944000 proc proc /proc ffff88012a264380 ffff880137668000 devtmpfs devtmpfs /dev ffff88012a264540 ffff8800b5209800 securityfs securityfs /sys/kernel/security ffff88012a264700 ffff8800b7e7a800 tmpfs tmpfs /dev/shm ffff88012a2648c0 ffff88012b258000 devpts devpts /dev/pts ffff88012a264a80 ffff8800b7e7b000 tmpfs tmpfs /run ffff88012a264c40 ffff8800b7e7b800 tmpfs tmpfs /sys/fs/cgroup ffff88012a264e00 ffff8800b7e7c000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012a264fc0 ffff8800b7e7c800 pstore pstore /sys/fs/pstore ffff88012a265180 ffff8800b7e7e800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a265340 ffff8800b7e7e000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a265500 ffff8800b7e7d800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a2656c0 ffff8800b7e7d000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a265880 ffff8800b7e7f000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012a265a40 ffff8800b7e7f800 cgroup cgroup /sys/fs/cgroup/devices ffff88012a265c00 ffff88012a310000 cgroup cgroup /sys/fs/cgroup/memory ffff88012a265dc0 ffff88012a310800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a318000 ffff88012a311000 cgroup cgroup /sys/fs/cgroup/pids ffff88012a3181c0 ffff88012a311800 cgroup cgroup /sys/fs/cgroup/blkio ffff8800b5392000 ffff8800b5257000 configfs configfs /sys/kernel/config ffff88012a318380 ffff88012a316000 ext4 /dev/nbd0 / ffff88012a318540 ffff8800b53c2800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8800b53921c0 ffff8800b46f4800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8800b5392380 ffff8800b46f3000 hugetlbfs hugetlbfs /dev/hugepages ffff88012a318c40 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b5392540 ffff88012b259000 mqueue mqueue /dev/mqueue ffff88012a318e00 ffff8800b520a800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff880137652540 ffff880129dec000 ramfs none /mnt ffff88012a318fc0 ffff8800b5255000 squashfs /dev/vda /home/green/git/lustre-release ffff8800b5392700 ffff880129def000 tmpfs none /var/lib/stateless/writable ffff880137652700 ffff880129def000 tmpfs none /var/cache/man ffff8801376528c0 ffff880129def000 tmpfs none /var/log ffff88012a319180 ffff880129def000 tmpfs none /var/lib/dbus ffff8800b53928c0 ffff880129def000 tmpfs none /tmp ffff88012a319340 ffff880129def000 tmpfs none /var/lib/dhclient ffff8800b5392a80 ffff880129def000 tmpfs none /var/tmp ffff88012a319500 ffff880129def000 tmpfs none /var/lib/NetworkManager ffff8800b5392c40 ffff880129def000 tmpfs none /var/lib/systemd/random-seed ffff88012a3196c0 ffff880129def000 tmpfs none /var/spool ffff8800b5392e00 ffff880129def000 tmpfs none /var/lib/nfs ffff8800b5392fc0 ffff880129def000 tmpfs none /var/lib/gssproxy ffff8800b5393180 ffff880129def000 tmpfs none /var/lib/logrotate ffff8800b5393340 ffff880129def000 tmpfs none /etc ffff8800b5393500 ffff880129def000 tmpfs none /var/lib/rsyslog ffff8800b53936c0 ffff880129def000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff880137652a80 ffff8800b40a2800 nfs4 192.168.200.253:/exports/state/oleg134-server.virtnet /var/lib/stateless/state ffff88012a319880 ffff8800b40a2800 nfs4 192.168.200.253:/exports/state/oleg134-server.virtnet /boot ffff8800b5393c00 ffff8800b40a2800 nfs4 192.168.200.253:/exports/state/oleg134-server.virtnet /etc/etc/kdump.conf ffff880137652e00 ffff8800b53c2800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff880137653c00 ffff8800b5255000 squashfs /dev/vda /usr/sbin/mount.lustre ffff88012a319a40 ffff88009016c000 lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 ffff88012a318700 ffff88012fc4e000 lustre lustre-ost1/ost1 /mnt/lustre-ost1 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 629.782158] LustreError: Skipped 122 previous similar messages [ 629.792351] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.134@tcp (at 0@lo) [ 629.804453] Lustre: Skipped 122 previous similar messages [ 637.156144] LustreError: 32632:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 637.164246] LustreError: 32632:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 673.882489] LustreError: 32632:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 673.889189] LustreError: 32632:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 680.502375] hrtimer: interrupt took 5005040 ns [ 758.017629] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 758.021969] Lustre: Skipped 265 previous similar messages [ 758.034426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.044303] Lustre: Skipped 265 previous similar messages [ 758.047396] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 758.051887] Lustre: Skipped 265 previous similar messages [ 758.057760] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 758.067926] LustreError: Skipped 265 previous similar messages [ 758.073125] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.134@tcp (at 0@lo) [ 758.081765] Lustre: Skipped 265 previous similar messages [ 882.991948] LustreError: 32632:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 882.997746] LustreError: 32632:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 923.640490] LustreError: 32632:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 923.653481] LustreError: 32632:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 983.218681] LustreError: 32632:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 983.227447] LustreError: 32632:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 1014.063441] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1014.067456] Lustre: Skipped 454 previous similar messages [ 1014.085658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.095222] Lustre: Skipped 454 previous similar messages [ 1014.109221] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1014.113032] Lustre: Skipped 454 previous similar messages [ 1014.119775] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1014.130857] LustreError: Skipped 454 previous similar messages [ 1014.136925] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.134@tcp (at 0@lo) [ 1014.141022] Lustre: Skipped 454 previous similar messages [ 1083.763072] Lustre: 32601:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880130ae2d80 x1792841323997056/t8589959361(0) o35->fc3dcc87-29f8-440c-a962-abd79f51e091@192.168.201.34@tcp:46/0 lens 392/456 e 0 to 0 dl 1709787391 ref 1 fl Interpret:/202/0 rc 0/0 job:'' uid:0 gid:0 [ 1254.223501] Lustre: 2226:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008a185880 x1792841325640256/t8589972179(0) o35->fc3dcc87-29f8-440c-a962-abd79f51e091@192.168.201.34@tcp:216/0 lens 392/456 e 0 to 0 dl 1709787561 ref 1 fl Interpret:/202/0 rc 0/0 job:'' uid:0 gid:0 [ 1429.207840] Lustre: 2226:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880133e57480 x1792841327065600/t8589983295(0) o35->fc3dcc87-29f8-440c-a962-abd79f51e091@192.168.201.34@tcp:391/0 lens 392/456 e 0 to 0 dl 1709787736 ref 1 fl Interpret:/202/0 rc 0/0 job:'' uid:0 gid:0 [ 1554.534926] Lustre: lustre-MDT0000: Client fc3dcc87-29f8-440c-a962-abd79f51e091 (at 192.168.201.34@tcp) reconnecting [ 1554.538890] Lustre: Skipped 32 previous similar messages [ 1565.526988] Lustre: 32600:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880087962a00 x1792841328009088/t8589990650(0) o101->fc3dcc87-29f8-440c-a962-abd79f51e091@192.168.201.34@tcp:527/0 lens 664/3488 e 0 to 0 dl 1709787872 ref 1 fl Interpret:H/202/0 rc 0/0 job:'' uid:0 gid:0 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 12.73s (real) 6.83s (CPU), Child processes: 5.82s