************************ crashinfo ************************* /exports/testreports/40573/testresults/conf-sanity-special3-zfs-centos7_x86_64-centos7_x86_64-retry2/oleg154-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.ddCvU/vmlinux [TAINTED] DUMPFILE: /exports/testreports/40573/testresults/conf-sanity-special3-zfs-centos7_x86_64-centos7_x86_64-retry2/oleg154-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu Mar 7 00:47:17 EST 2024 UPTIME: 01:08:50 LOAD AVERAGE: 0.41, 0.23, 0.23 TASKS: 323 NODENAME: oleg154-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2400 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 44 last 5s 113 last 60s 126 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 319 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 25281 kworker/2:0 0 ms (no user stack) 873 in:imjournal 0 ms /usr/sbin/rsyslogd -n 24362 kworker/1:0 12 ms (no user stack) 11 rcuos/0 12 ms (no user stack) 2796 zthr_procedure 67 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 697029 2.7 GB 72% of TOTAL MEM USED 258038 1008 MB 27% of TOTAL MEM SHARED 8310 32.5 MB 0% of TOTAL MEM BUFFERS 5106 19.9 MB 0% of TOTAL MEM CACHED 66636 260.3 MB 6% of TOTAL MEM SLAB 62793 245.3 MB 6% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 60035 234.5 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 5 LISTEN 3 NAGLE disabled (TCP_NODELAY): 4 user_data set (NFS etc.): 4 Unusual Situations: Doing Retransmission: 1 (run xportshow --retrans for details) UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 4127.8 eth0 n/a 0.2 RSS_TOTAL=52340 pages, %mem= 0.9 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012a2a21c0 ffff88012a2b0800 sysfs sysfs /sys ffff88012a2a2380 ffff880139944000 proc proc /proc ffff88012a2a2540 ffff880137668000 devtmpfs devtmpfs /dev ffff88012a2a2700 ffff88012a2b0000 securityfs securityfs /sys/kernel/security ffff88012a2a28c0 ffff88012a2b1000 tmpfs tmpfs /dev/shm ffff88012a2a2a80 ffff88012b248000 devpts devpts /dev/pts ffff88012a2a2c40 ffff88012a2b1800 tmpfs tmpfs /run ffff88012a2a2e00 ffff88012a2b2000 tmpfs tmpfs /sys/fs/cgroup ffff88012a2a2fc0 ffff88012a2b2800 cgroup cgroup /sys/fs/cgroup/systemd ffff88012a2a3180 ffff88012a2b3000 pstore pstore /sys/fs/pstore ffff88012a2a3340 ffff88012a2b5000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a2a3500 ffff88012a2b4800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a2a36c0 ffff88012a2b4000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012a2a3880 ffff88012a2b3800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a2a3a40 ffff88012a2b5800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a2a3c00 ffff88012a2b6000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a2a3dc0 ffff88012a2b6800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a27c000 ffff88012a2b7000 cgroup cgroup /sys/fs/cgroup/pids ffff88012a27c1c0 ffff88012a2b7800 cgroup cgroup /sys/fs/cgroup/devices ffff88012a27c380 ffff88012a300000 cgroup cgroup /sys/fs/cgroup/memory ffff8800b490a000 ffff8800b480f800 configfs configfs /sys/kernel/config ffff880137652540 ffff880129d04000 ext4 /dev/nbd0 / ffff88012a27c540 ffff8800b5220800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012a27c700 ffff880129d22000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880138ccb880 ffff88012b249000 mqueue mqueue /dev/mqueue ffff88012a27c8c0 ffff880129d27800 hugetlbfs hugetlbfs /dev/hugepages ffff880137652c40 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b490a1c0 ffff8800b5222000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012a27ca80 ffff880136811800 ramfs none /mnt ffff880138ccba40 ffff8800b42ae800 squashfs /dev/vda /home/green/git/lustre-release ffff88012a27cc40 ffff8800b5221800 tmpfs none /var/lib/stateless/writable ffff880137652e00 ffff8800b5221800 tmpfs none /var/cache/man ffff88012a27ce00 ffff8800b5221800 tmpfs none /var/log ffff880137652fc0 ffff8800b5221800 tmpfs none /var/lib/dbus ffff880137653180 ffff8800b5221800 tmpfs none /tmp ffff880138ccbc00 ffff8800b5221800 tmpfs none /var/lib/dhclient ffff88012a27cfc0 ffff8800b5221800 tmpfs none /var/tmp ffff88012a27d180 ffff8800b5221800 tmpfs none /var/lib/NetworkManager ffff88012a27d340 ffff8800b5221800 tmpfs none /var/lib/systemd/random-seed ffff88012a27d500 ffff8800b5221800 tmpfs none /var/spool ffff880137653340 ffff8800b5221800 tmpfs none /var/lib/nfs ffff88012a27d6c0 ffff8800b5221800 tmpfs none /var/lib/gssproxy ffff8800b490a540 ffff8800b5221800 tmpfs none /var/lib/logrotate ffff88012a27d880 ffff8800b5221800 tmpfs none /etc ffff88012a27da40 ffff8800b5221800 tmpfs none /var/lib/rsyslog ffff880137653500 ffff8800b5221800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012a27dc00 ffff880129d26000 nfs4 192.168.200.253:/exports/state/oleg154-server.virtnet /var/lib/stateless/state ffff8800b270c1c0 ffff880129d26000 nfs4 192.168.200.253:/exports/state/oleg154-server.virtnet /boot ffff8801376536c0 ffff880129d26000 nfs4 192.168.200.253:/exports/state/oleg154-server.virtnet /etc/etc/kdump.conf ffff880137653880 ffff8800b5220800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b270d180 ffff8800b42ae800 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b270c000 ffff88009c2ad800 lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 ffff8800b1a68700 ffff88009ecd0000 lustre lustre-ost1/ost1 /mnt/lustre-ost1 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 670.423791] Lustre: Skipped 127 previous similar messages [ 670.826745] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 670.830342] Lustre: Skipped 127 previous similar messages [ 670.838602] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 670.841339] Lustre: Skipped 127 previous similar messages [ 671.569561] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 671.574516] LustreError: Skipped 128 previous similar messages [ 671.577540] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.154@tcp (at 0@lo) [ 671.582024] Lustre: Skipped 128 previous similar messages [ 673.521340] LustreError: 331:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 673.532945] LustreError: 331:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 798.714371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 798.721611] Lustre: Skipped 261 previous similar messages [ 799.183445] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 799.203275] Lustre: Skipped 261 previous similar messages [ 799.221235] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 799.223847] Lustre: Skipped 261 previous similar messages [ 799.768972] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 799.772592] LustreError: Skipped 260 previous similar messages [ 799.776815] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.154@tcp (at 0@lo) [ 799.780575] Lustre: Skipped 260 previous similar messages [ 1014.671398] LustreError: 331:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 1014.677042] LustreError: 331:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 1055.081254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.085229] Lustre: Skipped 446 previous similar messages [ 1055.742503] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1055.746002] Lustre: Skipped 446 previous similar messages [ 1055.769762] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1055.772872] Lustre: Skipped 446 previous similar messages [ 1055.783193] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1055.788942] LustreError: Skipped 445 previous similar messages [ 1055.795095] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.154@tcp (at 0@lo) [ 1055.798284] Lustre: Skipped 445 previous similar messages [ 1118.113298] Lustre: 301:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a1c4c000 x1792841369873792/t8589957305(0) o35->d51c8d17-e831-4052-97ca-be7315139e26@192.168.201.54@tcp:90/0 lens 392/456 e 0 to 0 dl 1709787435 ref 1 fl Interpret:/202/0 rc 0/0 job:'' uid:0 gid:0 [ 1150.882965] Lustre: 20169:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880135871180 x1792841370136768/t8589959354(0) o101->d51c8d17-e831-4052-97ca-be7315139e26@192.168.201.54@tcp:123/0 lens 664/3488 e 0 to 0 dl 1709787468 ref 1 fl Interpret:H/202/0 rc 0/0 job:'' uid:0 gid:0 [ 1218.828970] Lustre: 302:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009e344050 x1792841370779840/t8589964369(0) o35->d51c8d17-e831-4052-97ca-be7315139e26@192.168.201.54@tcp:191/0 lens 392/456 e 0 to 0 dl 1709787536 ref 1 fl Interpret:/202/0 rc 0/0 job:'' uid:0 gid:0 [ 1592.367226] Lustre: lustre-MDT0000: Client d51c8d17-e831-4052-97ca-be7315139e26 (at 192.168.201.54@tcp) reconnecting [ 1592.372248] Lustre: Skipped 45 previous similar messages [ 1603.363114] Lustre: 301:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880133cdb100 x1792841374037248/t8589989777(0) o35->d51c8d17-e831-4052-97ca-be7315139e26@192.168.201.54@tcp:575/0 lens 392/456 e 0 to 0 dl 1709787920 ref 1 fl Interpret:/202/0 rc 0/0 job:'' uid:0 gid:0 [ 1733.078223] Lustre: 301:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880134de5450 x1792841374907520/t8589996561(0) o35->d51c8d17-e831-4052-97ca-be7315139e26@192.168.201.54@tcp:705/0 lens 392/456 e 0 to 0 dl 1709788050 ref 1 fl Interpret:/202/0 rc 0/0 job:'' uid:0 gid:0 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 15.07s (real) 7.65s (CPU), Child processes: 6.79s