************************ crashinfo ************************* /exports/testreports/45673/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg444-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.4dLeN/vmlinux [TAINTED] DUMPFILE: /exports/testreports/45673/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg444-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Sat Sep 14 05:05:26 EDT 2024 UPTIME: 00:17:16 LOAD AVERAGE: 0.00, 0.35, 0.67 TASKS: 322 NODENAME: oleg444-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 28 last 5s 81 last 60s 103 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 318 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 602 kworker/2:2 0 ms (no user stack) 8381 osp-pre-0-1 0 ms (no user stack) 4528 l2arc_feed 0 ms (no user stack) 15829 mdt_out00_006 0 ms (no user stack) 4526 dbuf_evict 0 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 664537 2.5 GB 69% of TOTAL MEM USED 290530 1.1 GB 30% of TOTAL MEM SHARED 40441 158 MB 4% of TOTAL MEM BUFFERS 38099 148.8 MB 3% of TOTAL MEM CACHED 98163 383.4 MB 10% of TOTAL MEM SLAB 25137 98.2 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 59006 230.5 MB 7% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 9 LISTEN 3 NAGLE disabled (TCP_NODELAY): 7 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 1031.7 eth0 n/a 1.5 RSS_TOTAL=52464 pages, %mem= 0.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff880138ccac40 ffff8800b631e800 sysfs sysfs /sys ffff880138ccae00 ffff880139944000 proc proc /proc ffff880138ccafc0 ffff880137678000 devtmpfs devtmpfs /dev ffff880138ccb180 ffff8800b631e000 securityfs securityfs /sys/kernel/security ffff88012b2c6540 ffff8800b51f9800 tmpfs tmpfs /dev/shm ffff88012b2c6700 ffff880137345000 devpts devpts /dev/pts ffff88012b2c68c0 ffff8800b51fa000 tmpfs tmpfs /run ffff88012b2c6a80 ffff8800b51fa800 tmpfs tmpfs /sys/fs/cgroup ffff88012b2c6c40 ffff8800b51fb000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012b2c6e00 ffff8800b51fb800 pstore pstore /sys/fs/pstore ffff88012b2c4380 ffff8800b5268800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012b2c4540 ffff8800b5269000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012b2c4700 ffff8800b5269800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012b2c48c0 ffff8800b526a000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012b2c6fc0 ffff8800b51fd800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012b2c7180 ffff8800b51fd000 cgroup cgroup /sys/fs/cgroup/memory ffff88012b2c7340 ffff8800b51fc800 cgroup cgroup /sys/fs/cgroup/devices ffff88012b2c7500 ffff8800b51fc000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012b2c76c0 ffff8800b51fe000 cgroup cgroup /sys/fs/cgroup/pids ffff88012b2c7880 ffff8800b51fe800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012b2c7a40 ffff8800b533f800 configfs configfs /sys/kernel/config ffff88012b2c5dc0 ffff880129d34800 ext4 /dev/nbd0 / ffff88012b2c7c00 ffff8800b549f000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8800000941c0 ffff8800b4db6800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880138ccb340 ffff8800b549f800 hugetlbfs hugetlbfs /dev/hugepages ffff88012b2c7dc0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b41f2000 ffff880137309000 mqueue mqueue /dev/mqueue ffff880138ccb500 ffff8800b549e000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b41f21c0 ffff8800b5390800 ramfs none /mnt ffff880000094540 ffff8800b4db3000 tmpfs none /var/lib/stateless/writable ffff880000094000 ffff8800b4db2800 squashfs /dev/vda /home/green/git/lustre-release ffff880138ccb6c0 ffff8800b4db3000 tmpfs none /var/cache/man ffff880000094700 ffff8800b4db3000 tmpfs none /var/log ffff8800b41f2380 ffff8800b4db3000 tmpfs none /var/lib/dbus ffff880137669180 ffff8800b4db3000 tmpfs none /tmp ffff880138ccb880 ffff8800b4db3000 tmpfs none /var/lib/dhclient ffff8800000948c0 ffff8800b4db3000 tmpfs none /var/tmp ffff880137669340 ffff8800b4db3000 tmpfs none /var/lib/NetworkManager ffff880137669500 ffff8800b4db3000 tmpfs none /var/lib/systemd/random-seed ffff880000094a80 ffff8800b4db3000 tmpfs none /var/spool ffff8800b41f2540 ffff8800b4db3000 tmpfs none /var/lib/nfs ffff880000094c40 ffff8800b4db3000 tmpfs none /var/lib/gssproxy ffff880000094e00 ffff8800b4db3000 tmpfs none /var/lib/logrotate ffff8801376696c0 ffff8800b4db3000 tmpfs none /etc ffff8800b41f2700 ffff8800b4db3000 tmpfs none /var/lib/rsyslog ffff880000094fc0 ffff8800b4db3000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff880000095180 ffff8800b4e6c800 nfs4 192.168.200.253:/exports/state/oleg444-server.virtnet /var/lib/stateless/state ffff880000095880 ffff8800b4e6c800 nfs4 192.168.200.253:/exports/state/oleg444-server.virtnet /boot ffff8800b41f28c0 ffff8800b4e6c800 nfs4 192.168.200.253:/exports/state/oleg444-server.virtnet /etc/etc/kdump.conf ffff880137669880 ffff8800b549f000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff880136830380 ffff8800b4db2800 squashfs /dev/vda /usr/sbin/mount.lustre ffff88012e2e3a40 ffff8800a8e4c000 lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 ffff88012e2e3500 ffff880072a4a000 lustre /dev/mapper/mds2_flakey /mnt/lustre-mds2 ffff88012e2e2fc0 ffff8800b4db7000 lustre /dev/mapper/ost1_flakey /mnt/lustre-ost1 ffff88012e2e2540 ffff88009e28b000 lustre /dev/mapper/ost2_flakey /mnt/lustre-ost2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 400.888269] [] kthread+0xe4/0xf0 [ 400.888272] [] ? kthread_create_on_node+0x140/0x140 [ 400.888275] [] ret_from_fork_nospec_begin+0x7/0x21 [ 400.888277] [] ? kthread_create_on_node+0x140/0x140 [ 400.888548] Lustre: 15712:0:(osd_internal.h:1314:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 3189 < left 3475, rollback = 2 [ 400.888550] Lustre: 15712:0:(osd_internal.h:1314:osd_trans_exec_op()) Skipped 101 previous similar messages [ 400.911488] Lustre: mdt_io00_004: service thread pid 15712 completed after 99.943s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 400.940583] Lustre: mdt_io00_002: service thread pid 7242 completed after 99.965s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 400.941081] Lustre: mdt_io00_003: service thread pid 15710 completed after 99.965s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 400.972907] Lustre: mdt_io00_012: service thread pid 15762 completed after 99.963s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 401.040112] Lustre: mdt_io00_010: service thread pid 15760 completed after 99.883s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 401.058295] Lustre: mdt_io00_001: service thread pid 7241 completed after 99.831s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 401.067320] Lustre: mdt_io00_011: service thread pid 15761 completed after 99.758s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 401.086313] Lustre: mdt_io00_013: service thread pid 15901 completed after 99.632s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 401.099265] Lustre: mdt_io00_014: service thread pid 15902 completed after 99.544s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 401.123913] Lustre: mdt_io00_015: service thread pid 15903 completed after 99.359s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 401.186835] Lustre: mdt_io00_016: service thread pid 15904 completed after 99.374s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 401.229553] Lustre: mdt_io00_017: service thread pid 15905 completed after 99.389s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 401.273116] Lustre: mdt_io00_018: service thread pid 15906 completed after 98.804s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 402.560268] Lustre: 15696:0:(lod_lov.c:1438:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x3160:0x0] with magic=0xbd60bd0 [ 402.569074] Lustre: 15696:0:(lod_lov.c:1438:lod_parse_striping()) Skipped 265 previous similar messages [ 404.024138] LustreError: 15709:0:(mdd_object.c:3884:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0x283f:0x0]: rc = -2 [ 404.033827] LustreError: 15709:0:(mdd_object.c:3884:mdd_close()) Skipped 2 previous similar messages [ 404.627137] LustreError: 15906:0:(mdt_reint.c:2533:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000402:0x1:0x0]/15 failed: rc = -16 [ 404.631555] LustreError: 15906:0:(mdt_reint.c:2533:mdt_reint_migrate()) Skipped 197 previous similar messages [ 426.388672] LustreError: 15902:0:(mdd_dir.c:4477:mdd_migrate_cmd_check()) lustre-MDD0001: '7' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 7' to finish migration: rc = -1 [ 426.396532] LustreError: 15902:0:(mdd_dir.c:4477:mdd_migrate_cmd_check()) Skipped 24 previous similar messages [ 445.364537] LustreError: 15779:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0000: [0x200000402:0x4882:0x0] migrate mdt count mismatch 2 != 1 [ 449.214333] Lustre: 15905:0:(osd_internal.h:1314:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 3193 < left 3269, rollback = 2 [ 449.220848] Lustre: 15905:0:(osd_internal.h:1314:osd_trans_exec_op()) Skipped 212 previous similar messages [ 449.225808] Lustre: 15905:0:(osd_handler.c:1955:osd_trans_dump_creds()) create: 3/12/5, destroy: 2/8/0 [ 449.229331] Lustre: 15905:0:(osd_handler.c:1955:osd_trans_dump_creds()) Skipped 213 previous similar messages [ 449.233250] Lustre: 15905:0:(osd_handler.c:1962:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 36/3269/0 [ 449.237907] Lustre: 15905:0:(osd_handler.c:1962:osd_trans_dump_creds()) Skipped 213 previous similar messages [ 449.243183] Lustre: 15905:0:(osd_handler.c:1972:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 7/105/0 [ 449.248125] Lustre: 15905:0:(osd_handler.c:1972:osd_trans_dump_creds()) Skipped 213 previous similar messages [ 449.252606] Lustre: 15905:0:(osd_handler.c:1979:osd_trans_dump_creds()) insert: 13/251/3, delete: 7/13/1 [ 449.258480] Lustre: 15905:0:(osd_handler.c:1979:osd_trans_dump_creds()) Skipped 213 previous similar messages [ 449.262570] Lustre: 15905:0:(osd_handler.c:1986:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 7/7/1 [ 449.268355] Lustre: 15905:0:(osd_handler.c:1986:osd_trans_dump_creds()) Skipped 213 previous similar messages ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 11.20s (real) 6.14s (CPU), Child processes: 5.01s