************************ crashinfo ************************* /exports/testreports/42627/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg303-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.agzu7/vmlinux [TAINTED] DUMPFILE: /exports/testreports/42627/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg303-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Mon May 13 04:15:39 EDT 2024 UPTIME: 00:17:06 LOAD AVERAGE: 0.00, 0.25, 0.49 TASKS: 336 NODENAME: oleg303-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 18 last 5s 60 last 60s 84 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 332 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 920 in:imjournal 0 ms /usr/sbin/rsyslogd -n 3687 monitor_thread 0 ms (no user stack) 11 rcuos/0 0 ms (no user stack) 3682 socknal_reaper 15 ms (no user stack) 51 kworker/2:1 97 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 665874 2.5 GB 69% of TOTAL MEM USED 289193 1.1 GB 30% of TOTAL MEM SHARED 35174 137.4 MB 3% of TOTAL MEM BUFFERS 32884 128.5 MB 3% of TOTAL MEM CACHED 100291 391.8 MB 10% of TOTAL MEM SLAB 24418 95.4 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 58972 230.4 MB 7% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 9 LISTEN 3 NAGLE disabled (TCP_NODELAY): 7 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 1023.0 eth0 n/a 2.8 RSS_TOTAL=53732 pages, %mem= 0.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aa72000 ffff88012b33d800 sysfs sysfs /sys ffff88012aa721c0 ffff880139944000 proc proc /proc ffff88012aa72380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aa72540 ffff88013702f000 securityfs securityfs /sys/kernel/security ffff880138ccaa80 ffff88012aa7d800 tmpfs tmpfs /dev/shm ffff880138ccac40 ffff88013702b800 devpts devpts /dev/pts ffff880138ccae00 ffff88012aa7e000 tmpfs tmpfs /run ffff880138ccafc0 ffff88012aa7e800 tmpfs tmpfs /sys/fs/cgroup ffff880138ccb180 ffff88012aa7f000 cgroup cgroup /sys/fs/cgroup/systemd ffff880138ccb340 ffff88012aa7f800 pstore pstore /sys/fs/pstore ffff880138ccb500 ffff88012a329800 cgroup cgroup /sys/fs/cgroup/blkio ffff880138ccb6c0 ffff88012a329000 cgroup cgroup /sys/fs/cgroup/pids ffff880138ccb880 ffff88012a328800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff880138ccba40 ffff88012a328000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff880138ccbc00 ffff88012a32a000 cgroup cgroup /sys/fs/cgroup/cpuset ffff880138ccbdc0 ffff88012a32a800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012a312000 ffff88012a32b000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a3121c0 ffff88012a32b800 cgroup cgroup /sys/fs/cgroup/devices ffff88012a312380 ffff88012a32c000 cgroup cgroup /sys/fs/cgroup/memory ffff88012aa72700 ffff88012b33e000 cgroup cgroup /sys/fs/cgroup/freezer ffff88013700ee00 ffff88012a33d000 configfs configfs /sys/kernel/config ffff880129ea4000 ffff88012a33f000 ext4 /dev/nbd0 / ffff880129ea41c0 ffff8800b5214800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012aa72e00 ffff88012a32d800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880129ea4380 ffff88013702d000 mqueue mqueue /dev/mqueue ffff88012a312a80 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8801376688c0 ffff8800b41e6800 hugetlbfs hugetlbfs /dev/hugepages ffff88012aa72fc0 ffff88012a32f800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff880137668a80 ffff8800b4078800 ramfs none /mnt ffff880137668c40 ffff8800b419f800 squashfs /dev/vda /home/green/git/lustre-release ffff88012aa73180 ffff8800b4078000 tmpfs none /var/lib/stateless/writable ffff880137668e00 ffff8800b4078000 tmpfs none /var/cache/man ffff880137668fc0 ffff8800b4078000 tmpfs none /var/log ffff88012aa73340 ffff8800b4078000 tmpfs none /var/lib/dbus ffff88012a312e00 ffff8800b4078000 tmpfs none /tmp ffff880137669180 ffff8800b4078000 tmpfs none /var/lib/dhclient ffff880129ea48c0 ffff8800b4078000 tmpfs none /var/tmp ffff88012aa73500 ffff8800b4078000 tmpfs none /var/lib/NetworkManager ffff88012aa736c0 ffff8800b4078000 tmpfs none /var/lib/systemd/random-seed ffff88012aa73880 ffff8800b4078000 tmpfs none /var/spool ffff88012aa73a40 ffff8800b4078000 tmpfs none /var/lib/nfs ffff880137669340 ffff8800b4078000 tmpfs none /var/lib/gssproxy ffff88012a312fc0 ffff8800b4078000 tmpfs none /var/lib/logrotate ffff880129ea4a80 ffff8800b4078000 tmpfs none /etc ffff880137669500 ffff8800b4078000 tmpfs none /var/lib/rsyslog ffff880129ea4c40 ffff8800b4078000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012aa73c00 ffff8800b419b800 nfs4 192.168.200.253:/exports/state/oleg303-server.virtnet /var/lib/stateless/state ffff880137669880 ffff8800b419b800 nfs4 192.168.200.253:/exports/state/oleg303-server.virtnet /boot ffff880137669a40 ffff8800b419b800 nfs4 192.168.200.253:/exports/state/oleg303-server.virtnet /etc/etc/kdump.conf ffff880129ea4e00 ffff8800b5214800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b28ba1c0 ffff8800b419f800 squashfs /dev/vda /usr/sbin/mount.lustre ffff88012a313c00 ffff8800b4b00800 lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 ffff88013700f500 ffff880129d6e000 lustre /dev/mapper/mds2_flakey /mnt/lustre-mds2 ffff88013700ea80 ffff8800b4b07000 lustre /dev/mapper/ost1_flakey /mnt/lustre-ost1 ffff88013700fc00 ffff8801308bf800 lustre /dev/mapper/ost2_flakey /mnt/lustre-ost2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 289.925231] Call Trace: [ 289.926654] [] dump_stack+0x19/0x1b [ 289.929951] [] osd_trans_start+0x67e/0x6f0 [osd_ldiskfs] [ 289.937756] [] ? __llog_ctxt_put+0xdb/0x140 [obdclass] [ 289.942282] [] top_trans_start+0x351/0xab0 [ptlrpc] [ 289.947774] [] lod_trans_start+0x7f/0x2e0 [lod] [ 289.950947] [] mdd_trans_start+0x14/0x20 [mdd] [ 289.953659] [] mdd_migrate_object+0x144f/0x1e10 [mdd] [ 289.955992] [] ? __wake_up_common_lock+0x91/0xc0 [ 289.959853] [] mdd_migrate+0x28/0x30 [mdd] [ 289.962845] [] mdt_reint_migrate+0x1b8d/0x2420 [mdt] [ 289.966595] [] mdt_reint_rec+0x87/0x240 [mdt] [ 289.969804] [] mdt_reint_internal+0x74c/0xbc0 [mdt] [ 289.973075] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 289.976450] [] mdt_reint+0x67/0x150 [mdt] [ 289.980432] [] tgt_request_handle+0x74e/0x1a50 [ptlrpc] [ 289.983312] [] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc] [ 289.985907] [] ptlrpc_main+0xc76/0x1690 [ptlrpc] [ 289.989752] [] ? put_prev_entity+0x31/0x400 [ 289.992589] [] ? do_raw_spin_unlock+0x49/0x90 [ 289.998194] [] ? ptlrpc_wait_event+0x610/0x610 [ptlrpc] [ 290.001862] [] kthread+0xe4/0xf0 [ 290.004394] [] ? kthread_create_on_node+0x140/0x140 [ 290.008203] [] ret_from_fork_nospec_begin+0x7/0x21 [ 290.012167] [] ? kthread_create_on_node+0x140/0x140 [ 292.103555] LustreError: 7149:0:(mdt_reint.c:2519:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000402:0x2:0x0]/10 failed: rc = -114 [ 292.111304] LustreError: 7149:0:(mdt_reint.c:2519:mdt_reint_migrate()) Skipped 348 previous similar messages [ 292.480899] LustreError: 7149:0:(mdd_dir.c:4472:mdd_migrate_cmd_check()) lustre-MDD0001: '4' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 4' to finish migration: rc = -1 [ 292.491559] LustreError: 7149:0:(mdd_dir.c:4472:mdd_migrate_cmd_check()) Skipped 30 previous similar messages [ 298.073038] Lustre: 15534:0:(lod_lov.c:1437:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x2cf8:0x0] with magic=0xbd60bd0 [ 298.081881] Lustre: 15534:0:(lod_lov.c:1437:lod_parse_striping()) Skipped 89 previous similar messages [ 299.141420] Lustre: 7150:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 3189 < left 3269, rollback = 2 [ 299.147778] Lustre: 7150:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 83 previous similar messages [ 326.149202] LustreError: 15508:0:(mdd_object.c:3873:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000402:0x2c6d:0x0]: rc = -2 [ 326.156982] LustreError: 15508:0:(mdd_object.c:3873:mdd_close()) Skipped 4 previous similar messages [ 395.325897] Lustre: 15492:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880095b06300 x1798923539487040/t4295034066(0) o101->2232cc14-bda9-4187-914f-58c1dc5d1b7c@192.168.203.3@tcp:398/0 lens 1384/4224 e 0 to 0 dl 1715587653 ref 1 fl Interpret:/202/0 rc 0/0 job:'ls.0' uid:0 gid:0 [ 472.633669] LustreError: 7127:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 146s: evicting client at 192.168.203.3@tcp ns: mdt-lustre-MDT0000_UUID lock: ffff880099a7db00/0x240e0e0219272758 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x3b9a:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.3@tcp remote: 0x88aba071892812bc expref: 1662 pid: 15607 timeout: 471 lvb_type: 0 [ 472.679521] LustreError: 15513:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.3@tcp arrived at 1715587585 with bad export cookie 2598029437207776063 [ 472.709933] LustreError: 15507:0:(mdt_reint.c:2519:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x3375:0x0]/17 failed: rc = -16 [ 472.714149] LustreError: 15507:0:(mdt_reint.c:2519:mdt_reint_migrate()) Skipped 221 previous similar messages ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 11.46s (real) 6.28s (CPU), Child processes: 5.13s