************************ crashinfo ************************* /exports/testreports/51278/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg342-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.74Sja/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51278/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg342-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 16:10:43 EDT 2025 UPTIME: 00:33:43 LOAD AVERAGE: 0.00, 0.03, 0.38 TASKS: 305 NODENAME: oleg342-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 19 last 5s 64 last 60s 81 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 301 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 4524 dbuf_evict 0 ms (no user stack) 34 rcuos/3 0 ms (no user stack) 9 rcu_sched 0 ms (no user stack) 1 systemd 4 ms /usr/lib/systemd/systemd --switched-root --system --deserialize 21 4522 arc_reap 80 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 694615 2.6 GB 72% of TOTAL MEM USED 260452 1017.4 MB 27% of TOTAL MEM SHARED 25039 97.8 MB 2% of TOTAL MEM BUFFERS 22738 88.8 MB 2% of TOTAL MEM CACHED 87746 342.8 MB 9% of TOTAL MEM SLAB 22697 88.7 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 59013 230.5 MB 7% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 9 LISTEN 3 NAGLE disabled (TCP_NODELAY): 7 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 2020.4 eth0 n/a 2.2 RSS_TOTAL=51436 pages, %mem= 0.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aa1a000 ffff880137318800 sysfs sysfs /sys ffff88012aa1a1c0 ffff880139944000 proc proc /proc ffff88012aa1a380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aa1a540 ffff8800b51f6800 securityfs securityfs /sys/kernel/security ffff88012aa1a700 ffff880137319000 tmpfs tmpfs /dev/shm ffff88012aa1a8c0 ffff88013771f800 devpts devpts /dev/pts ffff88012aa1aa80 ffff880137319800 tmpfs tmpfs /run ffff88012aa1ac40 ffff88013731a000 tmpfs tmpfs /sys/fs/cgroup ffff88012aa1ae00 ffff88013731a800 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aa1afc0 ffff88013731b000 pstore pstore /sys/fs/pstore ffff88012aa1b180 ffff88013731d000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aa1b340 ffff88013731c800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aa1b500 ffff88013731c000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aa1b6c0 ffff88013731b800 cgroup cgroup /sys/fs/cgroup/memory ffff88012aa1b880 ffff88013731d800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aa1ba40 ffff88013731e000 cgroup cgroup /sys/fs/cgroup/devices ffff88012aa1bc00 ffff88013731e800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aa1bdc0 ffff88013731f000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a366000 ffff88013731f800 cgroup cgroup /sys/fs/cgroup/pids ffff88012a3661c0 ffff88012a380000 cgroup cgroup /sys/fs/cgroup/cpuset ffff880137668540 ffff88012a316000 configfs configfs /sys/kernel/config ffff88012a367500 ffff880129f40800 ext4 /dev/nbd0 / ffff88013700e8c0 ffff88012a384800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012a367a40 ffff8800b425a800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880138ccb500 ffff880136818800 hugetlbfs hugetlbfs /dev/hugepages ffff880138ccb6c0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff88012a367c00 ffff88013702f000 mqueue mqueue /dev/mqueue ffff88013700e700 ffff8800b4302000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012a367880 ffff8800b425c000 ramfs none /mnt ffff88013700ea80 ffff880129e92800 tmpfs none /var/lib/stateless/writable ffff8801376688c0 ffff880136819800 squashfs /dev/vda /home/green/git/lustre-release ffff88012a366380 ffff880129e92800 tmpfs none /var/cache/man ffff8800b22de000 ffff880129e92800 tmpfs none /var/log ffff880137668a80 ffff880129e92800 tmpfs none /var/lib/dbus ffff880138ccb880 ffff880129e92800 tmpfs none /tmp ffff880137668c40 ffff880129e92800 tmpfs none /var/lib/dhclient ffff880137668e00 ffff880129e92800 tmpfs none /var/tmp ffff88013700ec40 ffff880129e92800 tmpfs none /var/lib/NetworkManager ffff880137668fc0 ffff880129e92800 tmpfs none /var/lib/systemd/random-seed ffff8800b22de1c0 ffff880129e92800 tmpfs none /var/spool ffff8800b22de380 ffff880129e92800 tmpfs none /var/lib/nfs ffff880137669180 ffff880129e92800 tmpfs none /var/lib/gssproxy ffff880137669340 ffff880129e92800 tmpfs none /var/lib/logrotate ffff880137669500 ffff880129e92800 tmpfs none /etc ffff8801376696c0 ffff880129e92800 tmpfs none /var/lib/rsyslog ffff880137669880 ffff880129e92800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b22de540 ffff880129e97800 nfs4 192.168.200.253:/exports/state/oleg342-server.virtnet /var/lib/stateless/state ffff8800b22de8c0 ffff880129e97800 nfs4 192.168.200.253:/exports/state/oleg342-server.virtnet /boot ffff8800b22dea80 ffff880129e97800 nfs4 192.168.200.253:/exports/state/oleg342-server.virtnet /etc/etc/kdump.conf ffff880137669a40 ffff88012a384800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b1b2c380 ffff880136819800 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b1a5ea80 ffff8800b4303800 lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 ffff8800b1a5e700 ffff8800b425d000 lustre /dev/mapper/mds2_flakey /mnt/lustre-mds2 ffff8800b1a5f880 ffff88007ca49000 lustre /dev/mapper/ost1_flakey /mnt/lustre-ost1 ffff8800b19ec380 ffff88006dd50800 lustre /dev/mapper/ost2_flakey /mnt/lustre-ost2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 499.031555] Lustre: mdt_io00_001: service thread pid 7253 completed after 41.862s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 499.041954] Lustre: mdt_io00_003: service thread pid 16200 completed after 41.841s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 499.058076] Lustre: mdt_io00_008: service thread pid 16299 completed after 41.855s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 499.992405] Lustre: mdt_io00_012: service thread pid 16461 was inactive for 40.040 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 500.001440] Lustre: Skipped 6 previous similar messages [ 523.032471] Lustre: mdt00_000: service thread pid 7238 was inactive for 72.125 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 523.043826] Lustre: Skipped 1 previous similar message [ 544.792504] LustreError: 7229:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 192.168.203.42@tcp ns: mdt-lustre-MDT0001_UUID lock: ffff88012bb37600/0xb3db38ac999b2dcc lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x15c7:0x0].0x0 bits 0x1b/0x0 rrc: 16 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.42@tcp remote: 0x811bc2aacd200750 expref: 934 pid: 16181 timeout: 544 lvb_type: 0 [ 544.815098] LustreError: 16187:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009dfad800 ns: mdt-lustre-MDT0001_UUID lock: ffff88009ac4c600/0xb3db38ac999c1225 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x15c7:0x0].0x0 bits 0x20/0x0 rrc: 14 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.42@tcp remote: 0x811bc2aacd204c3f expref: 412 pid: 16187 timeout: 0 lvb_type: 0 [ 544.815945] Lustre: mdt00_014: service thread pid 16189 completed after 100.158s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.835266] Lustre: mdt00_000: service thread pid 7238 completed after 93.928s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.835504] Lustre: mdt00_017: service thread pid 16302 completed after 96.592s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.835861] Lustre: mdt00_008: service thread pid 16181 completed after 95.157s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.836646] Lustre: mdt00_005: service thread pid 16178 completed after 99.572s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.837601] Lustre: mdt00_004: service thread pid 13367 completed after 93.454s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.838937] Lustre: mdt00_020: service thread pid 16444 completed after 96.023s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.840045] Lustre: mdt00_007: service thread pid 16180 completed after 92.121s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.841805] Lustre: mdt00_016: service thread pid 16215 completed after 93.866s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.843096] Lustre: mdt00_012: service thread pid 16187 completed after 96.599s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 544.844192] Lustre: mdt00_018: service thread pid 16307 completed after 100.096s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 545.323052] Lustre: 7238:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x1b79:0x0] with magic=0xbd60bd0 [ 545.328392] Lustre: 7238:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 37 previous similar messages [ 583.192478] Lustre: mdt_io00_003: service thread pid 16200 was inactive for 84.042 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 583.213428] Lustre: Skipped 3 previous similar messages [ 598.808914] LustreError: 7229:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 192.168.203.42@tcp ns: mdt-lustre-MDT0001_UUID lock: ffff880099a81600/0xb3db38ac999e4e49 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x1771:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.42@tcp remote: 0x811bc2aacd20f5b9 expref: 924 pid: 9598 timeout: 598 lvb_type: 0 [ 598.883676] Lustre: mdt_io00_000: service thread pid 7252 completed after 141.661s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 598.986596] LustreError: 10931:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.42@tcp arrived at 1746128818 with bad export cookie 12960014666647789502 [ 599.001400] Lustre: mdt_io00_009: service thread pid 16361 completed after 141.444s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 599.006605] Lustre: mdt_io00_010: service thread pid 16458 completed after 141.380s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 599.089513] LustreError: 16459:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0000: '8' migration was interrupted, run 'lfs migrate -m 1 -c 2 -H crush 8' to finish migration: rc = -1 [ 599.111805] LustreError: 16459:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) Skipped 7 previous similar messages [ 599.156449] LustreError: 16459:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000402:0x1:0x0]/8 failed: rc = -1 [ 599.160536] LustreError: 16459:0:(mdt_reint.c:2523:mdt_reint_migrate()) Skipped 13 previous similar messages [ 599.173435] Lustre: mdt_io00_011: service thread pid 16459 completed after 141.077s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 599.226679] Lustre: mdt_io00_012: service thread pid 16461 completed after 139.275s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 599.320500] Lustre: mdt_io00_008: service thread pid 16299 completed after 100.247s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 599.325137] Lustre: mdt_io00_007: service thread pid 16221 completed after 100.202s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 599.337735] Lustre: mdt_io00_003: service thread pid 16200 completed after 100.187s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 599.338711] Lustre: mdt_io00_002: service thread pid 7254 completed after 99.637s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). [ 599.357791] Lustre: mdt_io00_001: service thread pid 7253 completed after 99.251s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources). ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 13.16s (real) 7.07s (CPU), Child processes: 6.01s