************************ crashinfo ************************* /exports/testreports/50545/testresults/conf-sanity3-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg232-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.o5pFp/vmlinux [TAINTED] DUMPFILE: /exports/testreports/50545/testresults/conf-sanity3-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg232-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Mon Apr 7 08:25:16 EDT 2025 UPTIME: 02:30:18 LOAD AVERAGE: 0.19, 0.48, 0.56 TASKS: 163 NODENAME: oleg232-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=24839 CPU=1 CMD=insmod #0 crc32c_pcl_intel_update+0x7b #1 crypto_shash_update+0x47 #2 shash_ahash_update+0x3e #3 shash_async_update+0x12 #4 cfs_crypto_hash_update_page+0x85 #5 cfs_crypto_performance_test+0x12f #6 cfs_crypto_register+0x4d #7 init_module+0x11f #8 do_one_initcall+0xb8 #9 load_module+0x1682 #10 sys_finit_module+0xa6 #11 system_call_fastpath+0x1f, 477 bytes of data -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 54 last 5s 62 last 60s 74 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 158 TASK_RUNNING 2 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 827 gmain 0 ms /usr/sbin/NetworkManager --no-daemon 9 rcu_sched 0 ms (no user stack) 4509 arc_reap 0 ms (no user stack) 4511 dbuf_evict 124 ms (no user stack) 5536 kworker/2:1 169 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 817591 3.1 GB 85% of TOTAL MEM USED 137476 537 MB 14% of TOTAL MEM SHARED 10725 41.9 MB 1% of TOTAL MEM BUFFERS 6688 26.1 MB 0% of TOTAL MEM CACHED 75599 295.3 MB 7% of TOTAL MEM SLAB 15490 60.5 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 68058 265.9 MB 9% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=insmod ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 3 LISTEN 2 NAGLE disabled (TCP_NODELAY): 3 user_data set (NFS etc.): 1 Unusual Situations: Doing Retransmission: 1 (run xportshow --retrans for details) UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 9015.8 eth0 n/a 0.2 RSS_TOTAL=93884 pages, %mem= 1.1 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88013701c540 ffff8800b5230000 sysfs sysfs /sys ffff88013701c700 ffff880139944000 proc proc /proc ffff88013701c8c0 ffff880137678000 devtmpfs devtmpfs /dev ffff88013701ca80 ffff8800b51c1000 securityfs securityfs /sys/kernel/security ffff88013701cc40 ffff8800b5230800 tmpfs tmpfs /dev/shm ffff88013701ce00 ffff88013771a000 devpts devpts /dev/pts ffff88013701cfc0 ffff8800b5231000 tmpfs tmpfs /run ffff88013701d180 ffff8800b5231800 tmpfs tmpfs /sys/fs/cgroup ffff88013701d340 ffff8800b5232000 cgroup cgroup /sys/fs/cgroup/systemd ffff88013701d500 ffff8800b5232800 pstore pstore /sys/fs/pstore ffff88013701d6c0 ffff8800b5234800 cgroup cgroup /sys/fs/cgroup/pids ffff88013701d880 ffff8800b5234000 cgroup cgroup /sys/fs/cgroup/memory ffff88013701da40 ffff8800b5233800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88013701dc00 ffff8800b5233000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88013701ddc0 ffff8800b5235000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a35c000 ffff8800b5235800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a35c1c0 ffff8800b5236000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a35c380 ffff8800b5236800 cgroup cgroup /sys/fs/cgroup/devices ffff88012a35c540 ffff8800b5237000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012a35c700 ffff8800b5237800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a35ca80 ffff8800b523d800 configfs configfs /sys/kernel/config ffff8800b4b20000 ffff8800b4010000 ext4 /dev/nbd0 / ffff88012a35cfc0 ffff8800b523d000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8801376688c0 ffff8800b4ae9800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff88012a35c8c0 ffff88013771b800 mqueue mqueue /dev/mqueue ffff8800b4b201c0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b4b20380 ffff88012b35e800 hugetlbfs hugetlbfs /dev/hugepages ffff88012a35d180 ffff8800b51c2000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b4b20700 ffff88012b35e000 ramfs none /mnt ffff8800b4b208c0 ffff8800b4220800 tmpfs none /var/lib/stateless/writable ffff880137668a80 ffff8800b51c5800 squashfs /dev/vda /home/green/git/lustre-release ffff880137668c40 ffff8800b4220800 tmpfs none /var/cache/man ffff88012a35d340 ffff8800b4220800 tmpfs none /var/log ffff88012a35d500 ffff8800b4220800 tmpfs none /var/lib/dbus ffff8800b4962a80 ffff8800b4220800 tmpfs none /tmp ffff880137668e00 ffff8800b4220800 tmpfs none /var/lib/dhclient ffff880137668fc0 ffff8800b4220800 tmpfs none /var/tmp ffff880137669180 ffff8800b4220800 tmpfs none /var/lib/NetworkManager ffff88012a35d6c0 ffff8800b4220800 tmpfs none /var/lib/systemd/random-seed ffff88012a35d880 ffff8800b4220800 tmpfs none /var/spool ffff8800b4962c40 ffff8800b4220800 tmpfs none /var/lib/nfs ffff88012a35da40 ffff8800b4220800 tmpfs none /var/lib/gssproxy ffff880137669340 ffff8800b4220800 tmpfs none /var/lib/logrotate ffff8800b4962e00 ffff8800b4220800 tmpfs none /etc ffff88012a35dc00 ffff8800b4220800 tmpfs none /var/lib/rsyslog ffff880137669500 ffff8800b4220800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012a35ddc0 ffff8800b411b800 nfs4 192.168.200.253:/exports/state/oleg232-server.virtnet /var/lib/stateless/state ffff8800b53f2380 ffff8800b411b800 nfs4 192.168.200.253:/exports/state/oleg232-server.virtnet /boot ffff8800b53f2540 ffff8800b411b800 nfs4 192.168.200.253:/exports/state/oleg232-server.virtnet /etc/etc/kdump.conf ffff8801376696c0 ffff8800b523d000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff880129d39500 ffff8800b51c5800 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b4963180 ffff8800b18bb800 tmpfs tmpfs /run/user/0 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 8815.466838] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8817.616912] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 8817.620119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8818.682008] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8818.692081] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8819.301439] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8825.771947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8826.620366] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 8826.621846] Lustre: Skipped 2 previous similar messages [ 8832.705549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8832.708406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8834.907773] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 08:22:11 (1744028531) [ 8835.703357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.703793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8835.703795] Lustre: Skipped 3 previous similar messages [ 8835.708932] Lustre: Skipped 5 previous similar messages [ 8841.614389] Lustre: server umount lustre-MDT0000 complete [ 8842.557090] LustreError: 1318:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744028540 with bad export cookie 10778488833933417911 [ 8842.558222] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8842.563356] LustreError: 1318:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8842.685331] Lustre: server umount lustre-MDT0001 complete [ 8853.742488] Lustre: server umount lustre-OST0000 complete [ 8864.698209] Lustre: server umount lustre-OST0001 complete [ 8877.278049] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 08:22:53 (1744028573) [ 8888.410572] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8889.129762] Key type lgssc unregistered [ 8889.234994] LNet: 9719:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8890.241808] LNet: Removed LNI 192.168.202.132@tcp [ 8912.540359] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8920.666678] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8939.552325] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8946.732448] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8955.625085] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8977.668002] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8986.411855] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8995.530491] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9013.056239] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9017.827452] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9017.938659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9017.975683] alg: No test for adler32 (adler32-zlib) ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 6.26s (real) 3.41s (CPU), Child processes: 2.82s