************************ crashinfo ************************* /exports/testreports/51242/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg120-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.CfTu5/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51242/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg120-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 03:47:56 EDT 2025 UPTIME: 05:25:00 LOAD AVERAGE: 0.42, 0.42, 0.59 TASKS: 293 NODENAME: oleg120-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 17 last 5s 69 last 60s 138 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 289 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 21 watchdog/2 0 ms (no user stack) 13 watchdog/0 0 ms (no user stack) 28 watchdog/3 0 ms (no user stack) 14 watchdog/1 0 ms (no user stack) 9 rcu_sched 102 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 689056 2.6 GB 72% of TOTAL MEM USED 266011 1 GB 27% of TOTAL MEM SHARED 54742 213.8 MB 5% of TOTAL MEM BUFFERS 50413 196.9 MB 5% of TOTAL MEM CACHED 56540 220.9 MB 5% of TOTAL MEM SLAB 26647 104.1 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 92157 360 MB 12% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 9 LISTEN 3 NAGLE disabled (TCP_NODELAY): 7 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 19496.5 eth0 n/a 2.0 RSS_TOTAL=53520 pages, %mem= 0.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012a36a000 ffff880137308800 sysfs sysfs /sys ffff88012a36a1c0 ffff880139944000 proc proc /proc ffff88012a36a380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012a36a540 ffff88012b3d4000 securityfs securityfs /sys/kernel/security ffff88012a36a700 ffff880137309000 tmpfs tmpfs /dev/shm ffff88012a36a8c0 ffff88013771f000 devpts devpts /dev/pts ffff88012a36aa80 ffff880137309800 tmpfs tmpfs /run ffff88012a36ac40 ffff88013730a000 tmpfs tmpfs /sys/fs/cgroup ffff88012a36ae00 ffff88013730a800 cgroup cgroup /sys/fs/cgroup/systemd ffff88012a36afc0 ffff88013730b000 pstore pstore /sys/fs/pstore ffff88012a36b180 ffff88013730d000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a36b340 ffff88013730c800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a36b500 ffff88013730c000 cgroup cgroup /sys/fs/cgroup/pids ffff88012a36b6c0 ffff88013730b800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a36b880 ffff88013730d800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012a36ba40 ffff88013730e000 cgroup cgroup /sys/fs/cgroup/devices ffff88012a36bc00 ffff88013730e800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a36bdc0 ffff88013730f000 cgroup cgroup /sys/fs/cgroup/memory ffff88012a320000 ffff88013730f800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a3201c0 ffff88012a380000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a320540 ffff88012a387000 configfs configfs /sys/kernel/config ffff8801377f6540 ffff8800b5219800 ext4 /dev/nbd0 / ffff880138ccb500 ffff88012a383800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012a321880 ffff8800b4bd5000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8801376688c0 ffff880137678800 mqueue mqueue /dev/mqueue ffff8801377f6c40 ffff8800b41eb000 hugetlbfs hugetlbfs /dev/hugepages ffff88012a321a40 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8801377f6e00 ffff8800b41e9800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012a321c00 ffff8800b4bd4000 ramfs none /mnt ffff88012a321dc0 ffff8800b49b9800 tmpfs none /var/lib/stateless/writable ffff880137668a80 ffff880000073800 squashfs /dev/vda /home/green/git/lustre-release ffff8800b27dc000 ffff8800b49b9800 tmpfs none /var/cache/man ffff880138ccb6c0 ffff8800b49b9800 tmpfs none /var/log ffff880137668c40 ffff8800b49b9800 tmpfs none /var/lib/dbus ffff880137668e00 ffff8800b49b9800 tmpfs none /tmp ffff880137668fc0 ffff8800b49b9800 tmpfs none /var/lib/dhclient ffff8800b27dc1c0 ffff8800b49b9800 tmpfs none /var/tmp ffff880137669180 ffff8800b49b9800 tmpfs none /var/lib/NetworkManager ffff8800b27dc380 ffff8800b49b9800 tmpfs none /var/lib/systemd/random-seed ffff880137669340 ffff8800b49b9800 tmpfs none /var/spool ffff8800b27dc540 ffff8800b49b9800 tmpfs none /var/lib/nfs ffff880137669500 ffff8800b49b9800 tmpfs none /var/lib/gssproxy ffff8801377f7180 ffff8800b49b9800 tmpfs none /var/lib/logrotate ffff8800b27dc700 ffff8800b49b9800 tmpfs none /etc ffff8800b27dc8c0 ffff8800b49b9800 tmpfs none /var/lib/rsyslog ffff8800b27dca80 ffff8800b49b9800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b27dcc40 ffff8800b1da0000 nfs4 192.168.200.253:/exports/state/oleg120-server.virtnet /var/lib/stateless/state ffff8800b27dd180 ffff8800b1da0000 nfs4 192.168.200.253:/exports/state/oleg120-server.virtnet /boot ffff8801377f7500 ffff8800b1da0000 nfs4 192.168.200.253:/exports/state/oleg120-server.virtnet /etc/etc/kdump.conf ffff8800b27dd340 ffff88012a383800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800ad028e00 ffff880000073800 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b1edb500 ffff880129f68800 lustre /dev/mapper/ost2_flakey /mnt/lustre-ost2 ffff8800ad029dc0 ffff880129e96800 lustre /dev/mapper/ost1_flakey /mnt/lustre-ost1 ffff8800ad029a40 ffff880136839000 lustre /dev/mapper/mds2_flakey /mnt/lustre-mds2 ffff8800ad029500 ffff880122581000 lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [19061.973607] Lustre: *** cfs_fail_loc=20e, val=1*** [19061.975820] Lustre: Skipped 3 previous similar messages [19064.957257] Lustre: *** cfs_fail_loc=20e, val=1*** [19064.959921] Lustre: Skipped 3 previous similar messages [19068.986933] Lustre: *** cfs_fail_loc=20e, val=1*** [19068.988735] Lustre: Skipped 3 previous similar messages [19079.972980] Lustre: *** cfs_fail_loc=20e, val=1*** [19079.975578] Lustre: Skipped 7 previous similar messages [19104.013587] Lustre: *** cfs_fail_loc=20e, val=1*** [19104.016537] Lustre: Skipped 11 previous similar messages [19137.087082] Lustre: *** cfs_fail_loc=20f, val=1*** [19137.089129] Lustre: Skipped 37 previous similar messages [19208.182115] Lustre: *** cfs_fail_loc=20e, val=2*** [19208.185147] Lustre: Skipped 54 previous similar messages [19286.875025] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 03:44:21 (1746085461) [19294.523961] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 03:44:29 (1746085469) [19294.966388] Lustre: DEBUG MARKER: sanity test_398o: @@@@@@ FAIL: bad KMS [19297.607773] Lustre: DEBUG MARKER: == sanity test 398p: race aio with buffered i/o ========== 03:44:32 (1746085472) [19318.417538] Lustre: DEBUG MARKER: == sanity test 398q: race dio with buffered i/o ========== 03:44:53 (1746085493) [19359.555529] Lustre: DEBUG MARKER: == sanity test 398r: i/o error on file read ============== 03:45:34 (1746085534) [19359.892441] Lustre: *** cfs_fail_loc=20f, val=0*** [19359.894300] Lustre: Skipped 56 previous similar messages [19416.677037] Lustre: DEBUG MARKER: == sanity test 398s: i/o error on mirror file read ======= 03:46:31 (1746085591) [19417.233922] Lustre: DEBUG MARKER: sanity test_398s: @@@@@@ FAIL: mirror resync failed [19419.941832] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write ========================================================== 03:46:34 (1746085594) [19433.270156] Lustre: *** cfs_fail_loc=238, val=0*** [19446.005106] Lustre: DEBUG MARKER: == sanity test 399b: fake read should not be slower than normal read ========================================================== 03:47:00 (1746085620) [19458.005954] Lustre: DEBUG MARKER: SKIP: sanity test_400a skipping excluded test 400a [19458.482109] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled ==== 03:47:13 (1746085633) [19460.318269] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively ========================================================== 03:47:15 (1746085635) [19462.293801] Lustre: DEBUG MARKER: == sanity test 401aa: Verify that 'lctl list_param -p' lists the correct path names ========================================================== 03:47:17 (1746085637) [19464.122472] Lustre: DEBUG MARKER: == sanity test 401ab: Check that 'lctl list_param -r' lists only readable params ========================================================== 03:47:19 (1746085639) [19465.936919] Lustre: DEBUG MARKER: == sanity test 401ac: Check that 'lctl list_param -w' lists only writable params ========================================================== 03:47:20 (1746085640) [19467.959996] Lustre: DEBUG MARKER: == sanity test 401ad: Check that 'lctl list_param -wr' is conjunctive ========================================================== 03:47:22 (1746085642) [19470.196861] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error ========================================================== 03:47:25 (1746085645) [19472.055299] Lustre: DEBUG MARKER: == sanity test 401c: Verify 'lctl set_param' without value fails in either format. ========================================================== 03:47:26 (1746085646) [19473.918898] Lustre: DEBUG MARKER: == sanity test 401d: Verify 'lctl set_param' accepts values containing '=' ========================================================== 03:47:28 (1746085648) [19475.908315] Lustre: DEBUG MARKER: == sanity test 401db: Verify 'lctl set_param' does not add trailing '=' ========================================================== 03:47:30 (1746085650) [19476.216997] Lustre: Setting parameter general.at_min in log params [19478.596693] Lustre: Disabling parameter general.at_min in log params ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 13.01s (real) 6.95s (CPU), Child processes: 6.02s