************************ crashinfo ************************* /exports/testreports/51245/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg211-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.ESKtk/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51245/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg211-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 04:19:50 EDT 2025 UPTIME: 04:26:58 LOAD AVERAGE: 0.12, 0.47, 0.65 TASKS: 165 NODENAME: oleg211-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 23 last 5s 60 last 60s 73 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 161 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 11 rcuos/0 0 ms (no user stack) 1 systemd 0 ms /usr/lib/systemd/systemd --switched-root --system --deserialize 22 9 rcu_sched 9 ms (no user stack) 20 rcuos/1 61 ms (no user stack) 32136 sleep 70 ms sleep 1 +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 850801 3.2 GB 89% of TOTAL MEM USED 104278 407.3 MB 10% of TOTAL MEM SHARED 9099 35.5 MB 0% of TOTAL MEM BUFFERS 1026 4 MB 0% of TOTAL MEM CACHED 29869 116.7 MB 3% of TOTAL MEM SLAB 12487 48.8 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 130 520 KB 0% of TOTAL SWAP SWAP FREE 262013 1023.5 MB 99% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 84932 331.8 MB 11% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 10 TIME_WAIT 32 LISTEN 3 NAGLE disabled (TCP_NODELAY): 8 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 16016.4 eth0 n/a 0.2 RSS_TOTAL=134640 pages, %mem= 2.4 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aaf0000 ffff88012b39d000 sysfs sysfs /sys ffff88012aaf01c0 ffff880139944000 proc proc /proc ffff88012aaf0380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aaf0540 ffff8800b6d04800 securityfs securityfs /sys/kernel/security ffff88012aaf0700 ffff88012b39d800 tmpfs tmpfs /dev/shm ffff88012aaf08c0 ffff88012dc31000 devpts devpts /dev/pts ffff88012aaf0a80 ffff88012b39e000 tmpfs tmpfs /run ffff88012aaf0c40 ffff88012b39e800 tmpfs tmpfs /sys/fs/cgroup ffff88012aaf0e00 ffff88012b39f000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aaf0fc0 ffff88012b39f800 pstore pstore /sys/fs/pstore ffff88012aaf1180 ffff88012aa81800 cgroup cgroup /sys/fs/cgroup/pids ffff88012aaf1340 ffff88012aa81000 cgroup cgroup /sys/fs/cgroup/devices ffff88012aaf1500 ffff88012aa80800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aaf16c0 ffff88012aa80000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aaf1880 ffff88012aa82000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aaf1a40 ffff88012aa82800 cgroup cgroup /sys/fs/cgroup/memory ffff88012aaf1c00 ffff88012aa83000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aaf1dc0 ffff88012aa83800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aaf4000 ffff88012aa84000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aaf41c0 ffff88012aa84800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aaf4380 ffff8800b6148000 configfs configfs /sys/kernel/config ffff880137669500 ffff8800b61ba800 ext4 /dev/nbd0 / ffff88012aaf4540 ffff88012dc36000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012b25a8c0 ffff8800b609a000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff88012b25ae00 ffff88012dc32800 mqueue mqueue /dev/mqueue ffff88012aaf4c40 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff880138ccb340 ffff8800b6303800 hugetlbfs hugetlbfs /dev/hugepages ffff88012aaf4e00 ffff8800b614e800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8801376696c0 ffff8800b6f36000 ramfs none /mnt ffff88012b25afc0 ffff8800b5c5d000 tmpfs none /var/lib/stateless/writable ffff88012aaf5180 ffff8800b609d800 squashfs /dev/vda /home/green/git/lustre-release ffff88012aaf5340 ffff8800b5c5d000 tmpfs none /var/cache/man ffff88012aaf5500 ffff8800b5c5d000 tmpfs none /var/log ffff880137669a40 ffff8800b5c5d000 tmpfs none /var/lib/dbus ffff880138ccb500 ffff8800b5c5d000 tmpfs none /tmp ffff880137669c00 ffff8800b5c5d000 tmpfs none /var/lib/dhclient ffff880138ccb6c0 ffff8800b5c5d000 tmpfs none /var/tmp ffff880137669dc0 ffff8800b5c5d000 tmpfs none /var/lib/NetworkManager ffff88012aaf56c0 ffff8800b5c5d000 tmpfs none /var/lib/systemd/random-seed ffff88012aaf5880 ffff8800b5c5d000 tmpfs none /var/spool ffff8800b383c000 ffff8800b5c5d000 tmpfs none /var/lib/nfs ffff88012aaf5a40 ffff8800b5c5d000 tmpfs none /var/lib/gssproxy ffff8800b383c1c0 ffff8800b5c5d000 tmpfs none /var/lib/logrotate ffff88012aaf5c00 ffff8800b5c5d000 tmpfs none /etc ffff8800b383c380 ffff8800b5c5d000 tmpfs none /var/lib/rsyslog ffff88012aaf5dc0 ffff8800b5c5d000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012aaf4a80 ffff88012dc36800 nfs4 192.168.200.253:/exports/state/oleg211-client.virtnet /var/lib/stateless/state ffff880138ccb880 ffff88012dc36800 nfs4 192.168.200.253:/exports/state/oleg211-client.virtnet /boot ffff88012aaf48c0 ffff88012dc36800 nfs4 192.168.200.253:/exports/state/oleg211-client.virtnet /etc/etc/kdump.conf ffff8800b383c540 ffff88012dc36000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b38b2380 ffff8800b3924000 nfs4 192.168.200.253://exports/testreports/51245/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff88012e112380 ffff8800b609b000 tmpfs tmpfs /run/user/0 ffff88012f311180 ffff8800b609d800 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b38b2a80 ffff88009372c000 lustre 192.168.202.111@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [15777.231300] Lustre: DEBUG MARKER: == sanity test 807a: verify LSOM syncing tool ============ 04:15:48 (1746087348) [15780.019647] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing cancel_lru_locks osc [15788.994529] Lustre: DEBUG MARKER: == sanity test 807b: verify lfs somsync utility ========== 04:16:00 (1746087360) [15790.087383] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing cancel_lru_locks osc [15797.108784] Lustre: DEBUG MARKER: == sanity test 808: Check trusted.som xattr not logged in Changelogs ========================================================== 04:16:08 (1746087368) [15803.165441] Lustre: DEBUG MARKER: == sanity test 809: Verify no SOM xattr store for DoM-only files ========================================================== 04:16:14 (1746087374) [15805.737562] Lustre: DEBUG MARKER: == sanity test 810: partial page writes on ZFS (LU-11663) ========================================================== 04:16:16 (1746087376) [15805.805581] Lustre: *** cfs_fail_loc=411, val=0*** [15805.806726] Lustre: Skipped 1 previous similar message [15806.338835] Lustre: *** cfs_fail_loc=411, val=0*** [15806.340155] Lustre: Skipped 15 previous similar messages [15807.407016] Lustre: *** cfs_fail_loc=411, val=0*** [15807.408950] Lustre: Skipped 23 previous similar messages [15809.950199] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [15810.409040] Lustre: DEBUG MARKER: == sanity test 812a: do not drop reqs generated when imp is going to idle (LU-11951) ========================================================== 04:16:21 (1746087381) [15811.288936] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid 50 [15811.647876] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid in FULL state after 0 sec [15812.761215] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state CONNECTING osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid 50 [15825.531176] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid in CONNECTING state after 12 sec [15828.615297] Lustre: DEBUG MARKER: == sanity test 812b: do not drop no resend request for idle connect ========================================================== 04:16:39 (1746087399) [15829.860147] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid 50 [15830.387576] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid in FULL state after 0 sec [15831.734633] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state CONNECTING osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid 50 [15845.340885] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid in CONNECTING state after 13 sec [15846.516571] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid 50 [15860.335912] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid in IDLE state after 13 sec [15862.561861] Lustre: DEBUG MARKER: == sanity test 812c: idle import vs lock enqueue race ==== 04:17:13 (1746087433) [15863.451095] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid 50 [15863.791926] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid in FULL state after 0 sec [15874.310507] LustreError: 20060:0:(import.c:2037:ptlrpc_disconnect_and_idle_import()) cfs_race id 533 sleeping [15878.856990] LustreError: 28610:0:(osc_lock.c:1024:osc_lock_enqueue()) cfs_fail_race id 533 waking [15878.860928] LustreError: 20060:0:(import.c:2037:ptlrpc_disconnect_and_idle_import()) cfs_fail_race id 533 awake: rc=452 [15879.364168] LustreError: 28610:0:(osc_lock.c:1024:osc_lock_enqueue()) cfs_fail_race id 533 waking [15882.056346] Lustre: DEBUG MARKER: == sanity test 813: File heat verfication ================ 04:17:33 (1746087453) [16010.713889] Lustre: DEBUG MARKER: == sanity test 814: sparse cp works as expected (LU-12361) ========================================================== 04:19:42 (1746087582) [16012.734564] Lustre: DEBUG MARKER: == sanity test 815: zero byte tiny write doesn't hang (LU-12382) ========================================================== 04:19:43 (1746087583) [16014.671482] Lustre: DEBUG MARKER: == sanity test 816: do not reset lru_resize on idle reconnect ========================================================== 04:19:45 (1746087585) [16015.634562] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid 50 [16016.003374] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid in FULL state after 0 sec [16017.015324] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88009372c000.ost_server_uuid 50 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 11.85s (real) 6.06s (CPU), Child processes: 5.76s