************************ crashinfo ************************* /exports/testreports/51244/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg137-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.aJGCe/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51244/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg137-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 01:18:54 EDT 2025 UPTIME: 01:28:16 LOAD AVERAGE: 0.87, 0.62, 0.61 TASKS: 153 NODENAME: oleg137-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 27 last 5s 39 last 60s 44 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 149 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 17531 kworker/1:1 0 ms (no user stack) 19486 kworker/0:1 0 ms (no user stack) 34 rcuos/3 1 ms (no user stack) 9 rcu_sched 1 ms (no user stack) 1975 socknal_sd00_00 7 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 827620 3.2 GB 86% of TOTAL MEM USED 127459 497.9 MB 13% of TOTAL MEM SHARED 10455 40.8 MB 1% of TOTAL MEM BUFFERS 5233 20.4 MB 0% of TOTAL MEM CACHED 46411 181.3 MB 4% of TOTAL MEM SLAB 42031 164.2 MB 4% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 64107 250.4 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5293.5 eth0 n/a 0.0 RSS_TOTAL=80520 pages, %mem= 1.4 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aa681c0 ffff88012b365800 sysfs sysfs /sys ffff88012aa68380 ffff880139944000 proc proc /proc ffff88012aa68540 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aa68700 ffff88012b365000 securityfs securityfs /sys/kernel/security ffff88012aa688c0 ffff88012b366000 tmpfs tmpfs /dev/shm ffff88012aa68a80 ffff880137113000 devpts devpts /dev/pts ffff88012aa68c40 ffff88012b366800 tmpfs tmpfs /run ffff88012aa68e00 ffff88012b367000 tmpfs tmpfs /sys/fs/cgroup ffff88012aa68fc0 ffff88012b367800 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aa69180 ffff88012aa40000 pstore pstore /sys/fs/pstore ffff88012aa69340 ffff88012aa42000 cgroup cgroup /sys/fs/cgroup/devices ffff88012aa69500 ffff88012aa41800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aa696c0 ffff88012aa41000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aa69880 ffff88012aa40800 cgroup cgroup /sys/fs/cgroup/pids ffff88012aa69a40 ffff88012aa42800 cgroup cgroup /sys/fs/cgroup/memory ffff88012aa69c00 ffff88012aa43000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aa69dc0 ffff88012aa43800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a9dc000 ffff88012aa44000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a9dc1c0 ffff88012aa44800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a9dc380 ffff88012aa45000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012b268540 ffff88012b273800 configfs configfs /sys/kernel/config ffff88012a9dc540 ffff8800b5888000 ext4 /dev/nbd0 / ffff880138ccb880 ffff88012b277000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff880137669880 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff880137669a40 ffff8800b58e5800 hugetlbfs hugetlbfs /dev/hugepages ffff88012b268380 ffff88012b270000 mqueue mqueue /dev/mqueue ffff88012b268a80 ffff8800b605b800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880137669c00 ffff8800b58e6000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff880138ccba40 ffff8800b588f000 ramfs none /mnt ffff88012a9dc8c0 ffff8800b6328000 tmpfs none /var/lib/stateless/writable ffff88012a9dca80 ffff8800b632f800 squashfs /dev/vda /home/green/git/lustre-release ffff88012a9dcc40 ffff8800b6328000 tmpfs none /var/cache/man ffff88012b268c40 ffff8800b6328000 tmpfs none /var/log ffff88012a9dce00 ffff8800b6328000 tmpfs none /var/lib/dbus ffff88012a9dcfc0 ffff8800b6328000 tmpfs none /tmp ffff88012a9dd180 ffff8800b6328000 tmpfs none /var/lib/dhclient ffff88012a9dd340 ffff8800b6328000 tmpfs none /var/tmp ffff88012a9dd500 ffff8800b6328000 tmpfs none /var/lib/NetworkManager ffff8801376696c0 ffff8800b6328000 tmpfs none /var/lib/systemd/random-seed ffff880137669500 ffff8800b6328000 tmpfs none /var/spool ffff880137669340 ffff8800b6328000 tmpfs none /var/lib/nfs ffff88012a9dd6c0 ffff8800b6328000 tmpfs none /var/lib/gssproxy ffff8800b5bc0000 ffff8800b6328000 tmpfs none /var/lib/logrotate ffff88012a9dd880 ffff8800b6328000 tmpfs none /etc ffff88012b268e00 ffff8800b6328000 tmpfs none /var/lib/rsyslog ffff88012a9dda40 ffff8800b6328000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012a9ddc00 ffff88012a520000 nfs4 192.168.200.253:/exports/state/oleg137-client.virtnet /var/lib/stateless/state ffff88012e1361c0 ffff88012a520000 nfs4 192.168.200.253:/exports/state/oleg137-client.virtnet /boot ffff88012e136380 ffff88012a520000 nfs4 192.168.200.253:/exports/state/oleg137-client.virtnet /etc/etc/kdump.conf ffff8800b5bc01c0 ffff88012b277000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff880138ccbc00 ffff8800b6e0f000 nfs4 192.168.200.253://exports/testreports/51244/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff88012b2681c0 ffff8800b6e0f800 tmpfs tmpfs /run/user/0 ffff88012b269a40 ffff8800b632f800 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b3a4bdc0 ffff8800aaecf800 lustre 192.168.201.137@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 9.937859] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 9.959794] AVX version of gcm_enc/dec engaged. [ 9.962406] AES CTR mode by8 optimization enabled [ 9.973664] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 10.003952] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 10.010794] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 10.139031] EDAC MC: Ver: 3.0.0 [ 10.148982] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 10.148993] EDAC sbridge: Ver: 1.1.2 [ 12.242295] mount.nfs (766) used greatest stack depth: 10704 bytes left [ 52.310168] libcfs: loading out-of-tree module taints kernel. [ 52.391719] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 52.769641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 52.781385] alg: No test for adler32 (adler32-zlib) [ 53.935483] Lustre: Lustre: Build Version: 2.16.54_58_ga30b9d5 [ 54.396823] LNet: Added LNI 192.168.201.37@tcp [8/256/0/180] [ 54.408108] LNet: Accept secure, port 988 [ 56.065571] Key type lgssc registered [ 56.922410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.352865] Lustre: Mounted lustre-client [ 145.609862] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 158.250443] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing check_logdir /tmp/testlogs/ [ 160.947496] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing yml_node [ 164.718566] Lustre: DEBUG MARKER: Client: 2.16.54.58 [ 166.557248] Lustre: DEBUG MARKER: MDS: 2.16.54.58 [ 167.333487] Lustre: lustre-OST0000-osc-ffff8800b6e09000: disconnect after 24s idle [ 168.354548] Lustre: DEBUG MARKER: OSS: 2.16.54.58 [ 169.348523] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Apr 30 23:53:25 EDT 2025 [ 177.781656] hrtimer: interrupt took 3464458 ns [ 182.004292] Lustre: DEBUG MARKER: excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a [ 183.116216] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 189.540586] random: crng init done [ 193.173117] LustreError: 7712:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 193.207110] Lustre: Unmounted lustre-client [ 259.909831] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 274.880687] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 291.808474] Lustre: DEBUG MARKER: == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:55:27 (1746071727) [ 338.494904] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 351.752482] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 352.198802] Lustre: Mounted lustre-client ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 11.45s (real) 5.89s (CPU), Child processes: 5.50s