************************ crashinfo ************************* /exports/testreports/51293/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg339-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.5dFhY/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51293/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg339-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 22:25:00 EDT 2025 UPTIME: 01:25:53 LOAD AVERAGE: 0.40, 0.70, 0.67 TASKS: 141 NODENAME: oleg339-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 32 last 5s 39 last 60s 56 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 137 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 20965 ssh 39 ms ssh -oConnectTimeout=10 -2 -a -x oleg339-server (PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin; cd /root; LUSTRE="/home/green/git/lustre-release/lustre" bash -c "umount -d -f /mnt/lustre-mds1") 20483 kworker/0:2 57 ms (no user stack) 1952 monitor_thread 57 ms (no user stack) 11 rcuos/0 121 ms (no user stack) 9 rcu_sched 121 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 857543 3.3 GB 89% of TOTAL MEM USED 97536 381 MB 10% of TOTAL MEM SHARED 10776 42.1 MB 1% of TOTAL MEM BUFFERS 5232 20.4 MB 0% of TOTAL MEM CACHED 46411 181.3 MB 4% of TOTAL MEM SLAB 11819 46.2 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 63708 248.9 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 7 TIME_WAIT 13 LISTEN 3 NAGLE disabled (TCP_NODELAY): 7 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 30 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5150.7 eth0 n/a 0.1 RSS_TOTAL=86072 pages, %mem= 1.4 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff880137668380 ffff880137678800 sysfs sysfs /sys ffff880137668540 ffff880139944000 proc proc /proc ffff880137668700 ffff880137678000 devtmpfs devtmpfs /dev ffff8801376688c0 ffff8800b6ce8800 securityfs securityfs /sys/kernel/security ffff880137668a80 ffff880137679000 tmpfs tmpfs /dev/shm ffff880137668c40 ffff88013771f000 devpts devpts /dev/pts ffff880137668e00 ffff880137679800 tmpfs tmpfs /run ffff880137668fc0 ffff88013767a000 tmpfs tmpfs /sys/fs/cgroup ffff880137669180 ffff88013767a800 cgroup cgroup /sys/fs/cgroup/systemd ffff880137669340 ffff88013767b000 pstore pstore /sys/fs/pstore ffff880137669500 ffff88013767d000 cgroup cgroup /sys/fs/cgroup/perf_event ffff8801376696c0 ffff88013767c800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff880137669880 ffff88013767c000 cgroup cgroup /sys/fs/cgroup/blkio ffff880137669a40 ffff88013767b800 cgroup cgroup /sys/fs/cgroup/cpuset ffff880137669c00 ffff88013767d800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff880137669dc0 ffff88013767e000 cgroup cgroup /sys/fs/cgroup/devices ffff88012aade000 ffff88013767e800 cgroup cgroup /sys/fs/cgroup/pids ffff88012aade1c0 ffff88013767f000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aade380 ffff88013767f800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aade540 ffff88012ab40000 cgroup cgroup /sys/fs/cgroup/memory ffff88012aade700 ffff88012ab45800 configfs configfs /sys/kernel/config ffff88012aadea80 ffff88012ab40800 ext4 /dev/nbd0 / ffff8800b6e58380 ffff8801372fa000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8800b6e588c0 ffff880000092000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8801377f2700 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b6e58a80 ffff88012b288800 mqueue mqueue /dev/mqueue ffff8800b6e58c40 ffff8801372fc800 hugetlbfs hugetlbfs /dev/hugepages ffff8800b6e58e00 ffff8800b5991800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b6e58fc0 ffff8800b5997000 ramfs none /mnt ffff8800b6e59180 ffff8800b5878000 tmpfs none /var/lib/stateless/writable ffff8801377f28c0 ffff8800b5928000 squashfs /dev/vda /home/green/git/lustre-release ffff8800b6e59340 ffff8800b5878000 tmpfs none /var/cache/man ffff88012aadee00 ffff8800b5878000 tmpfs none /var/log ffff88012aadefc0 ffff8800b5878000 tmpfs none /var/lib/dbus ffff8800b596a8c0 ffff8800b5878000 tmpfs none /tmp ffff88012aadf180 ffff8800b5878000 tmpfs none /var/lib/dhclient ffff88012aadf340 ffff8800b5878000 tmpfs none /var/tmp ffff88012aadf500 ffff8800b5878000 tmpfs none /var/lib/NetworkManager ffff8800b596aa80 ffff8800b5878000 tmpfs none /var/lib/systemd/random-seed ffff88012aadf6c0 ffff8800b5878000 tmpfs none /var/spool ffff88012aadf880 ffff8800b5878000 tmpfs none /var/lib/nfs ffff88012aadfa40 ffff8800b5878000 tmpfs none /var/lib/gssproxy ffff88012aadfc00 ffff8800b5878000 tmpfs none /var/lib/logrotate ffff8800b6e59500 ffff8800b5878000 tmpfs none /etc ffff8800b6e596c0 ffff8800b5878000 tmpfs none /var/lib/rsyslog ffff88012aadfdc0 ffff8800b5878000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b6e59880 ffff8800b5ae6000 nfs4 192.168.200.253:/exports/state/oleg339-client.virtnet /var/lib/stateless/state ffff8800b6e59c00 ffff8800b5ae6000 nfs4 192.168.200.253:/exports/state/oleg339-client.virtnet /boot ffff8800b6e59dc0 ffff8800b5ae6000 nfs4 192.168.200.253:/exports/state/oleg339-client.virtnet /etc/etc/kdump.conf ffff8800b6e59a40 ffff8801372fa000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b3e34fc0 ffff88012a623800 nfs4 192.168.200.253://exports/testreports/51293/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8801377f3340 ffff8800b612f800 tmpfs tmpfs /run/user/0 ffff880138ccba40 ffff8800b5928000 squashfs /dev/vda /usr/sbin/mount.lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 8.114936] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.128879] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.224314] EDAC MC: Ver: 3.0.0 [ 8.229853] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 8.229865] EDAC sbridge: Ver: 1.1.2 [ 10.029889] mount.nfs (766) used greatest stack depth: 10704 bytes left [ 15.541689] mount.nfs (1114) used greatest stack depth: 10528 bytes left [ 24.590351] libcfs: loading out-of-tree module taints kernel. [ 24.592699] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 24.744571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 24.749671] alg: No test for adler32 (adler32-zlib) [ 25.672671] Lustre: Lustre: Build Version: 2.16.54_69_g06f2fd3 [ 25.877991] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 25.880131] LNet: Accept secure, port 988 [ 27.434484] Key type lgssc registered [ 27.795523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.508433] Lustre: Mounted lustre-client [ 75.345913] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 86.399347] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing check_logdir /tmp/testlogs/ [ 87.616914] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing yml_node [ 89.353010] Lustre: DEBUG MARKER: Client: 2.16.54.69 [ 90.377812] Lustre: DEBUG MARKER: MDS: 2.16.54.69 [ 91.364217] Lustre: DEBUG MARKER: OSS: 2.16.54.69 [ 91.927384] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 1 21:00:38 EDT 2025 [ 97.510459] Lustre: lustre-OST0000-osc-ffff8800b612c000: disconnect after 24s idle [ 98.789279] Lustre: DEBUG MARKER: excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a [ 99.323874] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 109.350039] LustreError: 7649:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 109.367502] Lustre: Unmounted lustre-client [ 135.519820] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 141.597520] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 148.428736] Lustre: DEBUG MARKER: == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:01:34 (1746147694) [ 168.533777] random: crng init done [ 179.611890] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 188.191252] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 188.472232] Lustre: Mounted lustre-client [ 1723.190329] hrtimer: interrupt took 2094278 ns [ 5144.097581] LustreError: 20883:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5144.104538] LustreError: 20883:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5144.131504] Lustre: Unmounted lustre-client ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 11.19s (real) 5.67s (CPU), Child processes: 5.39s