************************ crashinfo ************************* /exports/testreports/51276/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg142-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.yozZS/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51276/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg142-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 17:29:16 EDT 2025 UPTIME: 01:28:21 LOAD AVERAGE: 0.56, 0.54, 0.58 TASKS: 151 NODENAME: oleg142-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 27 last 5s 37 last 60s 44 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 147 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 11 rcuos/0 0 ms (no user stack) 23 ksoftirqd/2 0 ms (no user stack) 16718 kworker/3:1 3 ms (no user stack) 9 rcu_sched 4 ms (no user stack) 17857 kworker/2:0 4 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 808907 3.1 GB 84% of TOTAL MEM USED 146172 571 MB 15% of TOTAL MEM SHARED 10452 40.8 MB 1% of TOTAL MEM BUFFERS 5233 20.4 MB 0% of TOTAL MEM CACHED 46405 181.3 MB 4% of TOTAL MEM SLAB 60934 238 MB 6% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 64149 250.6 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 5 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5299.2 eth0 n/a 0.0 RSS_TOTAL=80972 pages, %mem= 1.3 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff880137668380 ffff8800b6cc0800 sysfs sysfs /sys ffff880137668540 ffff880139944000 proc proc /proc ffff880137668700 ffff880137678000 devtmpfs devtmpfs /dev ffff8801376688c0 ffff8800b6ccf800 securityfs securityfs /sys/kernel/security ffff880137668a80 ffff8800b6cc1000 tmpfs tmpfs /dev/shm ffff880137668c40 ffff88012b230000 devpts devpts /dev/pts ffff880137668e00 ffff8800b6cc1800 tmpfs tmpfs /run ffff880137668fc0 ffff8800b6cc2000 tmpfs tmpfs /sys/fs/cgroup ffff880137669180 ffff8800b6cc2800 cgroup cgroup /sys/fs/cgroup/systemd ffff880137669340 ffff8800b6cc3000 pstore pstore /sys/fs/pstore ffff880137669500 ffff8800b6cc5000 cgroup cgroup /sys/fs/cgroup/freezer ffff8801376696c0 ffff8800b6cc4800 cgroup cgroup /sys/fs/cgroup/devices ffff880137669880 ffff8800b6cc4000 cgroup cgroup /sys/fs/cgroup/memory ffff880137669a40 ffff8800b6cc3800 cgroup cgroup /sys/fs/cgroup/perf_event ffff880137669c00 ffff8800b6cc5800 cgroup cgroup /sys/fs/cgroup/pids ffff880137669dc0 ffff8800b6cc6000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aa6c000 ffff8800b6cc6800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aa6c1c0 ffff8800b6cc7000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aa6c380 ffff8800b6cc7800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aa6c540 ffff88012b337800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff8800b6d021c0 ffff8800b6d15000 configfs configfs /sys/kernel/config ffff88012aa6d6c0 ffff8800b6e67800 ext4 /dev/nbd0 / ffff88012aa6d880 ffff8800b6d15800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8800b6100380 ffff8800b6e20800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8800b6d028c0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b6100540 ffff88012b231800 mqueue mqueue /dev/mqueue ffff8800b6100700 ffff8800b6e26800 hugetlbfs hugetlbfs /dev/hugepages ffff88012aa6da40 ffff8800b5cb6800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012aa6dc00 ffff8800b5cb4000 ramfs none /mnt ffff88012aa6ddc0 ffff8800b5c5f800 tmpfs none /var/lib/stateless/writable ffff8800b6d02c40 ffff8800b5ce5000 squashfs /dev/vda /home/green/git/lustre-release ffff8800b6d02e00 ffff8800b5c5f800 tmpfs none /var/cache/man ffff8800b61008c0 ffff8800b5c5f800 tmpfs none /var/log ffff8800b3c7e1c0 ffff8800b5c5f800 tmpfs none /var/lib/dbus ffff8800b6d02fc0 ffff8800b5c5f800 tmpfs none /tmp ffff8800b6d03180 ffff8800b5c5f800 tmpfs none /var/lib/dhclient ffff8800b6d03340 ffff8800b5c5f800 tmpfs none /var/tmp ffff8800b6d03500 ffff8800b5c5f800 tmpfs none /var/lib/NetworkManager ffff8800b3c7e380 ffff8800b5c5f800 tmpfs none /var/lib/systemd/random-seed ffff8800b6100a80 ffff8800b5c5f800 tmpfs none /var/spool ffff8800b6d036c0 ffff8800b5c5f800 tmpfs none /var/lib/nfs ffff8800b3c7e540 ffff8800b5c5f800 tmpfs none /var/lib/gssproxy ffff8800b3c7e700 ffff8800b5c5f800 tmpfs none /var/lib/logrotate ffff8800b6100c40 ffff8800b5c5f800 tmpfs none /etc ffff8800b3c7e8c0 ffff8800b5c5f800 tmpfs none /var/lib/rsyslog ffff8800b6d03880 ffff8800b5c5f800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b3c7ea80 ffff8800b5ce4800 nfs4 192.168.200.253:/exports/state/oleg142-client.virtnet /var/lib/stateless/state ffff8800b6d03c00 ffff8800b5ce4800 nfs4 192.168.200.253:/exports/state/oleg142-client.virtnet /boot ffff8800b3c7ec40 ffff8800b5ce4800 nfs4 192.168.200.253:/exports/state/oleg142-client.virtnet /etc/etc/kdump.conf ffff8800b3c7ee00 ffff8800b6d15800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8801369c4a80 ffff88012aa78800 nfs4 192.168.200.253://exports/testreports/51276/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8801369c4c40 ffff8800b636e800 tmpfs tmpfs /run/user/0 ffff880138ccb880 ffff8800b5ce5000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8801369c4540 ffff88012c50c800 lustre 192.168.201.142@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 9.690165] AVX version of gcm_enc/dec engaged. [ 9.691532] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.694274] AES CTR mode by8 optimization enabled [ 9.721111] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 9.754153] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 9.757306] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 9.880387] EDAC MC: Ver: 3.0.0 [ 9.890988] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.891007] EDAC sbridge: Ver: 1.1.2 [ 11.919587] mount.nfs (767) used greatest stack depth: 10704 bytes left [ 17.588725] hrtimer: interrupt took 7680626 ns [ 47.783715] libcfs: loading out-of-tree module taints kernel. [ 47.787284] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 48.170590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.183628] alg: No test for adler32 (adler32-zlib) [ 49.402180] Lustre: Lustre: Build Version: 2.16.54_86_g77fe414 [ 49.876087] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 49.884257] LNet: Accept secure, port 988 [ 51.591126] Key type lgssc registered [ 52.959050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 141.623205] Lustre: Mounted lustre-client [ 145.999938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 165.916375] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing check_logdir /tmp/testlogs/ [ 166.619195] Lustre: lustre-OST0000-osc-ffff8800b36a4000: disconnect after 23s idle [ 168.432324] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing yml_node [ 172.191800] Lustre: DEBUG MARKER: Client: 2.16.54.86 [ 174.401359] Lustre: DEBUG MARKER: MDS: 2.16.54.86 [ 176.344477] Lustre: DEBUG MARKER: OSS: 2.16.54.86 [ 177.590413] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 1 16:03:50 EDT 2025 [ 182.299718] random: crng init done [ 190.988452] Lustre: DEBUG MARKER: excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a [ 192.545632] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 202.574811] LustreError: 7760:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 202.604656] Lustre: Unmounted lustre-client [ 264.622489] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 280.075316] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 295.139804] Lustre: DEBUG MARKER: == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:05:47 (1746129947) [ 348.993715] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 363.014213] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 363.531281] Lustre: Mounted lustre-client ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 12.49s (real) 5.92s (CPU), Child processes: 6.41s