************************ crashinfo ************************* /exports/testreports/51277/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg153-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.BmsdR/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51277/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg153-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 17:43:22 EDT 2025 UPTIME: 01:27:34 LOAD AVERAGE: 0.68, 0.57, 0.59 TASKS: 152 NODENAME: oleg153-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 32 last 5s 41 last 60s 42 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 148 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 1962 socknal_sd00_01 0 ms (no user stack) 1961 socknal_sd00_00 0 ms (no user stack) 19161 createmany 2 ms /home/green/git/lustre-release/lustre/tests/createmany -o /mnt/lustre/d106.conf-sanity/f- 64768 19566 kworker/0:2 3 ms (no user stack) 6 ksoftirqd/0 3 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 794337 3 GB 83% of TOTAL MEM USED 160742 627.9 MB 16% of TOTAL MEM SHARED 10463 40.9 MB 1% of TOTAL MEM BUFFERS 5235 20.4 MB 0% of TOTAL MEM CACHED 46405 181.3 MB 4% of TOTAL MEM SLAB 75675 295.6 MB 7% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 64103 250.4 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5252.1 eth0 n/a 0.0 RSS_TOTAL=79924 pages, %mem= 1.3 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aab0000 ffff88012aab8000 sysfs sysfs /sys ffff88012aab01c0 ffff880139944000 proc proc /proc ffff88012aab0380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aab0540 ffff8800b6cf3000 securityfs securityfs /sys/kernel/security ffff88012aab0700 ffff88012aab8800 tmpfs tmpfs /dev/shm ffff88012aab08c0 ffff880136b79000 devpts devpts /dev/pts ffff88012aab0a80 ffff88012aab9000 tmpfs tmpfs /run ffff88012aab0c40 ffff88012aab9800 tmpfs tmpfs /sys/fs/cgroup ffff88012aab0e00 ffff88012aaba000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aab0fc0 ffff88012aaba800 pstore pstore /sys/fs/pstore ffff88012b27ca80 ffff88012b3cd800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012b27cc40 ffff88012b3cd000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012b27ce00 ffff88012b3cc800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012b27cfc0 ffff88012b3cc000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012b27d180 ffff88012b3ce000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012b27d340 ffff88012b3ce800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012b27d500 ffff88012b3cf000 cgroup cgroup /sys/fs/cgroup/pids ffff88012b27d6c0 ffff88012b3cf800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012b27d880 ffff88012aaf8000 cgroup cgroup /sys/fs/cgroup/memory ffff88012b27da40 ffff88012aaf8800 cgroup cgroup /sys/fs/cgroup/devices ffff880138ccae00 ffff88012a449800 configfs configfs /sys/kernel/config ffff880137668700 ffff8800b6cde800 ext4 /dev/nbd0 / ffff8800b63ba700 ffff88012aabe000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8800b6efa380 ffff88012a5d6000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8800b6efa540 ffff88012a5d2800 hugetlbfs hugetlbfs /dev/hugepages ffff8800b6efa700 ffff880136b7a800 mqueue mqueue /dev/mqueue ffff8800b63ba8c0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff880138ccafc0 ffff88012a606800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b6efa8c0 ffff8800b634f000 ramfs none /mnt ffff880138ccb180 ffff8800b62ee800 tmpfs none /var/lib/stateless/writable ffff880138ccb340 ffff8800b62ed000 squashfs /dev/vda /home/green/git/lustre-release ffff8800b63bac40 ffff8800b62ee800 tmpfs none /var/cache/man ffff880138ccb500 ffff8800b62ee800 tmpfs none /var/log ffff880137668a80 ffff8800b62ee800 tmpfs none /var/lib/dbus ffff880137668c40 ffff8800b62ee800 tmpfs none /tmp ffff8800b63bae00 ffff8800b62ee800 tmpfs none /var/lib/dhclient ffff880137668e00 ffff8800b62ee800 tmpfs none /var/tmp ffff880137668fc0 ffff8800b62ee800 tmpfs none /var/lib/NetworkManager ffff880137669180 ffff8800b62ee800 tmpfs none /var/lib/systemd/random-seed ffff880138ccb6c0 ffff8800b62ee800 tmpfs none /var/spool ffff880137669340 ffff8800b62ee800 tmpfs none /var/lib/nfs ffff880137669500 ffff8800b62ee800 tmpfs none /var/lib/gssproxy ffff8800b63bafc0 ffff8800b62ee800 tmpfs none /var/lib/logrotate ffff8801376696c0 ffff8800b62ee800 tmpfs none /etc ffff880137669880 ffff8800b62ee800 tmpfs none /var/lib/rsyslog ffff880137669a40 ffff8800b62ee800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b63bb180 ffff88012aaf9800 nfs4 192.168.200.253:/exports/state/oleg153-client.virtnet /var/lib/stateless/state ffff8800b63bb6c0 ffff88012aaf9800 nfs4 192.168.200.253:/exports/state/oleg153-client.virtnet /boot ffff8800b63bb500 ffff88012aaf9800 nfs4 192.168.200.253:/exports/state/oleg153-client.virtnet /etc/etc/kdump.conf ffff880137669c00 ffff88012aabe000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff88012b51a1c0 ffff8800b358f000 nfs4 192.168.200.253://exports/testreports/51277/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800b6efb340 ffff88012a44a000 tmpfs tmpfs /run/user/0 ffff8800b3e8ba40 ffff8800b62ed000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b63ba540 ffff8800a994d000 lustre 192.168.201.153@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 9.574470] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 9.593846] AVX version of gcm_enc/dec engaged. [ 9.595951] AES CTR mode by8 optimization enabled [ 9.610448] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.647240] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 9.650145] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 9.763575] EDAC MC: Ver: 3.0.0 [ 9.769425] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.769436] EDAC sbridge: Ver: 1.1.2 [ 11.410518] mount.nfs (768) used greatest stack depth: 10704 bytes left [ 34.086240] libcfs: loading out-of-tree module taints kernel. [ 34.099914] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 34.407492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 34.418899] alg: No test for adler32 (adler32-zlib) [ 35.501373] Lustre: Lustre: Build Version: 2.16.54_85_g2dcc910 [ 35.918984] LNet: Added LNI 192.168.201.53@tcp [8/256/0/180] [ 35.923586] LNet: Accept secure, port 988 [ 37.612305] Key type lgssc registered [ 38.422955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.555365] Lustre: Mounted lustre-client [ 123.187005] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 136.842588] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing check_logdir /tmp/testlogs/ [ 139.086821] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing yml_node [ 142.117433] Lustre: DEBUG MARKER: Client: 2.16.54.85 [ 143.760972] Lustre: DEBUG MARKER: MDS: 2.16.54.85 [ 144.551160] Lustre: lustre-OST0000-osc-ffff8800b3525800: disconnect after 23s idle [ 145.507474] Lustre: DEBUG MARKER: OSS: 2.16.54.85 [ 146.718292] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 1 16:18:12 EDT 2025 [ 158.429671] Lustre: DEBUG MARKER: excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a [ 159.465206] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 169.504892] LustreError: 7702:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 169.539201] Lustre: Unmounted lustre-client [ 189.504867] random: crng init done [ 219.670579] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 232.934370] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 249.036551] Lustre: DEBUG MARKER: == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:19:54 (1746130794) [ 296.353257] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 309.145138] Lustre: DEBUG MARKER: oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 309.554597] Lustre: Mounted lustre-client [ 1044.510278] hrtimer: interrupt took 4161368 ns ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 11.37s (real) 5.83s (CPU), Child processes: 5.47s