************************ crashinfo ************************* /exports/testreports/51272/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg126-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.bAGUN/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51272/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg126-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 16:28:56 EDT 2025 UPTIME: 01:27:17 LOAD AVERAGE: 0.72, 0.71, 0.77 TASKS: 151 NODENAME: oleg126-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 27 last 5s 36 last 60s 44 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 147 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 18735 kworker/3:0 0 ms (no user stack) 18779 kworker/2:2 0 ms (no user stack) 20 rcuos/1 4 ms (no user stack) 1964 socknal_sd01_00 7 ms (no user stack) 19443 createmany 9 ms /home/green/git/lustre-release/lustre/tests/createmany -o /mnt/lustre/d106.conf-sanity/f- 64768 +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 828073 3.2 GB 86% of TOTAL MEM USED 127006 496.1 MB 13% of TOTAL MEM SHARED 10466 40.9 MB 1% of TOTAL MEM BUFFERS 5235 20.4 MB 0% of TOTAL MEM CACHED 46413 181.3 MB 4% of TOTAL MEM SLAB 41244 161.1 MB 4% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 64210 250.8 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5231.9 eth0 n/a 0.0 RSS_TOTAL=82640 pages, %mem= 1.3 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff880137668380 ffff88012b3ec000 sysfs sysfs /sys ffff880137668540 ffff880139944000 proc proc /proc ffff880137668700 ffff880137678000 devtmpfs devtmpfs /dev ffff8801376688c0 ffff88012b32d000 securityfs securityfs /sys/kernel/security ffff880137668a80 ffff88012b3ec800 tmpfs tmpfs /dev/shm ffff880137668c40 ffff880137332000 devpts devpts /dev/pts ffff880137668e00 ffff88012b3ed000 tmpfs tmpfs /run ffff880137668fc0 ffff88012b3ed800 tmpfs tmpfs /sys/fs/cgroup ffff880137669180 ffff88012b3ee000 cgroup cgroup /sys/fs/cgroup/systemd ffff880137669340 ffff88012b3ee800 pstore pstore /sys/fs/pstore ffff880137669500 ffff88012ab30800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff8801376696c0 ffff88012ab30000 cgroup cgroup /sys/fs/cgroup/pids ffff880137669880 ffff88012ab31000 cgroup cgroup /sys/fs/cgroup/devices ffff880137669a40 ffff88012ab31800 cgroup cgroup /sys/fs/cgroup/blkio ffff880137669c00 ffff88012ab32000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff880137669dc0 ffff88012ab32800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012ab3a000 ffff88012ab33000 cgroup cgroup /sys/fs/cgroup/memory ffff88012ab3a1c0 ffff88012ab33800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012ab3a380 ffff88012ab34000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012ab3a540 ffff88012ab34800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff880137012a80 ffff8800b6cec800 configfs configfs /sys/kernel/config ffff880138ccaa80 ffff8800b6e3b000 ext4 /dev/nbd0 / ffff880138ccac40 ffff88012b32a000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8801370bbc00 ffff88012a632000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880137013180 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff88012ab3a8c0 ffff8800b61c5800 hugetlbfs hugetlbfs /dev/hugepages ffff88012ab3aa80 ffff880137333800 mqueue mqueue /dev/mqueue ffff8801370ba380 ffff88012a631000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8801370bba40 ffff88012a637000 ramfs none /mnt ffff88012ab3ac40 ffff8800b61c5000 tmpfs none /var/lib/stateless/writable ffff880137013500 ffff8800b62dc000 squashfs /dev/vda /home/green/git/lustre-release ffff88012ab3ae00 ffff8800b61c5000 tmpfs none /var/cache/man ffff8801370bb880 ffff8800b61c5000 tmpfs none /var/log ffff8801370136c0 ffff8800b61c5000 tmpfs none /var/lib/dbus ffff88012ab3afc0 ffff8800b61c5000 tmpfs none /tmp ffff8801370ba700 ffff8800b61c5000 tmpfs none /var/lib/dhclient ffff8800b620e000 ffff8800b61c5000 tmpfs none /var/tmp ffff88012ab3b180 ffff8800b61c5000 tmpfs none /var/lib/NetworkManager ffff880137013880 ffff8800b61c5000 tmpfs none /var/lib/systemd/random-seed ffff880138ccae00 ffff8800b61c5000 tmpfs none /var/spool ffff880137013a40 ffff8800b61c5000 tmpfs none /var/lib/nfs ffff880137013c00 ffff8800b61c5000 tmpfs none /var/lib/gssproxy ffff88012ab3b340 ffff8800b61c5000 tmpfs none /var/lib/logrotate ffff880138ccafc0 ffff8800b61c5000 tmpfs none /etc ffff8800b620e1c0 ffff8800b61c5000 tmpfs none /var/lib/rsyslog ffff88012ab3b500 ffff8800b61c5000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff880138ccb180 ffff8800b635f000 nfs4 192.168.200.253:/exports/state/oleg126-client.virtnet /var/lib/stateless/state ffff8800b620e380 ffff8800b635f000 nfs4 192.168.200.253:/exports/state/oleg126-client.virtnet /boot ffff880138ccb340 ffff8800b635f000 nfs4 192.168.200.253:/exports/state/oleg126-client.virtnet /etc/etc/kdump.conf ffff880138ccb500 ffff88012b32a000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff880138ccb6c0 ffff8800b5d07800 nfs4 192.168.200.253://exports/testreports/51272/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff880138ccbc00 ffff88012e175800 tmpfs tmpfs /run/user/0 ffff88012e17e1c0 ffff8800b62dc000 squashfs /dev/vda /usr/sbin/mount.lustre ffff880136866e00 ffff8800b6cea000 lustre 192.168.201.126@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 23.837129] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.014620] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 24.045547] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 24.154094] AVX version of gcm_enc/dec engaged. [ 24.163477] AES CTR mode by8 optimization enabled [ 24.328666] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 24.342887] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 25.183256] EDAC MC: Ver: 3.0.0 [ 25.208560] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 25.208573] EDAC sbridge: Ver: 1.1.2 [ 31.271169] mount.nfs (777) used greatest stack depth: 10704 bytes left [ 53.180226] libcfs: loading out-of-tree module taints kernel. [ 53.200676] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 53.556781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 53.566007] alg: No test for adler32 (adler32-zlib) [ 54.597909] Lustre: Lustre: Build Version: 2.16.54_66_g664720d [ 54.898650] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 54.902064] LNet: Accept secure, port 988 [ 56.493737] Key type lgssc registered [ 57.150380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.572677] Lustre: Mounted lustre-client [ 124.976667] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 134.881518] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing check_logdir /tmp/testlogs/ [ 136.564111] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing yml_node [ 138.908760] Lustre: DEBUG MARKER: Client: 2.16.54.66 [ 140.193997] Lustre: DEBUG MARKER: MDS: 2.16.54.66 [ 141.522681] Lustre: DEBUG MARKER: OSS: 2.16.54.66 [ 142.471571] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 1 15:03:59 EDT 2025 [ 147.584371] Lustre: lustre-OST0000-osc-ffff8800b3b5c000: disconnect after 23s idle [ 150.439635] Lustre: DEBUG MARKER: excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a [ 151.077093] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 161.110996] LustreError: 7676:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 161.124006] Lustre: Unmounted lustre-client [ 194.129605] random: crng init done [ 203.516633] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 215.844602] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 232.474260] Lustre: DEBUG MARKER: == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:05:29 (1746126329) [ 276.955456] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 288.931823] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 289.339356] Lustre: Mounted lustre-client ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 12.81s (real) 6.70s (CPU), Child processes: 5.86s