************************ crashinfo ************************* /exports/testreports/51289/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg155-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.x71jl/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51289/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg155-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 21:21:03 EDT 2025 UPTIME: 01:27:24 LOAD AVERAGE: 0.63, 0.59, 0.65 TASKS: 151 NODENAME: oleg155-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 29 last 5s 52 last 60s 53 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 147 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 9 rcu_sched 0 ms (no user stack) 19142 kworker/0:1 0 ms (no user stack) 19505 kworker/1:0 2 ms (no user stack) 27 rcuos/2 3 ms (no user stack) 1959 socknal_sd00_00 5 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 785188 3 GB 82% of TOTAL MEM USED 169891 663.6 MB 17% of TOTAL MEM SHARED 10462 40.9 MB 1% of TOTAL MEM BUFFERS 5232 20.4 MB 0% of TOTAL MEM CACHED 46408 181.3 MB 4% of TOTAL MEM SLAB 83604 326.6 MB 8% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 64142 250.6 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 TIME_WAIT 4 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5242.0 eth0 n/a 0.0 RSS_TOTAL=83600 pages, %mem= 1.4 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aab0000 ffff88012aab8000 sysfs sysfs /sys ffff88012aab01c0 ffff880139944000 proc proc /proc ffff88012aab0380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aab0540 ffff8800b6cd2000 securityfs securityfs /sys/kernel/security ffff88012aab0700 ffff88012aab8800 tmpfs tmpfs /dev/shm ffff88012aab08c0 ffff88013710f800 devpts devpts /dev/pts ffff88012aab0a80 ffff88012aab9000 tmpfs tmpfs /run ffff88012aab0c40 ffff88012aab9800 tmpfs tmpfs /sys/fs/cgroup ffff88012aab0e00 ffff88012aaba000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aab0fc0 ffff88012aaba800 pstore pstore /sys/fs/pstore ffff88012a9ca000 ffff88012aac9800 cgroup cgroup /sys/fs/cgroup/devices ffff88012a9ca1c0 ffff88012aac9000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012a9ca380 ffff88012aac8800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a9ca540 ffff88012aac8000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a9ca700 ffff88012aaca000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a9ca8c0 ffff88012aaca800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aab1180 ffff88012aabb000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aab1340 ffff88012aabb800 cgroup cgroup /sys/fs/cgroup/memory ffff88012aab1500 ffff88012aabc000 cgroup cgroup /sys/fs/cgroup/pids ffff88012aab16c0 ffff88012aabc800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a9caa80 ffff88012aace800 configfs configfs /sys/kernel/config ffff8800b636ae00 ffff8800b63bc800 ext4 /dev/nbd0 / ffff880137668540 ffff88012aabf800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012a9cac40 ffff88012a552000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880138ccbc00 ffff8800b6cd3800 hugetlbfs hugetlbfs /dev/hugepages ffff8800b636b500 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8800b636b6c0 ffff88012b241000 mqueue mqueue /dev/mqueue ffff880137668700 ffff8800b6c85800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b636ba40 ffff8800b6f1a000 ramfs none /mnt ffff8801376688c0 ffff8800b63bb000 tmpfs none /var/lib/stateless/writable ffff88012a9cae00 ffff8800b5bd5000 squashfs /dev/vda /home/green/git/lustre-release ffff880137668a80 ffff8800b63bb000 tmpfs none /var/cache/man ffff880137668c40 ffff8800b63bb000 tmpfs none /var/log ffff880137668e00 ffff8800b63bb000 tmpfs none /var/lib/dbus ffff880137668fc0 ffff8800b63bb000 tmpfs none /tmp ffff880137669180 ffff8800b63bb000 tmpfs none /var/lib/dhclient ffff880137669340 ffff8800b63bb000 tmpfs none /var/tmp ffff880137669500 ffff8800b63bb000 tmpfs none /var/lib/NetworkManager ffff8801376696c0 ffff8800b63bb000 tmpfs none /var/lib/systemd/random-seed ffff880137669880 ffff8800b63bb000 tmpfs none /var/spool ffff8800b636bdc0 ffff8800b63bb000 tmpfs none /var/lib/nfs ffff880137669a40 ffff8800b63bb000 tmpfs none /var/lib/gssproxy ffff880137669c00 ffff8800b63bb000 tmpfs none /var/lib/logrotate ffff880137669dc0 ffff8800b63bb000 tmpfs none /etc ffff88012aab1880 ffff8800b63bb000 tmpfs none /var/lib/rsyslog ffff8800b5be0000 ffff8800b63bb000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b636b340 ffff88012a550000 nfs4 192.168.200.253:/exports/state/oleg155-client.virtnet /var/lib/stateless/state ffff88012abee1c0 ffff88012a550000 nfs4 192.168.200.253:/exports/state/oleg155-client.virtnet /boot ffff88012abee380 ffff88012a550000 nfs4 192.168.200.253:/exports/state/oleg155-client.virtnet /etc/etc/kdump.conf ffff88012abee540 ffff88012aabf800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b5be0a80 ffff8800b35d2800 nfs4 192.168.200.253://exports/testreports/51289/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800b5be0fc0 ffff8800b35d3000 tmpfs tmpfs /run/user/0 ffff8800b5be1340 ffff8800b5bd5000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b4300e00 ffff8800a9584000 lustre 192.168.201.155@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 9.285880] AVX version of gcm_enc/dec engaged. [ 9.288082] AES CTR mode by8 optimization enabled [ 9.331730] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.338764] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 9.351996] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 9.356426] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 9.470836] EDAC MC: Ver: 3.0.0 [ 9.477146] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.477157] EDAC sbridge: Ver: 1.1.2 [ 11.593412] mount.nfs (765) used greatest stack depth: 10704 bytes left [ 38.890725] libcfs: loading out-of-tree module taints kernel. [ 38.901992] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 39.192431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 39.203381] alg: No test for adler32 (adler32-zlib) [ 40.288797] Lustre: Lustre: Build Version: 2.16.54_73_ga2fdbc1 [ 40.663152] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 40.666677] LNet: Accept secure, port 988 [ 42.291344] Key type lgssc registered [ 43.539993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.874918] Lustre: Mounted lustre-client [ 116.990669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 135.179446] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing check_logdir /tmp/testlogs/ [ 137.468518] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing yml_node [ 138.893939] Lustre: lustre-OST0000-osc-ffff8800b6f18000: disconnect after 24s idle [ 140.540014] Lustre: DEBUG MARKER: Client: 2.16.54.73 [ 142.275214] Lustre: DEBUG MARKER: MDS: 2.16.54.73 [ 144.061588] Lustre: DEBUG MARKER: OSS: 2.16.54.73 [ 145.074712] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 1 19:56:02 EDT 2025 [ 156.253929] Lustre: DEBUG MARKER: excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a [ 157.298981] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 167.331423] LustreError: 7726:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 167.360534] Lustre: Unmounted lustre-client [ 180.428641] random: crng init done [ 216.020158] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 227.427815] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 238.749317] Lustre: DEBUG MARKER: == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:57:36 (1746143856) [ 275.572706] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 287.566120] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 288.174298] Lustre: Mounted lustre-client [ 376.746033] hrtimer: interrupt took 3003978 ns ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.98s (real) 5.66s (CPU), Child processes: 5.24s