************************ crashinfo ************************* /exports/testreports/51274/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg117-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.EMxYl/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51274/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg117-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 16:28:52 EDT 2025 UPTIME: 01:27:23 LOAD AVERAGE: 0.77, 0.71, 0.69 TASKS: 152 NODENAME: oleg117-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 ioread8+0x42, 449 bytes of data #0 vp_interrupt+0x1e #1 __handle_irq_event_percpu+0x4e #2 handle_irq_event_percpu+0x32 #3 handle_irq_event+0x3d #4 handle_fasteoi_irq+0x5d #5 handle_irq+0xde #6 do_IRQ+0x4d, 19 bytes of data #7 ret_from_intr+0x0 #-1 poll_idle+0x92, 477 bytes of data #8 poll_idle+0x7b #9 cpuidle_enter_state+0x40 #10 cpuidle_idle_call+0xd8 #11 arch_cpu_idle+0xe #12 cpu_startup_entry+0x14a #13 rest_init+0x8e #14 start_kernel+0x456 #15 x86_64_start_reservations+0x2a #16 x86_64_start_kernel+0x152 #17 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 24 last 5s 38 last 60s 60 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 148 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 20294 kworker/0:0 0 ms (no user stack) 19521 kworker/1:0 0 ms (no user stack) 1986 socknal_sd00_01 6 ms (no user stack) 1985 socknal_sd00_00 6 ms (no user stack) 27 rcuos/2 8 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 855550 3.3 GB 89% of TOTAL MEM USED 99529 388.8 MB 10% of TOTAL MEM SHARED 10463 40.9 MB 1% of TOTAL MEM BUFFERS 5234 20.4 MB 0% of TOTAL MEM CACHED 46413 181.3 MB 4% of TOTAL MEM SLAB 13007 50.8 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 64139 250.5 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 TIME_WAIT 13 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5239.3 eth0 n/a 0.0 RSS_TOTAL=83300 pages, %mem= 1.3 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff880138ccb180 ffff88013771f800 sysfs sysfs /sys ffff880138ccb340 ffff880139944000 proc proc /proc ffff880138ccb500 ffff880137678000 devtmpfs devtmpfs /dev ffff880138ccb6c0 ffff88012b2f6000 securityfs securityfs /sys/kernel/security ffff880138ccb880 ffff88012aaa0000 tmpfs tmpfs /dev/shm ffff880138ccba40 ffff880137324000 devpts devpts /dev/pts ffff880138ccbc00 ffff88012aaa0800 tmpfs tmpfs /run ffff880138ccbdc0 ffff88012aaa1000 tmpfs tmpfs /sys/fs/cgroup ffff88012ab02000 ffff88012aaa1800 cgroup cgroup /sys/fs/cgroup/systemd ffff88012ab021c0 ffff88012aaa2000 pstore pstore /sys/fs/pstore ffff880137668a80 ffff8800b6d14000 cgroup cgroup /sys/fs/cgroup/pids ffff880137668c40 ffff8800b6d14800 cgroup cgroup /sys/fs/cgroup/freezer ffff880137668e00 ffff8800b6d15000 cgroup cgroup /sys/fs/cgroup/perf_event ffff880137668fc0 ffff8800b6d15800 cgroup cgroup /sys/fs/cgroup/memory ffff880137669180 ffff8800b6d16000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012ab02380 ffff88012aaa4000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012ab02540 ffff88012aaa3800 cgroup cgroup /sys/fs/cgroup/devices ffff88012ab02700 ffff88012aaa3000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012ab028c0 ffff88012aaa2800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012ab02a80 ffff88012aaa4800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012ab02c40 ffff88012aaa5000 configfs configfs /sys/kernel/config ffff88012aab5500 ffff88012a458800 ext4 /dev/nbd0 / ffff8801376696c0 ffff8800b6e95800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8801377f6700 ffff88012b2f2000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8801377f68c0 ffff88013771a800 mqueue mqueue /dev/mqueue ffff880137669880 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8801377f6a80 ffff88012b2f7800 hugetlbfs hugetlbfs /dev/hugepages ffff880137669a40 ffff8800b6ec6800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8801377f6c40 ffff8800b6e95000 ramfs none /mnt ffff88012aab5880 ffff8800b6351000 tmpfs none /var/lib/stateless/writable ffff880137669dc0 ffff8800b6ec6000 squashfs /dev/vda /home/green/git/lustre-release ffff88012ab02e00 ffff8800b6351000 tmpfs none /var/cache/man ffff8801377f6e00 ffff8800b6351000 tmpfs none /var/log ffff8801377f6fc0 ffff8800b6351000 tmpfs none /var/lib/dbus ffff88012aab5c00 ffff8800b6351000 tmpfs none /tmp ffff8801377f7180 ffff8800b6351000 tmpfs none /var/lib/dhclient ffff8801377f7340 ffff8800b6351000 tmpfs none /var/tmp ffff88012aab5dc0 ffff8800b6351000 tmpfs none /var/lib/NetworkManager ffff8800b5cd4000 ffff8800b6351000 tmpfs none /var/lib/systemd/random-seed ffff88012ab02fc0 ffff8800b6351000 tmpfs none /var/spool ffff8801377f7500 ffff8800b6351000 tmpfs none /var/lib/nfs ffff8800b5cd41c0 ffff8800b6351000 tmpfs none /var/lib/gssproxy ffff8801377f76c0 ffff8800b6351000 tmpfs none /var/lib/logrotate ffff8801377f7880 ffff8800b6351000 tmpfs none /etc ffff8801376688c0 ffff8800b6351000 tmpfs none /var/lib/rsyslog ffff8801377f7a40 ffff8800b6351000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b5cd4380 ffff8800b5e2f000 nfs4 192.168.200.253:/exports/state/oleg117-client.virtnet /var/lib/stateless/state ffff8800b5cd4a80 ffff8800b5e2f000 nfs4 192.168.200.253:/exports/state/oleg117-client.virtnet /boot ffff8801377f7c00 ffff8800b5e2f000 nfs4 192.168.200.253:/exports/state/oleg117-client.virtnet /etc/etc/kdump.conf ffff880137669500 ffff8800b6e95800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b35401c0 ffff8800b6394800 nfs4 192.168.200.253://exports/testreports/51274/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff88012ab03500 ffff8800b5db5000 tmpfs tmpfs /run/user/0 ffff88012e34ce00 ffff8800b6ec6000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b5e3d340 ffff8800b6395000 lustre 192.168.201.117@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 17.189658] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 17.462899] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.550345] sed (677) used greatest stack depth: 11856 bytes left [ 17.913790] AVX version of gcm_enc/dec engaged. [ 17.915243] AES CTR mode by8 optimization enabled [ 17.957808] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 17.964224] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 18.621972] EDAC MC: Ver: 3.0.0 [ 18.646183] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 18.646193] EDAC sbridge: Ver: 1.1.2 [ 28.669081] mount.nfs (781) used greatest stack depth: 10448 bytes left [ 59.819787] libcfs: loading out-of-tree module taints kernel. [ 59.822382] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 60.071635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 60.077632] alg: No test for adler32 (adler32-zlib) [ 61.133663] Lustre: Lustre: Build Version: 2.16.54_67_gd9b168e [ 61.513711] LNet: Added LNI 192.168.201.17@tcp [8/256/0/180] [ 61.517202] LNet: Accept secure, port 988 [ 63.156863] Key type lgssc registered [ 63.873512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.312513] Lustre: Mounted lustre-client [ 132.087755] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 142.736876] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing check_logdir /tmp/testlogs/ [ 143.885104] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing yml_node [ 145.761561] Lustre: DEBUG MARKER: Client: 2.16.54.67 [ 146.910395] Lustre: DEBUG MARKER: MDS: 2.16.54.67 [ 148.073964] Lustre: DEBUG MARKER: OSS: 2.16.54.67 [ 148.900841] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 1 15:03:56 EDT 2025 [ 154.342095] Lustre: lustre-OST0000-osc-ffff88012a45e000: disconnect after 24s idle [ 157.747045] Lustre: DEBUG MARKER: excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a [ 158.328930] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 168.356425] LustreError: 7701:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 168.376984] Lustre: Unmounted lustre-client [ 202.236905] random: crng init done [ 214.064467] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 225.726291] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 237.617354] Lustre: DEBUG MARKER: == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:05:24 (1746126324) [ 281.597781] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 293.382519] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 293.715083] Lustre: Mounted lustre-client ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 12.18s (real) 6.18s (CPU), Child processes: 5.92s