************************ crashinfo ************************* /exports/testreports/51281/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg147-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.u9fMb/vmlinux [TAINTED] DUMPFILE: /exports/testreports/51281/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg147-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Thu May 1 18:49:05 EDT 2025 UPTIME: 01:27:44 LOAD AVERAGE: 0.73, 0.74, 0.74 TASKS: 152 NODENAME: oleg147-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 ioread8+0x42, 449 bytes of data #0 vp_interrupt+0x1e #1 __handle_irq_event_percpu+0x4e #2 handle_irq_event_percpu+0x32 #3 handle_irq_event+0x3d #4 handle_fasteoi_irq+0x5d #5 handle_irq+0xde #6 do_IRQ+0x4d, 19 bytes of data #7 ret_from_intr+0x0 #-1 native_safe_halt+0xb, 477 bytes of data #8 default_idle+0x1e #9 default_enter_idle+0x45 #10 cpuidle_enter_state+0x40 #11 cpuidle_idle_call+0xd8 #12 arch_cpu_idle+0xe #13 cpu_startup_entry+0x14a #14 rest_init+0x8e #15 start_kernel+0x456 #16 x86_64_start_reservations+0x2a #17 x86_64_start_kernel+0x152 #18 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 31 last 5s 39 last 60s 45 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 148 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 19393 createmany 0 ms /home/green/git/lustre-release/lustre/tests/createmany -o /mnt/lustre/d106.conf-sanity/f- 64768 1968 socknal_sd00_00 0 ms (no user stack) 19278 kworker/0:1 0 ms (no user stack) 18528 kworker/1:0 1 ms (no user stack) 1969 socknal_sd00_01 1 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 835157 3.2 GB 87% of TOTAL MEM USED 119922 468.4 MB 12% of TOTAL MEM SHARED 10460 40.9 MB 1% of TOTAL MEM BUFFERS 5233 20.4 MB 0% of TOTAL MEM CACHED 46406 181.3 MB 4% of TOTAL MEM SLAB 35527 138.8 MB 3% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 64119 250.5 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5262.1 eth0 n/a 0.0 RSS_TOTAL=79928 pages, %mem= 1.3 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aa361c0 ffff88012aab0800 sysfs sysfs /sys ffff88012aa36380 ffff880139944000 proc proc /proc ffff88012aa36540 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aa36700 ffff88012aab0000 securityfs securityfs /sys/kernel/security ffff88012aa368c0 ffff88012aab1000 tmpfs tmpfs /dev/shm ffff88012aa36a80 ffff880136bc1000 devpts devpts /dev/pts ffff88012aa36c40 ffff88012aab1800 tmpfs tmpfs /run ffff88012aa36e00 ffff88012aab2000 tmpfs tmpfs /sys/fs/cgroup ffff88012aa36fc0 ffff88012aab2800 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aa37180 ffff88012aab3000 pstore pstore /sys/fs/pstore ffff880138ccaa80 ffff88012aa05000 cgroup cgroup /sys/fs/cgroup/cpuset ffff880138ccac40 ffff88012aa05800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff880138ccae00 ffff88012aa06000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff880138ccafc0 ffff88012aa06800 cgroup cgroup /sys/fs/cgroup/devices ffff880138ccb180 ffff88012aa07000 cgroup cgroup /sys/fs/cgroup/freezer ffff880138ccb340 ffff88012aa07800 cgroup cgroup /sys/fs/cgroup/blkio ffff880138ccb500 ffff88012aaf0000 cgroup cgroup /sys/fs/cgroup/perf_event ffff880138ccb6c0 ffff88012aaf0800 cgroup cgroup /sys/fs/cgroup/pids ffff880138ccb880 ffff88012aaf1000 cgroup cgroup /sys/fs/cgroup/memory ffff880138ccba40 ffff88012aaf1800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff880137668e00 ffff8800b6ceb000 configfs configfs /sys/kernel/config ffff8800b58741c0 ffff8800b6ce8800 ext4 /dev/nbd0 / ffff88012b26c700 ffff88012aab7800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8800b5874380 ffff88012e156000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880138ccbdc0 ffff8800b6cec000 hugetlbfs hugetlbfs /dev/hugepages ffff8800b5874700 ffff880137128800 mqueue mqueue /dev/mqueue ffff8800b5874540 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff88012aa37340 ffff880000032000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012aa37500 ffff880000034000 ramfs none /mnt ffff88012b26ce00 ffff8800b59bf800 tmpfs none /var/lib/stateless/writable ffff880137668a80 ffff8800b6ceb800 squashfs /dev/vda /home/green/git/lustre-release ffff8800b5874c40 ffff8800b59bf800 tmpfs none /var/cache/man ffff8800b5874e00 ffff8800b59bf800 tmpfs none /var/log ffff8800b5874fc0 ffff8800b59bf800 tmpfs none /var/lib/dbus ffff8800b5875180 ffff8800b59bf800 tmpfs none /tmp ffff8800b5875340 ffff8800b59bf800 tmpfs none /var/lib/dhclient ffff8800b5875500 ffff8800b59bf800 tmpfs none /var/tmp ffff8800b58756c0 ffff8800b59bf800 tmpfs none /var/lib/NetworkManager ffff88012aa376c0 ffff8800b59bf800 tmpfs none /var/lib/systemd/random-seed ffff8800b5875880 ffff8800b59bf800 tmpfs none /var/spool ffff8800b5875a40 ffff8800b59bf800 tmpfs none /var/lib/nfs ffff88012aa37880 ffff8800b59bf800 tmpfs none /var/lib/gssproxy ffff8800b34a6000 ffff8800b59bf800 tmpfs none /var/lib/logrotate ffff8800b5875c00 ffff8800b59bf800 tmpfs none /etc ffff8800b5875dc0 ffff8800b59bf800 tmpfs none /var/lib/rsyslog ffff8800b3bf6000 ffff8800b59bf800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012aa37a40 ffff8800b59a7800 nfs4 192.168.200.253:/exports/state/oleg147-client.virtnet /var/lib/stateless/state ffff88012aa37dc0 ffff8800b59a7800 nfs4 192.168.200.253:/exports/state/oleg147-client.virtnet /boot ffff88012b26cfc0 ffff8800b59a7800 nfs4 192.168.200.253:/exports/state/oleg147-client.virtnet /etc/etc/kdump.conf ffff8800b3bf6540 ffff88012aab7800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b38d9a40 ffff8801373a5000 nfs4 192.168.200.253://exports/testreports/51281/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800b34a7880 ffff88012ab8d000 tmpfs tmpfs /run/user/0 ffff88012b26d500 ffff8800b6ceb800 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b34a6e00 ffff8800aa1f2800 lustre 192.168.201.147@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 9.601641] AVX version of gcm_enc/dec engaged. [ 9.603658] AES CTR mode by8 optimization enabled [ 9.662514] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 9.672571] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.692302] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 9.696627] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 9.808905] EDAC MC: Ver: 3.0.0 [ 9.815298] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.815312] EDAC sbridge: Ver: 1.1.2 [ 11.661844] mount.nfs (769) used greatest stack depth: 10704 bytes left [ 40.036199] libcfs: loading out-of-tree module taints kernel. [ 40.063795] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 40.605930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.625570] alg: No test for adler32 (adler32-zlib) [ 41.765078] Lustre: Lustre: Build Version: 2.16.54_81_g388e29d [ 42.295571] LNet: Added LNI 192.168.201.47@tcp [8/256/0/180] [ 42.299076] LNet: Accept secure, port 988 [ 43.947850] Key type lgssc registered [ 45.113348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.754506] hrtimer: interrupt took 3845236 ns [ 131.807730] Lustre: Mounted lustre-client [ 135.251866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 148.797922] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing check_logdir /tmp/testlogs/ [ 151.093701] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing yml_node [ 154.198450] Lustre: DEBUG MARKER: Client: 2.16.54.81 [ 155.873301] Lustre: DEBUG MARKER: MDS: 2.16.54.81 [ 156.800965] Lustre: lustre-OST0000-osc-ffff880000034800: disconnect after 23s idle [ 157.701944] Lustre: DEBUG MARKER: OSS: 2.16.54.81 [ 158.756037] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 1 17:23:57 EDT 2025 [ 171.034328] Lustre: DEBUG MARKER: excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a [ 172.347013] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 182.375840] LustreError: 7714:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 182.388870] Lustre: Unmounted lustre-client [ 182.824799] random: crng init done [ 235.406146] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 247.324763] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 259.438832] Lustre: DEBUG MARKER: == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:25:38 (1746134738) [ 300.045318] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 311.681364] Lustre: DEBUG MARKER: oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 311.995882] Lustre: Mounted lustre-client ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.53s (real) 5.40s (CPU), Child processes: 5.05s