************************ crashinfo ************************* /exports/testreports/44870/testresults/conf-sanity3-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg446-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.LNV7k/vmlinux [TAINTED] DUMPFILE: /exports/testreports/44870/testresults/conf-sanity3-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg446-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Fri Aug 9 23:13:06 EDT 2024 UPTIME: 01:57:01 LOAD AVERAGE: 0.53, 0.47, 0.33 TASKS: 143 NODENAME: oleg446-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 26 last 5s 37 last 60s 72 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 139 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 11 rcuos/0 0 ms (no user stack) 9 rcu_sched 0 ms (no user stack) 16929 ssh 3 ms ssh -oConnectTimeout=10 -2 -a -x oleg446-server (PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin; cd /root; LUSTRE="/home/green/git/lustre-release/lustre" MODOPTS_PTLRPC="lbug_on_grant_miscount=1" mds1_FSTYPE=ldiskfs ost1_FSTYPE=ldiskfs VERBOSE=false FSTYPE=ldiskfs LNETLND=socklnd/ksocklnd NETTYPE=tcp bash -c "PATH=/home/green/git/lustre-release/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/sbin:/bin:/usr/sbin: NAME=ncli TESTLOG_PREFIX=/tmp/testlogs//conf-sanity TESTNAME=test_153b CONFIG=/home/green/git/lustre-release/lustre/tests/cfg/ncli.sh bash rpc.sh set_default_debug \"-1\" \"all\" ") 16928 pdsh 10 ms pdsh -S -Rssh -w oleg446-server (PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin; cd /root; LUSTRE="/home/green/git/lustre-release/lustre" MODOPTS_PTLRPC="lbug_on_grant_miscount=1" mds1_FSTYPE=ldiskfs ost1_FSTYPE=ldiskfs VERBOSE=false FSTYPE=ldiskfs LNETLND=socklnd/ksocklnd NETTYPE=tcp bash -c "PATH=/home/green/git/lustre-release/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/sbin:/bin:/usr/sbin: NAME=ncli TESTLOG_PREFIX=/tmp/testlogs//conf-sanity TESTNAME=test_153b CONFIG=/home/green/git/lustre-release/lustre/tests/cfg/ncli.sh bash rpc.sh set_default_debug \"-1\" \"all\" ") 16926 pdsh 10 ms pdsh -S -Rssh -w oleg446-server (PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin; cd /root; LUSTRE="/home/green/git/lustre-release/lustre" MODOPTS_PTLRPC="lbug_on_grant_miscount=1" mds1_FSTYPE=ldiskfs ost1_FSTYPE=ldiskfs VERBOSE=false FSTYPE=ldiskfs LNETLND=socklnd/ksocklnd NETTYPE=tcp bash -c "PATH=/home/green/git/lustre-release/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/sbin:/bin:/usr/sbin: NAME=ncli TESTLOG_PREFIX=/tmp/testlogs//conf-sanity TESTNAME=test_153b CONFIG=/home/green/git/lustre-release/lustre/tests/cfg/ncli.sh bash rpc.sh set_default_debug \"-1\" \"all\" ") +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 857480 3.3 GB 89% of TOTAL MEM USED 97599 381.2 MB 10% of TOTAL MEM SHARED 10712 41.8 MB 1% of TOTAL MEM BUFFERS 5451 21.3 MB 0% of TOTAL MEM CACHED 50330 196.6 MB 5% of TOTAL MEM SLAB 11315 44.2 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 66966 261.6 MB 9% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 4 TIME_WAIT 98 LISTEN 3 NAGLE disabled (TCP_NODELAY): 3 user_data set (NFS etc.): 1 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 30 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 7019.5 eth0 n/a 0.0 RSS_TOTAL=86360 pages, %mem= 1.5 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aa641c0 ffff88012aac0800 sysfs sysfs /sys ffff88012aa64380 ffff880139944000 proc proc /proc ffff88012aa64540 ffff880137678000 devtmpfs devtmpfs /dev ffff88012aa64700 ffff88012aac0000 securityfs securityfs /sys/kernel/security ffff88012aa648c0 ffff88012aac1000 tmpfs tmpfs /dev/shm ffff88012aa64a80 ffff880137159800 devpts devpts /dev/pts ffff88012aa64c40 ffff88012aac1800 tmpfs tmpfs /run ffff88012aa64e00 ffff88012aac2000 tmpfs tmpfs /sys/fs/cgroup ffff88012aa64fc0 ffff88012aac2800 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aa65180 ffff88012aac3000 pstore pstore /sys/fs/pstore ffff88012aa65340 ffff88012aac5000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aa65500 ffff88012aac4800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aa656c0 ffff88012aac4000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aa65880 ffff88012aac3800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aa65a40 ffff88012aac5800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aa65c00 ffff88012aac6000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aa65dc0 ffff88012aac6800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012ab18000 ffff88012aac7000 cgroup cgroup /sys/fs/cgroup/memory ffff88012ab181c0 ffff88012aac7800 cgroup cgroup /sys/fs/cgroup/devices ffff88012ab18380 ffff88012ab20000 cgroup cgroup /sys/fs/cgroup/pids ffff880138ccba40 ffff8800b6ef7000 configfs configfs /sys/kernel/config ffff88012a6ca000 ffff88012a456000 ext4 /dev/nbd0 / ffff8800b6ffcc40 ffff88012ab21800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012ab18c40 ffff88012a5f8000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8800b6ffce00 ffff88013715b000 mqueue mqueue /dev/mqueue ffff8801376688c0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff880137668a80 ffff88012a45c000 hugetlbfs hugetlbfs /dev/hugepages ffff88012a6ca1c0 ffff8800b6ef1000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b6ffcfc0 ffff88012ab22000 ramfs none /mnt ffff88012a6ca380 ffff8800b581c000 tmpfs none /var/lib/stateless/writable ffff880137668c40 ffff88013767c800 squashfs /dev/vda /home/green/git/lustre-release ffff880137668e00 ffff8800b581c000 tmpfs none /var/cache/man ffff880137668fc0 ffff8800b581c000 tmpfs none /var/log ffff880137669180 ffff8800b581c000 tmpfs none /var/lib/dbus ffff88012ab18fc0 ffff8800b581c000 tmpfs none /tmp ffff8800b6ffd340 ffff8800b581c000 tmpfs none /var/lib/dhclient ffff88012ab19180 ffff8800b581c000 tmpfs none /var/tmp ffff880137669340 ffff8800b581c000 tmpfs none /var/lib/NetworkManager ffff880137669500 ffff8800b581c000 tmpfs none /var/lib/systemd/random-seed ffff88012ab19340 ffff8800b581c000 tmpfs none /var/spool ffff88012a6ca540 ffff8800b581c000 tmpfs none /var/lib/nfs ffff88012ab19500 ffff8800b581c000 tmpfs none /var/lib/gssproxy ffff8800b6ffd500 ffff8800b581c000 tmpfs none /var/lib/logrotate ffff88012a6ca700 ffff8800b581c000 tmpfs none /etc ffff88012ab196c0 ffff8800b581c000 tmpfs none /var/lib/rsyslog ffff88012ab19880 ffff8800b581c000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012ab19a40 ffff88012a5fc000 nfs4 192.168.200.253:/exports/state/oleg446-client.virtnet /var/lib/stateless/state ffff88012ab18700 ffff88012a5fc000 nfs4 192.168.200.253:/exports/state/oleg446-client.virtnet /boot ffff88012ab19dc0 ffff88012a5fc000 nfs4 192.168.200.253:/exports/state/oleg446-client.virtnet /etc/etc/kdump.conf ffff8801376696c0 ffff88012ab21800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff880137668540 ffff8800b3b6c800 nfs4 192.168.200.253://exports/testreports/44870/testresults/conf-sanity3-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800b3e101c0 ffff8800b3b6f800 tmpfs tmpfs /run/user/0 ffff8800b6ffda40 ffff88013767c800 squashfs /dev/vda /usr/sbin/mount.lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 6627.094619] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6666.773563] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6667.594466] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6673.074271] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6691.468421] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:07:34 (1723259254) [ 6710.545935] Lustre: Mounted lustre-client [ 6712.437137] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6725.080207] Lustre: DEBUG MARKER: ADD OST3 [ 6731.025828] Lustre: DEBUG MARKER: STOP OST3 [ 6733.702718] Lustre: DEBUG MARKER: START OST3 again [ 6735.984800] Lustre: lustre-OST0000-osc-ffff8800b6ef4000: disconnect after 24s idle [ 6735.988072] Lustre: lustre-OST0003-osc-ffff8800b6ef4000: Connection to lustre-OST0003 (at 192.168.204.146@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6736.653104] Lustre: lustre-OST0003-osc-ffff8800b6ef4000: Connection restored to (at 192.168.204.146@tcp) [ 6749.982948] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:08:33 (1723259313) [ 6750.125031] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 6750.129181] LustreError: 6406:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b6ef4000: inode [0x240000bd0:0x1:0x0] mdc close failed: rc = -108 [ 6760.167564] Lustre: Unmounted lustre-client [ 6795.110414] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6795.959909] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6801.578801] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6825.050408] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6825.823771] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6830.107942] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6830.618625] LNetError: 28870:0:(lib-move.c:2285:lnet_handle_find_routed_path()) peer 192.168.252.112@tcp2 has no available nets [ 6830.650547] Lustre: Mounted lustre-client [ 6835.757593] Lustre: Unmounted lustre-client [ 6851.447995] Key type lgssc unregistered [ 6882.513923] LNet: 11515:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6958.522914] LNet: Removed LNI 192.168.204.46@tcp [ 6962.747683] Lustre: DEBUG MARKER: == conf-sanity test 153b: added IPv6 NID support ========= 23:12:06 (1723259526) [ 6967.352237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6967.355657] alg: No test for adler32 (adler32-zlib) [ 6968.164026] Lustre: Lustre: Build Version: 2.15.64_147_gadac521 [ 6968.211878] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 6968.213322] LNet: Accept secure, port 988 [ 6969.742662] Key type lgssc registered [ 6969.947963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6992.183190] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6992.930351] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6999.056262] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.59s (real) 5.42s (CPU), Child processes: 5.14s