************************ crashinfo ************************* /exports/testreports/44259/testresults/sanity-pfl-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg221-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.ciTcH/vmlinux [TAINTED] DUMPFILE: /exports/testreports/44259/testresults/sanity-pfl-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg221-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Wed Jul 17 09:06:13 EDT 2024 UPTIME: 01:06:59 LOAD AVERAGE: 0.00, 0.01, 0.05 TASKS: 279 NODENAME: oleg221-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 27 last 5s 61 last 60s 86 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 275 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 19105 ldlm_bl_05 0 ms (no user stack) 25839 kworker/u8:0 0 ms (no user stack) 21294 ll_mgs_0001 0 ms (no user stack) 49 kworker/2:1 0 ms (no user stack) 3318 ptlrpcd_00_03 45 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 707392 2.7 GB 74% of TOTAL MEM USED 247675 967.5 MB 25% of TOTAL MEM SHARED 38456 150.2 MB 4% of TOTAL MEM BUFFERS 35633 139.2 MB 3% of TOTAL MEM CACHED 69313 270.8 MB 7% of TOTAL MEM SLAB 19127 74.7 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 60920 238 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 5 LISTEN 3 NAGLE disabled (TCP_NODELAY): 4 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 4016.5 eth0 n/a 0.2 RSS_TOTAL=52252 pages, %mem= 0.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012a360000 ffff88012a368000 sysfs sysfs /sys ffff88012a3601c0 ffff880139944000 proc proc /proc ffff88012a360380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012a360540 ffff8800b5221800 securityfs securityfs /sys/kernel/security ffff88012a360700 ffff88012a368800 tmpfs tmpfs /dev/shm ffff88012a3608c0 ffff880137167000 devpts devpts /dev/pts ffff88012a360a80 ffff88012a369000 tmpfs tmpfs /run ffff88012a360c40 ffff88012a369800 tmpfs tmpfs /sys/fs/cgroup ffff88012a360e00 ffff88012a36a000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012a360fc0 ffff88012a36a800 pstore pstore /sys/fs/pstore ffff88012a361180 ffff88012a36c800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a361340 ffff88012a36c000 cgroup cgroup /sys/fs/cgroup/memory ffff88012a361500 ffff88012a36b800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a3616c0 ffff88012a36b000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a361880 ffff88012a36d000 cgroup cgroup /sys/fs/cgroup/devices ffff88012a361a40 ffff88012a36d800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a361c00 ffff88012a36e000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a361dc0 ffff88012a36e800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a32c000 ffff88012a36f000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012a32c1c0 ffff88012a36f800 cgroup cgroup /sys/fs/cgroup/pids ffff88012b25aa80 ffff88012b3c6800 configfs configfs /sys/kernel/config ffff88012b25afc0 ffff8800b4dae800 ext4 /dev/nbd0 / ffff88012a32c540 ffff880129c24800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012b25b180 ffff8800b4d11000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff88012b25b340 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff880137668700 ffff88012b288800 mqueue mqueue /dev/mqueue ffff88012a32cc40 ffff880129c68800 hugetlbfs hugetlbfs /dev/hugepages ffff88012a32ce00 ffff880129c6a800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8801376688c0 ffff8800b4dac000 ramfs none /mnt ffff88012a32d180 ffff8800b416a000 squashfs /dev/vda /home/green/git/lustre-release ffff88012b25b500 ffff880129c22800 tmpfs none /var/lib/stateless/writable ffff880137668a80 ffff880129c22800 tmpfs none /var/cache/man ffff88012a32d340 ffff880129c22800 tmpfs none /var/log ffff88012a32d500 ffff880129c22800 tmpfs none /var/lib/dbus ffff88012a32d6c0 ffff880129c22800 tmpfs none /tmp ffff88012a32d880 ffff880129c22800 tmpfs none /var/lib/dhclient ffff880137668c40 ffff880129c22800 tmpfs none /var/tmp ffff88012a32da40 ffff880129c22800 tmpfs none /var/lib/NetworkManager ffff88012a32dc00 ffff880129c22800 tmpfs none /var/lib/systemd/random-seed ffff88012a32ddc0 ffff880129c22800 tmpfs none /var/spool ffff88012a32ca80 ffff880129c22800 tmpfs none /var/lib/nfs ffff8800b4c961c0 ffff880129c22800 tmpfs none /var/lib/gssproxy ffff880137668e00 ffff880129c22800 tmpfs none /var/lib/logrotate ffff8800b4c96380 ffff880129c22800 tmpfs none /etc ffff88012a32c8c0 ffff880129c22800 tmpfs none /var/lib/rsyslog ffff880137668fc0 ffff880129c22800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012a32c700 ffff880129c6d800 nfs4 192.168.200.253:/exports/state/oleg221-server.virtnet /var/lib/stateless/state ffff88012a32c380 ffff880129c6d800 nfs4 192.168.200.253:/exports/state/oleg221-server.virtnet /boot ffff88012b25ba40 ffff880129c6d800 nfs4 192.168.200.253:/exports/state/oleg221-server.virtnet /etc/etc/kdump.conf ffff88012c47c000 ffff880129c24800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b43b1880 ffff8800b416a000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b247e540 ffff88007be91000 lustre /dev/mapper/mds2_flakey /mnt/lustre-mds2 ffff8800b247e700 ffff8800894a7800 lustre /dev/mapper/ost1_flakey /mnt/lustre-ost1 ffff8800b247efc0 ffff88009c719000 lustre /dev/mapper/ost2_flakey /mnt/lustre-ost2 ffff88013329e1c0 ffff8800a6786800 lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 948.656687] Lustre: DEBUG MARKER: sanity-pfl test_15: @@@@@@ FAIL: got '' with stripe_count=2, not f5 [ 950.209438] Lustre: DEBUG MARKER: == sanity-pfl test 16a: Verify setstripe/getstripe with YAML config file ========================================================== 08:15:03 (1721218503) [ 1012.515579] LustreError: 6827:0:(lod_qos.c:1362:lod_ost_alloc_specific()) can't lstripe objid [0x200000402:0x9423:0x0]: have 1 want 2 [ 1012.950299] Lustre: DEBUG MARKER: sanity-pfl test_16a: @@@@@@ FAIL: setstripe /mnt/lustre/d16a.sanity-pfl/f16a.sanity-pfl.copy failed [ 1014.590503] Lustre: DEBUG MARKER: == sanity-pfl test 16b: Verify setstripe/getstripe with YAML config file + overstriping ========================================================== 08:16:07 (1721218567) [ 1137.112410] Lustre: mdt00_003: service thread pid 8051 was inactive for 120.298 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 1137.117334] Pid: 8051, comm: mdt00_003 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 1137.119821] Call Trace: [ 1137.120504] [<0>] osp_precreate_reserve+0x59f/0xaa0 [osp] [ 1137.122046] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [ 1137.123886] [<0>] lod_sub_declare_create+0xe6/0x280 [lod] [ 1137.125564] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [ 1137.127567] [<0>] lod_qos_prep_create+0xa67/0x1ab0 [lod] [ 1137.129252] [<0>] lod_prepare_create+0x23b/0x320 [lod] [ 1137.131187] [<0>] lod_declare_striped_create+0xf8/0xa50 [lod] [ 1137.132741] [<0>] lod_declare_xattr_set+0x139/0x20d0 [lod] [ 1137.134946] [<0>] mdd_create_data+0x462/0x720 [mdd] [ 1137.136424] [<0>] mdt_mfd_open+0xccc/0xee0 [mdt] [ 1137.138395] [<0>] mdt_finish_open+0x803/0x9e0 [mdt] [ 1137.140553] [<0>] mdt_open_by_fid_lock+0x55b/0xb90 [mdt] [ 1137.141913] [<0>] mdt_reint_open+0x778/0x2b30 [mdt] [ 1137.143158] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 1137.144423] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 1137.145713] [<0>] mdt_intent_open+0x93/0x480 [mdt] [ 1137.146923] [<0>] mdt_intent_opc+0x1dd/0xc10 [mdt] [ 1137.148077] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [ 1137.149314] [<0>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc] [ 1137.150702] [<0>] ldlm_handle_enqueue0+0x8c6/0x1780 [ptlrpc] [ 1137.152016] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [ 1137.153265] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 1137.154939] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 1137.156507] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 1137.157824] [<0>] kthread+0xe4/0xf0 [ 1137.159110] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 1137.160420] [<0>] 0xfffffffffffffffe [ 1611.864614] Lustre: 6827:0:(service.c:1437:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff88009bdf9c80 x1804827542951232/t0(0) o101->7cb31399-071b-4930-9127-f8a8aee99435@192.168.202.21@tcp:370/0 lens 1072/3848 e 21 to 0 dl 1721219170 ref 2 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 1618.836966] Lustre: lustre-MDT0000: Client 7cb31399-071b-4930-9127-f8a8aee99435 (at 192.168.202.21@tcp) reconnecting [ 2219.868927] Lustre: lustre-MDT0000: Client 7cb31399-071b-4930-9127-f8a8aee99435 (at 192.168.202.21@tcp) reconnecting [ 2820.889759] Lustre: lustre-MDT0000: Client 7cb31399-071b-4930-9127-f8a8aee99435 (at 192.168.202.21@tcp) reconnecting [ 3421.920514] Lustre: lustre-MDT0000: Client 7cb31399-071b-4930-9127-f8a8aee99435 (at 192.168.202.21@tcp) reconnecting ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.42s (real) 5.69s (CPU), Child processes: 4.69s