************************ crashinfo ************************* /exports/testreports/44316/testresults/sanity-pfl-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg251-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.prku8/vmlinux [TAINTED] DUMPFILE: /exports/testreports/44316/testresults/sanity-pfl-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg251-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Fri Jul 19 06:43:31 EDT 2024 UPTIME: 01:07:05 LOAD AVERAGE: 0.00, 0.01, 0.05 TASKS: 283 NODENAME: oleg251-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 16 last 5s 62 last 60s 79 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 279 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 4078 dbuf_evict 0 ms (no user stack) 20 rcuos/1 0 ms (no user stack) 9 rcu_sched 0 ms (no user stack) 1 systemd 5 ms /usr/lib/systemd/systemd --switched-root --system --deserialize 22 46 kworker/0:1 85 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 704804 2.7 GB 73% of TOTAL MEM USED 250263 977.6 MB 26% of TOTAL MEM SHARED 40975 160.1 MB 4% of TOTAL MEM BUFFERS 38138 149 MB 3% of TOTAL MEM CACHED 69050 269.7 MB 7% of TOTAL MEM SLAB 19206 75 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 60638 236.9 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 5 LISTEN 3 NAGLE disabled (TCP_NODELAY): 4 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 4023.0 eth0 n/a 1.4 RSS_TOTAL=52204 pages, %mem= 0.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012a2c4000 ffff88012a338000 sysfs sysfs /sys ffff88012a2c41c0 ffff880139944000 proc proc /proc ffff88012a2c4380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012a2c4540 ffff8800b51e2000 securityfs securityfs /sys/kernel/security ffff88012a2c4700 ffff88012a338800 tmpfs tmpfs /dev/shm ffff88012a2c48c0 ffff880137149800 devpts devpts /dev/pts ffff88012a2c4a80 ffff88012a339000 tmpfs tmpfs /run ffff88012a2c4c40 ffff88012a339800 tmpfs tmpfs /sys/fs/cgroup ffff88012a2c4e00 ffff88012a33a000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012a2c4fc0 ffff88012a33a800 pstore pstore /sys/fs/pstore ffff88012a2c5180 ffff88012a33c800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a2c5340 ffff88012a33c000 cgroup cgroup /sys/fs/cgroup/devices ffff88012a2c5500 ffff88012a33b800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a2c56c0 ffff88012a33b000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a2c5880 ffff88012a33d000 cgroup cgroup /sys/fs/cgroup/pids ffff88012a2c5a40 ffff88012a33d800 cgroup cgroup /sys/fs/cgroup/memory ffff88012a2c5c00 ffff88012a33e000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a2c5dc0 ffff88012a33e800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a280000 ffff88012a33f000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a2801c0 ffff88012a33f800 cgroup cgroup /sys/fs/cgroup/perf_event ffff880138ccba40 ffff8800b51e6800 configfs configfs /sys/kernel/config ffff88012a280540 ffff88012a3fd800 ext4 /dev/nbd0 / ffff8801376696c0 ffff88012a3fc000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff880137669880 ffff8800b51fd800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff8800b40a2700 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff880137669a40 ffff880129ccb800 hugetlbfs hugetlbfs /dev/hugepages ffff880137669c00 ffff880137679000 mqueue mqueue /dev/mqueue ffff880137669dc0 ffff880129cca800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800b40a28c0 ffff8801372e6800 ramfs none /mnt ffff8800b2580000 ffff88013689a000 tmpfs none /var/lib/stateless/writable ffff8800b25801c0 ffff88013689e800 squashfs /dev/vda /home/green/git/lustre-release ffff88012a280700 ffff88013689a000 tmpfs none /var/cache/man ffff8800b40a2a80 ffff88013689a000 tmpfs none /var/log ffff8800b2580540 ffff88013689a000 tmpfs none /var/lib/dbus ffff88012a2808c0 ffff88013689a000 tmpfs none /tmp ffff88012a280a80 ffff88013689a000 tmpfs none /var/lib/dhclient ffff8800b40a2c40 ffff88013689a000 tmpfs none /var/tmp ffff880138ccbc00 ffff88013689a000 tmpfs none /var/lib/NetworkManager ffff8800b2580700 ffff88013689a000 tmpfs none /var/lib/systemd/random-seed ffff8800b25808c0 ffff88013689a000 tmpfs none /var/spool ffff880138ccbdc0 ffff88013689a000 tmpfs none /var/lib/nfs ffff880129e2a000 ffff88013689a000 tmpfs none /var/lib/gssproxy ffff880129e2a1c0 ffff88013689a000 tmpfs none /var/lib/logrotate ffff8800b2580a80 ffff88013689a000 tmpfs none /etc ffff880129e2a380 ffff88013689a000 tmpfs none /var/lib/rsyslog ffff880129e2a540 ffff88013689a000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012a280c40 ffff88012a3f8800 nfs4 192.168.200.253:/exports/state/oleg251-server.virtnet /var/lib/stateless/state ffff8800b40a2e00 ffff88012a3f8800 nfs4 192.168.200.253:/exports/state/oleg251-server.virtnet /boot ffff8800b40a2fc0 ffff88012a3f8800 nfs4 192.168.200.253:/exports/state/oleg251-server.virtnet /etc/etc/kdump.conf ffff8800b40a3180 ffff88012a3fc000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b2580380 ffff88013689e800 squashfs /dev/vda /usr/sbin/mount.lustre ffff880129f55340 ffff88012e1fb000 lustre /dev/mapper/mds2_flakey /mnt/lustre-mds2 ffff880129f556c0 ffff88012e233800 lustre /dev/mapper/ost1_flakey /mnt/lustre-ost1 ffff880129f55180 ffff8800926bc000 lustre /dev/mapper/ost2_flakey /mnt/lustre-ost2 ffff8800990868c0 ffff8800923d1800 lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 916.481063] Lustre: DEBUG MARKER: == sanity-pfl test 15: Verify component options for lfs find ========================================================== 05:51:42 (1721382702) [ 918.272305] Lustre: DEBUG MARKER: == sanity-pfl test 16a: Verify setstripe/getstripe with YAML config file ========================================================== 05:51:43 (1721382703) [ 940.002839] Lustre: DEBUG MARKER: sanity-pfl test_16a: @@@@@@ FAIL: Create /mnt/lustre/d16a.sanity-pfl/f16a.sanity-pfl failed [ 941.572402] Lustre: DEBUG MARKER: == sanity-pfl test 16b: Verify setstripe/getstripe with YAML config file + overstriping ========================================================== 05:52:06 (1721382726) [ 1005.448386] Lustre: mdt00_002: service thread pid 6836 was inactive for 62.229 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 1005.456401] Pid: 6836, comm: mdt00_002 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 1005.459744] Call Trace: [ 1005.460972] [<0>] osp_precreate_reserve+0x59f/0xaa0 [osp] [ 1005.462145] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [ 1005.463390] [<0>] lod_sub_declare_create+0xe6/0x280 [lod] [ 1005.464661] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [ 1005.466036] [<0>] lod_qos_prep_create+0xa67/0x1ab0 [lod] [ 1005.467709] [<0>] lod_prepare_create+0x23b/0x320 [lod] [ 1005.469418] [<0>] lod_declare_striped_create+0xf8/0xa50 [lod] [ 1005.470701] [<0>] lod_declare_xattr_set+0x139/0x20d0 [lod] [ 1005.472902] [<0>] mdd_create_data+0x462/0x720 [mdd] [ 1005.474596] [<0>] mdt_mfd_open+0xccc/0xee0 [mdt] [ 1005.475748] [<0>] mdt_finish_open+0x803/0x9e0 [mdt] [ 1005.476928] [<0>] mdt_open_by_fid_lock+0x55b/0xb90 [mdt] [ 1005.478172] [<0>] mdt_reint_open+0x778/0x2b30 [mdt] [ 1005.479413] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 1005.480577] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 1005.481783] [<0>] mdt_intent_open+0x93/0x480 [mdt] [ 1005.482894] [<0>] mdt_intent_opc+0x1dd/0xc10 [mdt] [ 1005.484015] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [ 1005.485273] [<0>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc] [ 1005.486453] [<0>] ldlm_handle_enqueue0+0x8c6/0x1780 [ptlrpc] [ 1005.487772] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [ 1005.489080] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 1005.490496] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 1005.492044] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 1005.493174] [<0>] kthread+0xe4/0xf0 [ 1005.494063] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 1005.495163] [<0>] 0xfffffffffffffffe [ 1538.200332] Lustre: 19195:0:(service.c:1437:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff88011ff64280 x1804999766107648/t0(0) o101->15c0e9fc-736c-41f2-b052-1123923b1b93@192.168.202.51@tcp:693/0 lens 1072/3848 e 23 to 0 dl 1721383328 ref 2 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 1544.235749] Lustre: lustre-MDT0000: Client 15c0e9fc-736c-41f2-b052-1123923b1b93 (at 192.168.202.51@tcp) reconnecting [ 2145.258537] Lustre: lustre-MDT0000: Client 15c0e9fc-736c-41f2-b052-1123923b1b93 (at 192.168.202.51@tcp) reconnecting [ 2746.285093] Lustre: lustre-MDT0000: Client 15c0e9fc-736c-41f2-b052-1123923b1b93 (at 192.168.202.51@tcp) reconnecting [ 3347.307366] Lustre: lustre-MDT0000: Client 15c0e9fc-736c-41f2-b052-1123923b1b93 (at 192.168.202.51@tcp) reconnecting [ 3948.330636] Lustre: lustre-MDT0000: Client 15c0e9fc-736c-41f2-b052-1123923b1b93 (at 192.168.202.51@tcp) reconnecting ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.84s (real) 5.92s (CPU), Child processes: 4.87s