************************ crashinfo ************************* /exports/testreports/35272/testresults/sanity-quota-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg205-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.374I5/vmlinux [TAINTED] DUMPFILE: /exports/testreports/35272/testresults/sanity-quota-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg205-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Mon Sep 11 09:42:14 EDT 2023 UPTIME: 01:38:08 LOAD AVERAGE: 1.00, 1.00, 0.96 TASKS: 156 NODENAME: oleg205-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=29078 CPU=0 CMD=umount #-1 debug_object_active_state+0x83, 449 bytes of data #0 __call_rcu+0x55 #1 call_rcu_sched+0x1d #2 xas_free_nodes+0xa4 #3 xa_destroy+0xdf #4 osc_quota_cleanup+0x15 #5 osc_cleanup_common+0xbf #6 class_free_dev+0x219 #7 class_export_put+0x1e0 #8 class_unlink_export+0x125 #9 class_decref+0x80 #10 class_detach+0x1c1 #11 class_process_config+0x163b #12 class_manual_cleanup+0x1e0 #13 lov_tgts_putref+0x385 #14 lov_disconnect+0x237 #15 obd_connect.constprop.58+0x396 #16 ll_put_super+0x81c #17 generic_shutdown_super+0x6a #18 kill_anon_super+0x12 #19 lustre_kill_super+0x2b #20 deactivate_locked_super+0x49 #21 deactivate_super+0x46 #22 cleanup_mnt+0x3f #23 __cleanup_mnt+0x12 #24 task_work_run+0xb5 #25 do_notify_resume+0x92 #26 int_signal+0x12, 477 bytes of data -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 15 last 5s 20 last 60s 25 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 151 TASK_RUNNING 2 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 1 systemd 0 ms /usr/lib/systemd/systemd --switched-root --system --deserialize 22 9 rcu_sched 0 ms (no user stack) 14 watchdog/1 0 ms (no user stack) 668 kworker/1:2 101 ms (no user stack) 874 in:imjournal 371 ms /usr/sbin/rsyslogd -n +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955108 3.6 GB ---- FREE 835256 3.2 GB 87% of TOTAL MEM USED 119852 468.2 MB 12% of TOTAL MEM SHARED 10500 41 MB 1% of TOTAL MEM BUFFERS 5163 20.2 MB 0% of TOTAL MEM CACHED 56829 222 MB 5% of TOTAL MEM SLAB 17767 69.4 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739697 2.8 GB ---- COMMITTED 65999 257.8 MB 8% of TOTAL LIMIT +++ERROR+++ SLAB corruption kmem: xarray_cache: slab: ffffea00032e3100 duplicate freelist object: ffff8800cb8c6fe8 +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=umount ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5886.9 eth0 n/a 1.1 RSS_TOTAL=87904 pages, %mem= 1.5 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880118cca000 ffff880119940800 rootfs rootfs / ffff880117652380 ffff8800d7ff0800 sysfs sysfs /sys ffff880117652540 ffff880119944000 proc proc /proc ffff880117652700 ffff880117668000 devtmpfs devtmpfs /dev ffff8801176528c0 ffff8800d7466000 securityfs securityfs /sys/kernel/security ffff880117652a80 ffff8800d7ff1000 tmpfs tmpfs /dev/shm ffff880117652c40 ffff8801173b1000 devpts devpts /dev/pts ffff880117652e00 ffff8800d7ff1800 tmpfs tmpfs /run ffff880117652fc0 ffff8800d7ff2000 tmpfs tmpfs /sys/fs/cgroup ffff880117653180 ffff8800d7ff2800 cgroup cgroup /sys/fs/cgroup/systemd ffff880117653340 ffff8800d7ff3000 pstore pstore /sys/fs/pstore ffff880117653500 ffff8800d7ff5000 cgroup cgroup /sys/fs/cgroup/memory ffff8801176536c0 ffff8800d7ff4800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff880117653880 ffff8800d7ff4000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff880117653a40 ffff8800d7ff3800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff880117653c00 ffff8800d7ff5800 cgroup cgroup /sys/fs/cgroup/perf_event ffff880117653dc0 ffff8800d7ff6000 cgroup cgroup /sys/fs/cgroup/freezer ffff8800d74c0000 ffff8800d7ff6800 cgroup cgroup /sys/fs/cgroup/cpuset ffff8800d74c01c0 ffff8800d7ff7000 cgroup cgroup /sys/fs/cgroup/devices ffff8800d74c0380 ffff8800d7ff7800 cgroup cgroup /sys/fs/cgroup/pids ffff8800d74c0540 ffff8800d74c8000 cgroup cgroup /sys/fs/cgroup/blkio ffff880118ccba40 ffff8800d7467000 configfs configfs /sys/kernel/config ffff880118ccbc00 ffff8800d68ba800 ext4 /dev/nbd0 / ffff8800d74c1dc0 ffff8800d74cb000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8800d6be0000 ffff8800d6a14000 hugetlbfs hugetlbfs /dev/hugepages ffff8800d7650380 ffff8800d74c9000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880118ccbdc0 ffff8801173b2000 mqueue mqueue /dev/mqueue ffff8800d6be01c0 ffff880119947800 debugfs debugfs /sys/kernel/debug ffff8800d74c08c0 ffff8800d6b5c800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8800d6be0380 ffff8800d68b9800 ramfs none /mnt ffff8800d6be0540 ffff8800d68bf800 tmpfs none /var/lib/stateless/writable ffff8800d7650540 ffff8800d4074000 squashfs /dev/vda /home/green/git/lustre-release ffff8800d7650700 ffff8800d68bf800 tmpfs none /var/cache/man ffff8800d63f2000 ffff8800d68bf800 tmpfs none /var/log ffff8800d6be0700 ffff8800d68bf800 tmpfs none /var/lib/dbus ffff8800d76508c0 ffff8800d68bf800 tmpfs none /tmp ffff8800d7650a80 ffff8800d68bf800 tmpfs none /var/lib/dhclient ffff8800d7650c40 ffff8800d68bf800 tmpfs none /var/tmp ffff8800d7650e00 ffff8800d68bf800 tmpfs none /var/lib/NetworkManager ffff8800d63f21c0 ffff8800d68bf800 tmpfs none /var/lib/systemd/random-seed ffff8800d6be08c0 ffff8800d68bf800 tmpfs none /var/spool ffff8800d6be0a80 ffff8800d68bf800 tmpfs none /var/lib/nfs ffff8800d63f2380 ffff8800d68bf800 tmpfs none /var/lib/gssproxy ffff8800d7650fc0 ffff8800d68bf800 tmpfs none /var/lib/logrotate ffff8800d63f2540 ffff8800d68bf800 tmpfs none /etc ffff8800d7651180 ffff8800d68bf800 tmpfs none /var/lib/rsyslog ffff8800d7651340 ffff8800d68bf800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800d7651500 ffff8800d6808000 nfs4 192.168.200.253:/exports/state/oleg205-client.virtnet /var/lib/stateless/state ffff8800d6be0c40 ffff8800d6808000 nfs4 192.168.200.253:/exports/state/oleg205-client.virtnet /boot ffff8800d63f2700 ffff8800d6808000 nfs4 192.168.200.253:/exports/state/oleg205-client.virtnet /etc/etc/kdump.conf ffff8800d6be0e00 ffff8800d74cb000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800d7778380 ffff8800d6b58000 nfs4 192.168.200.253://exports/testreports/35272/testresults/sanity-quota-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8800d63f3c00 ffff8800d742c800 tmpfs tmpfs /run/user/0 ffff8800c90c2000 ffff8800d4074000 squashfs /dev/vda /usr/sbin/mount.lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 5600.514053] [] ? cleanup_mnt+0x3f/0x80 [ 5600.514960] [] ? __cleanup_mnt+0x12/0x20 [ 5600.515866] [] ? task_work_run+0xb5/0xf0 [ 5600.516763] [] ? do_notify_resume+0x92/0xb0 [ 5600.517752] [] ? int_signal+0x12/0x17 [ 5780.489003] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=3480118 jiffies, g=145399, c=145398, q=0) [ 5780.491090] Task dump for CPU 0: [ 5780.491649] umount R running task 12664 29078 29070 0x00000008 [ 5780.492836] Call Trace: [ 5780.493292] [] ? xa_destroy+0xdf/0xf0 [libcfs] [ 5780.494279] [] ? osc_quota_cleanup+0x15/0x20 [osc] [ 5780.495327] [] ? osc_cleanup_common+0xbf/0x1b0 [osc] [ 5780.496472] [] ? class_free_dev+0x219/0x730 [obdclass] [ 5780.497586] [] ? class_export_put+0x1e0/0x2e0 [obdclass] [ 5780.498741] [] ? class_unlink_export+0x125/0x160 [obdclass] [ 5780.499961] [] ? class_decref+0x80/0x160 [obdclass] [ 5780.501051] [] ? class_detach+0x1c1/0x310 [obdclass] [ 5780.502189] [] ? class_process_config+0x163b/0x27c0 [obdclass] [ 5780.503426] [] ? __kmalloc+0x1e0/0x340 [ 5780.504341] [] ? class_manual_cleanup+0x1e0/0x770 [obdclass] [ 5780.505541] [] ? lov_tgts_putref+0x385/0xad0 [lov] [ 5780.506599] [] ? lov_disconnect+0x237/0x280 [lov] [ 5780.507671] [] ? obd_disconnect+0x56/0x300 [lustre] [ 5780.508756] [] ? ll_put_super+0x81c/0xf30 [lustre] [ 5780.509842] [] ? call_rcu_sched+0x1d/0x20 [ 5780.510786] [] ? ll_destroy_inode+0x1c/0x20 [lustre] [ 5780.511862] [] ? destroy_inode+0x38/0x60 [ 5780.512777] [] ? evict+0x10e/0x180 [ 5780.513586] [] ? _raw_spin_unlock+0xe/0x20 [ 5780.514530] [] ? fsnotify_unmount_inodes+0x1d6/0x1e0 [ 5780.515599] [] ? generic_shutdown_super+0x6a/0xf0 [ 5780.516613] [] ? kill_anon_super+0x12/0x20 [ 5780.517579] [] ? lustre_kill_super+0x2b/0x30 [lustre] [ 5780.518694] [] ? deactivate_locked_super+0x49/0x60 [ 5780.519746] [] ? deactivate_super+0x46/0x60 [ 5780.520681] [] ? cleanup_mnt+0x3f/0x80 [ 5780.521576] [] ? __cleanup_mnt+0x12/0x20 [ 5780.522450] [] ? task_work_run+0xb5/0xf0 [ 5780.523363] [] ? do_notify_resume+0x92/0xb0 [ 5780.524309] [] ? int_signal+0x12/0x17 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details --------------------- A list of all +++ERROR+++ messages --------------------- SLAB corruption ------------------------------------------------------------------------------ ** Execution took 10.18s (real) 5.25s (CPU), Child processes: 4.90s