************************ crashinfo ************************* /exports/testreports/37274/testresults/sanity-quota-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg345-client-vmcore (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.OmalY/vmlinux [TAINTED] DUMPFILE: /exports/testreports/37274/testresults/sanity-quota-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg345-client-vmcore [PARTIAL DUMP] CPUS: 4 DATE: Tue Oct 31 09:52:46 EDT 2023 UPTIME: 01:04:56 LOAD AVERAGE: 1.41, 0.49, 0.36 TASKS: 155 NODENAME: oleg345-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "Kernel panic - not syncing: softlockup: hung tasks" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 rest_init+0x8e #12 start_kernel+0x456 #13 x86_64_start_reservations+0x2a #14 x86_64_start_kernel+0x152 #15 start_cpu+0x5 -- CPU#1 -- PID=1855 CPU=1 CMD=ptlrpcd_00_00 #0 machine_kexec+0x19e #1 __crash_kexec+0x72 #2 panic+0xf3 #3 watchdog_timer_fn+0x1fa #4 __hrtimer_run_queues+0x139 #5 hrtimer_interrupt+0xaf #6 local_apic_timer_interrupt+0x35 #7 smp_apic_timer_interrupt+0x3d #8 apic_timer_interrupt+0x16a, 19 bytes of data #9 apic_timer_interrupt+0x16a #-1 xas_free_nodes+0xdd, 477 bytes of data #10 xas_store+0x184 #11 __xa_insert+0xdc #12 osc_quota_setdq+0x220 #13 osc_brw_fini_request+0xa9c #14 brw_interpret+0x57 #15 ptlrpc_check_set+0x428 #16 ptlrpcd+0xa94 #17 kthread+0xe4 #18 ret_from_fork_nospec_begin+0x7 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ *** Panic *** watchdog timer +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 18 last 5s 31 last 60s 66 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 147 TASK_RUNNING 5 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 9 rcu_sched 0 ms (no user stack) 21 watchdog/2 0 ms (no user stack) 13 watchdog/0 12 ms (no user stack) 1852 lnet_discovery 16 ms (no user stack) 28 watchdog/3 61 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 846594 3.2 GB 88% of TOTAL MEM USED 108485 423.8 MB 11% of TOTAL MEM SHARED 12911 50.4 MB 1% of TOTAL MEM BUFFERS 5214 20.4 MB 0% of TOTAL MEM CACHED 57556 224.8 MB 6% of TOTAL MEM SLAB 12833 50.1 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 64767 253 MB 8% of TOTAL LIMIT +++ERROR+++ SLAB corruption kmem: xarray_cache: slab: ffffea0004dbeb00 duplicate freelist object: ffff880136faefe8 +++WARNING+++ Panic while we were doing sync on FS +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=ptlrpcd_00_00 16638 kworker/1:2 0.02878 -- RT Queues --- prio=0 len=1 1 Real-Time processes on this CPU ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 TIME_WAIT 30 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 3894.1 eth0 n/a 1.1 RSS_TOTAL=76988 pages, %mem= 1.2 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aa94000 ffff88012b385000 sysfs sysfs /sys ffff88012aa941c0 ffff880139944000 proc proc /proc ffff88012aa94380 ffff880137668000 devtmpfs devtmpfs /dev ffff88012aa94540 ffff8800b6d11800 securityfs securityfs /sys/kernel/security ffff88012aa94700 ffff88012b385800 tmpfs tmpfs /dev/shm ffff88012aa948c0 ffff88012b210800 devpts devpts /dev/pts ffff88012aa94a80 ffff88012b386000 tmpfs tmpfs /run ffff88012aa94c40 ffff88012b386800 tmpfs tmpfs /sys/fs/cgroup ffff88012aa94e00 ffff88012b387000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aa94fc0 ffff88012b387800 pstore pstore /sys/fs/pstore ffff88012aa95180 ffff88012aaa9800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aa95340 ffff88012aaa9000 cgroup cgroup /sys/fs/cgroup/memory ffff88012aa95500 ffff88012aaa8800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aa956c0 ffff88012aaa8000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aa95880 ffff88012aaaa000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aa95a40 ffff88012aaaa800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aa95c00 ffff88012aaab000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aa95dc0 ffff88012aaab800 cgroup cgroup /sys/fs/cgroup/devices ffff88012a9e4000 ffff88012aaac000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a9e41c0 ffff88012aaac800 cgroup cgroup /sys/fs/cgroup/pids ffff88012b20a8c0 ffff8800b60a1800 configfs configfs /sys/kernel/config ffff880138ccafc0 ffff88012a5d2000 ext4 /dev/nbd0 / ffff88012a9e4540 ffff88012aaaf000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff880137652540 ffff8800b60a4000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff88012a9e4c40 ffff88012a625800 hugetlbfs hugetlbfs /dev/hugepages ffff88012a9e4e00 ffff88012b211800 mqueue mqueue /dev/mqueue ffff88012a9e4fc0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff88012b20aa80 ffff8800b58fe000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8801376528c0 ffff8800b6d15000 ramfs none /mnt ffff880138ccb180 ffff8800b624f000 tmpfs none /var/lib/stateless/writable ffff88012b20ac40 ffff8800b611f800 squashfs /dev/vda /home/green/git/lustre-release ffff880137652a80 ffff8800b624f000 tmpfs none /var/cache/man ffff880137652c40 ffff8800b624f000 tmpfs none /var/log ffff88012a9e5180 ffff8800b624f000 tmpfs none /var/lib/dbus ffff880138ccb340 ffff8800b624f000 tmpfs none /tmp ffff88012a9e5340 ffff8800b624f000 tmpfs none /var/lib/dhclient ffff880137652e00 ffff8800b624f000 tmpfs none /var/tmp ffff880137652fc0 ffff8800b624f000 tmpfs none /var/lib/NetworkManager ffff880138ccb500 ffff8800b624f000 tmpfs none /var/lib/systemd/random-seed ffff88012a9e5500 ffff8800b624f000 tmpfs none /var/spool ffff88012a9e56c0 ffff8800b624f000 tmpfs none /var/lib/nfs ffff880137653180 ffff8800b624f000 tmpfs none /var/lib/gssproxy ffff88012a9e5880 ffff8800b624f000 tmpfs none /var/lib/logrotate ffff880137653340 ffff8800b624f000 tmpfs none /etc ffff88012a9e5a40 ffff8800b624f000 tmpfs none /var/lib/rsyslog ffff880137653500 ffff8800b624f000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8801376536c0 ffff8800b6cd4000 nfs4 192.168.200.253:/exports/state/oleg345-client.virtnet /var/lib/stateless/state ffff88012a9e5dc0 ffff8800b6cd4000 nfs4 192.168.200.253:/exports/state/oleg345-client.virtnet /boot ffff880137653880 ffff8800b6cd4000 nfs4 192.168.200.253:/exports/state/oleg345-client.virtnet /etc/etc/kdump.conf ffff88012a9e5c00 ffff88012aaaf000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff88012f38a1c0 ffff88012a61e000 nfs4 192.168.200.253://exports/testreports/37274/testresults/sanity-quota-ldiskfs-DNE-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff880138ccb6c0 ffff88012a62f800 tmpfs tmpfs /run/user/0 ffff880137653a40 ffff8800b611f800 squashfs /dev/vda /usr/sbin/mount.lustre ffff880138ccbdc0 ffff8800abbbf000 lustre 192.168.203.145@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 3896.838887] [] osc_quota_setdq+0x220/0x510 [osc] [ 3896.840285] [] osc_brw_fini_request+0xa9c/0x1b10 [osc] [ 3896.841694] [] brw_interpret+0x57/0xdb0 [osc] [ 3896.843082] [] ptlrpc_check_set+0x428/0x2170 [ptlrpc] [ 3896.844618] [] ptlrpcd+0xa94/0xb70 [ptlrpc] [ 3896.845884] [] ? abort_exclusive_wait+0xa0/0xa0 [ 3896.847259] [] ? ptlrpcd_partners+0x3a0/0x3a0 [ptlrpc] [ 3896.848744] [] kthread+0xe4/0xf0 [ 3896.849779] [] ? kthread_create_on_node+0x140/0x140 [ 3896.851231] [] ret_from_fork_nospec_begin+0x7/0x21 [ 3896.852681] [] ? kthread_create_on_node+0x140/0x140 [ 3896.854083] Code: 5d c3 0f 1f 40 00 41 0f b6 4d 00 4c 89 eb e9 5c ff ff ff 0f 1f 00 48 81 fa 00 10 00 00 0f 86 6b ff ff ff 48 8d 5a fe 0f b6 4a fe <45> 31 e4 e9 3c ff ff ff 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f [ 3896.859259] Kernel panic - not syncing: softlockup: hung tasks [ 3896.860544] CPU: 1 PID: 1855 Comm: ptlrpcd_00_00 Kdump: loaded Tainted: G OEL ------------ 3.10.0-7.9-debug #1 [ 3896.863326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 3896.865307] Call Trace: [ 3896.865969] [] dump_stack+0x19/0x1b [ 3896.867297] [] panic+0xe8/0x20d [ 3896.868344] [] watchdog_timer_fn+0x1fa/0x210 [ 3896.869666] [] ? watchdog+0x40/0x40 [ 3896.870797] [] __hrtimer_run_queues+0x139/0x2e0 [ 3896.872207] [] hrtimer_interrupt+0xaf/0x1e0 [ 3896.873398] [] ? call_softirq+0x1c/0x30 [ 3896.874577] [] local_apic_timer_interrupt+0x35/0x60 [ 3896.876025] [] smp_apic_timer_interrupt+0x3d/0x50 [ 3896.877384] [] apic_timer_interrupt+0x16a/0x170 [ 3896.878799] [] ? xas_free_nodes+0xdd/0xf0 [libcfs] [ 3896.880399] [] xas_store+0x184/0x540 [libcfs] [ 3896.881692] [] __xa_insert+0xdc/0x150 [libcfs] [ 3896.883099] [] osc_quota_setdq+0x220/0x510 [osc] [ 3896.884577] [] osc_brw_fini_request+0xa9c/0x1b10 [osc] [ 3896.885989] [] brw_interpret+0x57/0xdb0 [osc] [ 3896.887370] [] ptlrpc_check_set+0x428/0x2170 [ptlrpc] [ 3896.888788] [] ptlrpcd+0xa94/0xb70 [ptlrpc] [ 3896.890015] [] ? abort_exclusive_wait+0xa0/0xa0 [ 3896.891366] [] ? ptlrpcd_partners+0x3a0/0x3a0 [ptlrpc] [ 3896.892807] [] kthread+0xe4/0xf0 [ 3896.893843] [] ? kthread_create_on_node+0x140/0x140 [ 3896.895250] [] ret_from_fork_nospec_begin+0x7/0x21 [ 3896.896575] [] ? kthread_create_on_node+0x140/0x140 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details Panic while we were doing sync on FS --------------------- A list of all +++ERROR+++ messages --------------------- SLAB corruption ------------------------------------------------------------------------------ ** Execution took 9.84s (real) 5.21s (CPU), Child processes: 4.60s