************************ crashinfo ************************* /exports/testreports/35992/testresults/sanity-quota-zfs-centos7_x86_64-centos7_x86_64/oleg105-client-vmcore (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.Dkr1p/vmlinux [TAINTED] DUMPFILE: /exports/testreports/35992/testresults/sanity-quota-zfs-centos7_x86_64-centos7_x86_64/oleg105-client-vmcore [PARTIAL DUMP] CPUS: 4 DATE: Mon Sep 25 15:22:49 EDT 2023 UPTIME: 01:30:07 LOAD AVERAGE: 0.25, 0.16, 0.21 TASKS: 157 NODENAME: oleg105-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "BUG: unable to handle kernel NULL pointer dereference at (null)" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=9909 CPU=0 CMD=umount #0 machine_kexec+0x19e #1 __crash_kexec+0x72 #2 crash_kexec+0x30 #3 oops_end+0xa8 #4 no_context+0x144 #5 __bad_area_nosemaphore+0x112 #6 bad_area_nosemaphore+0x14 #7 __do_page_fault+0x348 #8 trace_do_page_fault+0x56 #9 do_async_page_fault+0x22 #10 async_page_fault+0x28 #-1 xas_free_nodes+0xb8, 477 bytes of data #11 xa_destroy+0xdf #12 osc_quota_cleanup+0x15 #13 osc_cleanup_common+0xbf #14 class_free_dev+0x219 #15 class_export_put+0x1e0 #16 class_unlink_export+0x125 #17 class_decref_free+0x4e #18 class_decref+0x48 #19 class_detach+0x1c1 #20 class_process_config+0x163b #21 class_manual_cleanup+0x1e0 #22 lov_tgts_putref+0x385 #23 lov_disconnect+0x237 #24 obd_connect.constprop.58+0x396 #25 ll_put_super+0x81c #26 generic_shutdown_super+0x6a #27 kill_anon_super+0x12 #28 lustre_kill_super+0x2b #29 deactivate_locked_super+0x49 #30 deactivate_super+0x46 #31 cleanup_mnt+0x3f #32 __cleanup_mnt+0x12 #33 task_work_run+0xb5 #34 do_notify_resume+0x92 #35 int_signal+0x12, 477 bytes of data -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ *** Dump was triggered by kernel *** BUG: 73 of 73 active objects replaced +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 53 last 5s 57 last 60s 73 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 152 TASK_RUNNING 2 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 9909 umount 0 ms (no user stack) 23117 kworker/0:2 0 ms (no user stack) 9 rcu_sched 0 ms (no user stack) 1802 socknal_sd00_00 0 ms (no user stack) 790 gmain 0 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 852123 3.3 GB 89% of TOTAL MEM USED 102956 402.2 MB 10% of TOTAL MEM SHARED 10002 39.1 MB 1% of TOTAL MEM BUFFERS 5203 20.3 MB 0% of TOTAL MEM CACHED 54705 213.7 MB 5% of TOTAL MEM SLAB 12812 50 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 63829 249.3 MB 8% of TOTAL LIMIT +++ERROR+++ SLAB corruption kmem: xarray_cache: slab: ffffea0004dbbd00 duplicate freelist object: ffff880136ef5918 +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=umount ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 TIME_WAIT 57 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5404.9 eth0 n/a 0.1 RSS_TOTAL=71540 pages, %mem= 1.2 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aad8000 ffff88012aae0000 sysfs sysfs /sys ffff88012aad81c0 ffff880139944000 proc proc /proc ffff88012aad8380 ffff880137668000 devtmpfs devtmpfs /dev ffff88012aad8540 ffff8800b6d03000 securityfs securityfs /sys/kernel/security ffff88012aad8700 ffff88012aae0800 tmpfs tmpfs /dev/shm ffff88012aad88c0 ffff880136ba1000 devpts devpts /dev/pts ffff88012aad8a80 ffff88012aae1000 tmpfs tmpfs /run ffff88012aad8c40 ffff88012aae1800 tmpfs tmpfs /sys/fs/cgroup ffff88012aad8e00 ffff88012aae2000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aad8fc0 ffff88012aae2800 pstore pstore /sys/fs/pstore ffff88012aad9180 ffff88012aae4800 cgroup cgroup /sys/fs/cgroup/memory ffff88012aad9340 ffff88012aae4000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aad9500 ffff88012aae3800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aad96c0 ffff88012aae3000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aad9880 ffff88012aae5000 cgroup cgroup /sys/fs/cgroup/devices ffff88012aad9a40 ffff88012aae5800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aad9c00 ffff88012aae6000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aad9dc0 ffff88012aae6800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aa34000 ffff88012aae7000 cgroup cgroup /sys/fs/cgroup/pids ffff88012aa341c0 ffff88012aae7800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aa34380 ffff88012ab5e000 configfs configfs /sys/kernel/config ffff880138ccac40 ffff88012a4fe000 ext4 /dev/nbd0 / ffff88012aa34540 ffff8800b6e4e000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012aa34c40 ffff8800b5889000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff88012b2468c0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff880138ccae00 ffff880136ba2000 mqueue mqueue /dev/mqueue ffff880137652a80 ffff8800b5877800 hugetlbfs hugetlbfs /dev/hugepages ffff880138ccafc0 ffff88012a598800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012b246a80 ffff8800b5947000 ramfs none /mnt ffff88012b246c40 ffff88012b3a5000 tmpfs none /var/lib/stateless/writable ffff88012aa34e00 ffff88012a4f8800 squashfs /dev/vda /home/green/git/lustre-release ffff880137652e00 ffff88012b3a5000 tmpfs none /var/cache/man ffff88012aa34fc0 ffff88012b3a5000 tmpfs none /var/log ffff88012aa35180 ffff88012b3a5000 tmpfs none /var/lib/dbus ffff88012aa35340 ffff88012b3a5000 tmpfs none /tmp ffff880137652fc0 ffff88012b3a5000 tmpfs none /var/lib/dhclient ffff88012aa35500 ffff88012b3a5000 tmpfs none /var/tmp ffff88012b246e00 ffff88012b3a5000 tmpfs none /var/lib/NetworkManager ffff88012aa356c0 ffff88012b3a5000 tmpfs none /var/lib/systemd/random-seed ffff880137653180 ffff88012b3a5000 tmpfs none /var/spool ffff88012aa35880 ffff88012b3a5000 tmpfs none /var/lib/nfs ffff88012aa35a40 ffff88012b3a5000 tmpfs none /var/lib/gssproxy ffff880137653340 ffff88012b3a5000 tmpfs none /var/lib/logrotate ffff880138ccb180 ffff88012b3a5000 tmpfs none /etc ffff880137653500 ffff88012b3a5000 tmpfs none /var/lib/rsyslog ffff88012b246fc0 ffff88012b3a5000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012b247180 ffff8800b40d3000 nfs4 192.168.200.253:/exports/state/oleg105-client.virtnet /var/lib/stateless/state ffff88012aa35dc0 ffff8800b40d3000 nfs4 192.168.200.253:/exports/state/oleg105-client.virtnet /boot ffff8801376536c0 ffff8800b40d3000 nfs4 192.168.200.253:/exports/state/oleg105-client.virtnet /etc/etc/kdump.conf ffff88012aa35c00 ffff8800b6e4e000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff88012e18e380 ffff8800b673b000 nfs4 192.168.200.253://exports/testreports/35992/testresults/sanity-quota-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff880137653880 ffff88012a59f000 tmpfs tmpfs /run/user/0 ffff88012e18e8c0 ffff88012a4f8800 squashfs /dev/vda /usr/sbin/mount.lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 5407.765708] FS: 00007f19ba87f880(0000) GS:ffff88013e200000(0000) knlGS:0000000000000000 [ 5407.767225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5407.768257] CR2: 0000000000000000 CR3: 00000000a4d64000 CR4: 00000000000006f0 [ 5407.769490] Call Trace: [ 5407.769952] [] xa_destroy+0xdf/0xf0 [libcfs] [ 5407.771357] [] osc_quota_cleanup+0x15/0x20 [osc] [ 5407.773200] [] osc_cleanup_common+0xbf/0x1b0 [osc] [ 5407.774839] [] class_free_dev+0x219/0x730 [obdclass] [ 5407.776139] [] class_export_put+0x1e0/0x2e0 [obdclass] [ 5407.777489] [] class_unlink_export+0x125/0x160 [obdclass] [ 5407.778876] [] class_decref_free+0x4e/0x90 [obdclass] [ 5407.780371] [] class_decref+0x48/0xf0 [obdclass] [ 5407.781595] [] class_detach+0x1c1/0x310 [obdclass] [ 5407.782914] [] class_process_config+0x163b/0x27c0 [obdclass] [ 5407.784281] [] ? __kmalloc+0x1e0/0x340 [ 5407.785303] [] class_manual_cleanup+0x1e0/0x770 [obdclass] [ 5407.786693] [] lov_tgts_putref+0x385/0xad0 [lov] [ 5407.787835] [] lov_disconnect+0x237/0x280 [lov] [ 5407.789099] [] obd_disconnect+0x56/0x300 [lustre] [ 5407.790299] [] ll_put_super+0x81c/0xf30 [lustre] [ 5407.791405] [] ? call_rcu_sched+0x1d/0x20 [ 5407.792505] [] ? ll_destroy_inode+0x1c/0x20 [lustre] [ 5407.793782] [] ? destroy_inode+0x38/0x60 [ 5407.794773] [] ? evict+0x10e/0x180 [ 5407.795688] [] ? _raw_spin_unlock+0xe/0x20 [ 5407.796742] [] ? fsnotify_unmount_inodes+0x1d6/0x1e0 [ 5407.798195] [] generic_shutdown_super+0x6a/0xf0 [ 5407.799346] [] kill_anon_super+0x12/0x20 [ 5407.800327] [] lustre_kill_super+0x2b/0x30 [lustre] [ 5407.801523] [] deactivate_locked_super+0x49/0x60 [ 5407.802874] [] deactivate_super+0x46/0x60 [ 5407.803920] [] cleanup_mnt+0x3f/0x80 [ 5407.804842] [] __cleanup_mnt+0x12/0x20 [ 5407.805856] [] task_work_run+0xb5/0xf0 [ 5407.806867] [] do_notify_resume+0x92/0xb0 [ 5407.807933] [] int_signal+0x12/0x17 [ 5407.809287] Code: 8d 7b 18 48 c7 43 10 01 00 00 00 48 c7 c6 60 de 16 a0 e8 dc ff fd e0 4c 39 f3 75 b7 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 40 00 <41> 0f b6 4d 00 4c 89 eb e9 5c ff ff ff 0f 1f 00 48 81 fa 00 10 [ 5407.814944] RIP [] xas_free_nodes+0xb8/0xf0 [libcfs] [ 5407.816273] RSP [ 5407.817001] CR2: 0000000000000000 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details --------------------- A list of all +++ERROR+++ messages --------------------- SLAB corruption ------------------------------------------------------------------------------ ** Execution took 9.96s (real) 5.16s (CPU), Child processes: 4.75s