************************ crashinfo ************************* /exports/testreports/37062/testresults/sanity-quota-zfs-centos7_x86_64-centos7_x86_64/oleg401-client-vmcore (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.Hs8xZ/vmlinux [TAINTED] DUMPFILE: /exports/testreports/37062/testresults/sanity-quota-zfs-centos7_x86_64-centos7_x86_64/oleg401-client-vmcore [PARTIAL DUMP] CPUS: 4 DATE: Tue Oct 24 07:12:36 EDT 2023 UPTIME: 01:29:47 LOAD AVERAGE: 0.06, 0.14, 0.22 TASKS: 157 NODENAME: oleg401-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "BUG: unable to handle kernel NULL pointer dereference at (null)" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 rest_init+0x8e #12 start_kernel+0x456 #13 x86_64_start_reservations+0x2a #14 x86_64_start_kernel+0x152 #15 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 -- CPU#2 -- PID=12374 CPU=2 CMD=umount #0 machine_kexec+0x19e #1 __crash_kexec+0x72 #2 crash_kexec+0x30 #3 oops_end+0xa8 #4 no_context+0x144 #5 __bad_area_nosemaphore+0x112 #6 bad_area_nosemaphore+0x14 #7 __do_page_fault+0x348 #8 trace_do_page_fault+0x56 #9 do_async_page_fault+0x22 #10 async_page_fault+0x28 #-1 xas_free_nodes+0xb8, 477 bytes of data #11 xa_destroy+0xdf #12 osc_quota_cleanup+0x15 #13 osc_cleanup_common+0xbf #14 class_free_dev+0x219 #15 class_export_put+0x1e0 #16 class_unlink_export+0x125 #17 class_decref_free+0x4e #18 class_decref+0x48 #19 class_detach+0x1c1 #20 class_process_config+0x163b #21 class_manual_cleanup+0x1e0 #22 lov_tgts_putref+0x385 #23 lov_disconnect+0x237 #24 obd_connect.constprop.59+0x396 #25 ll_put_super+0x767 #26 generic_shutdown_super+0x6a #27 kill_anon_super+0x12 #28 lustre_kill_super+0x2b #29 deactivate_locked_super+0x49 #30 deactivate_super+0x46 #31 cleanup_mnt+0x3f #32 __cleanup_mnt+0x12 #33 task_work_run+0xb5 #34 do_notify_resume+0x92 #35 int_signal+0x12, 477 bytes of data -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ *** Dump was triggered by kernel *** BUG: 73 of 73 active objects replaced +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 53 last 5s 61 last 60s 70 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 152 TASK_RUNNING 2 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 34 rcuos/3 0 ms (no user stack) 1801 socknal_reaper 0 ms (no user stack) 21856 kworker/3:0 0 ms (no user stack) 9 rcu_sched 0 ms (no user stack) 24744 kworker/1:2 119 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 851422 3.2 GB 89% of TOTAL MEM USED 103657 404.9 MB 10% of TOTAL MEM SHARED 10016 39.1 MB 1% of TOTAL MEM BUFFERS 5208 20.3 MB 0% of TOTAL MEM CACHED 54734 213.8 MB 5% of TOTAL MEM SLAB 12465 48.7 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 63807 249.2 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=umount ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 TIME_WAIT 52 LISTEN 3 NAGLE disabled (TCP_NODELAY): 6 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5385.8 eth0 n/a 0.4 RSS_TOTAL=75492 pages, %mem= 1.2 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012ab08000 ffff88012ab10000 sysfs sysfs /sys ffff88012ab081c0 ffff880139944000 proc proc /proc ffff88012ab08380 ffff880137668000 devtmpfs devtmpfs /dev ffff88012ab08540 ffff8800b6cd1800 securityfs securityfs /sys/kernel/security ffff88012ab08700 ffff88012ab10800 tmpfs tmpfs /dev/shm ffff88012ab088c0 ffff88013731d000 devpts devpts /dev/pts ffff88012ab08a80 ffff88012ab11000 tmpfs tmpfs /run ffff88012ab08c40 ffff88012ab11800 tmpfs tmpfs /sys/fs/cgroup ffff88012ab08e00 ffff88012ab12000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012ab08fc0 ffff88012ab12800 pstore pstore /sys/fs/pstore ffff88012ab09180 ffff88012ab14800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012ab09340 ffff88012ab14000 cgroup cgroup /sys/fs/cgroup/devices ffff88012ab09500 ffff88012ab13800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012ab096c0 ffff88012ab13000 cgroup cgroup /sys/fs/cgroup/pids ffff88012ab09880 ffff88012ab15000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012ab09a40 ffff88012ab15800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012ab09c00 ffff88012ab16000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012ab09dc0 ffff88012ab16800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aa70000 ffff88012ab17000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aa701c0 ffff88012ab17800 cgroup cgroup /sys/fs/cgroup/memory ffff880138ccb340 ffff8800b6108000 configfs configfs /sys/kernel/config ffff880137652700 ffff8800b6cd6800 ext4 /dev/nbd0 / ffff8801376528c0 ffff8800b6cd3800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff880137652a80 ffff88012a616000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff88012aa708c0 ffff8800b6d08000 hugetlbfs hugetlbfs /dev/hugepages ffff8801377de540 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff88012aa70a80 ffff8801377fa000 mqueue mqueue /dev/mqueue ffff8801377de700 ffff8800b5821000 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8801377dea80 ffff8800b5820000 ramfs none /mnt ffff880137652c40 ffff88012a57e800 tmpfs none /var/lib/stateless/writable ffff880138ccb500 ffff8800b63da000 squashfs /dev/vda /home/green/git/lustre-release ffff880137652e00 ffff88012a57e800 tmpfs none /var/cache/man ffff88012aa70c40 ffff88012a57e800 tmpfs none /var/log ffff880137652fc0 ffff88012a57e800 tmpfs none /var/lib/dbus ffff88012aa70e00 ffff88012a57e800 tmpfs none /tmp ffff880137653180 ffff88012a57e800 tmpfs none /var/lib/dhclient ffff880137653340 ffff88012a57e800 tmpfs none /var/tmp ffff880137653500 ffff88012a57e800 tmpfs none /var/lib/NetworkManager ffff8801376536c0 ffff88012a57e800 tmpfs none /var/lib/systemd/random-seed ffff880137653880 ffff88012a57e800 tmpfs none /var/spool ffff880137653a40 ffff88012a57e800 tmpfs none /var/lib/nfs ffff8801377dec40 ffff88012a57e800 tmpfs none /var/lib/gssproxy ffff880137653c00 ffff88012a57e800 tmpfs none /var/lib/logrotate ffff880138ccb6c0 ffff88012a57e800 tmpfs none /etc ffff880137653dc0 ffff88012a57e800 tmpfs none /var/lib/rsyslog ffff880137652540 ffff88012a57e800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012aa70fc0 ffff8800b5822800 nfs4 192.168.200.253:/exports/state/oleg401-client.virtnet /var/lib/stateless/state ffff88012aa71180 ffff8800b5822800 nfs4 192.168.200.253:/exports/state/oleg401-client.virtnet /boot ffff8800b5b04000 ffff8800b5822800 nfs4 192.168.200.253:/exports/state/oleg401-client.virtnet /etc/etc/kdump.conf ffff88012aa71340 ffff8800b6cd3800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b5b041c0 ffff8800b623f800 nfs4 192.168.200.253://exports/testreports/37062/testresults/sanity-quota-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff880138ccb880 ffff8800b63db800 tmpfs tmpfs /run/user/0 ffff8800b1d93a40 ffff8800b63da000 squashfs /dev/vda /usr/sbin/mount.lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 5388.481433] FS: 00007fbcfa47f880(0000) GS:ffff88013e300000(0000) knlGS:0000000000000000 [ 5388.482807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5388.483784] CR2: 0000000000000000 CR3: 00000000b6230000 CR4: 00000000000006e0 [ 5388.484962] Call Trace: [ 5388.485372] [] xa_destroy+0xdf/0xf0 [libcfs] [ 5388.486415] [] osc_quota_cleanup+0x15/0x20 [osc] [ 5388.487506] [] osc_cleanup_common+0xbf/0x1b0 [osc] [ 5388.488607] [] class_free_dev+0x219/0x730 [obdclass] [ 5388.489606] [] class_export_put+0x1e0/0x2e0 [obdclass] [ 5388.490799] [] class_unlink_export+0x125/0x160 [obdclass] [ 5388.492043] [] class_decref_free+0x4e/0x90 [obdclass] [ 5388.493112] [] class_decref+0x48/0xf0 [obdclass] [ 5388.494192] [] class_detach+0x1c1/0x310 [obdclass] [ 5388.495278] [] class_process_config+0x163b/0x27c0 [obdclass] [ 5388.496501] [] ? __kmalloc+0x1e0/0x340 [ 5388.497465] [] class_manual_cleanup+0x1e0/0x770 [obdclass] [ 5388.498699] [] lov_tgts_putref+0x385/0xad0 [lov] [ 5388.499813] [] lov_disconnect+0x237/0x280 [lov] [ 5388.500829] [] obd_disconnect+0x56/0x300 [lustre] [ 5388.501904] [] ll_put_super+0x767/0xce0 [lustre] [ 5388.503002] [] ? call_rcu_sched+0x1d/0x20 [ 5388.503986] [] ? ll_destroy_inode+0x1c/0x20 [lustre] [ 5388.505124] [] ? destroy_inode+0x38/0x60 [ 5388.506008] [] ? evict+0x10e/0x180 [ 5388.506891] [] ? _raw_spin_unlock+0xe/0x20 [ 5388.507887] [] ? fsnotify_unmount_inodes+0x1d6/0x1e0 [ 5388.509019] [] generic_shutdown_super+0x6a/0xf0 [ 5388.510087] [] kill_anon_super+0x12/0x20 [ 5388.511048] [] lustre_kill_super+0x2b/0x30 [lustre] [ 5388.512173] [] deactivate_locked_super+0x49/0x60 [ 5388.513269] [] deactivate_super+0x46/0x60 [ 5388.514218] [] cleanup_mnt+0x3f/0x80 [ 5388.515137] [] __cleanup_mnt+0x12/0x20 [ 5388.516044] [] task_work_run+0xb5/0xf0 [ 5388.516974] [] do_notify_resume+0x92/0xb0 [ 5388.517922] [] int_signal+0x12/0x17 [ 5388.518743] Code: 8d 7b 18 48 c7 43 10 01 00 00 00 48 c7 c6 60 be 18 a0 e8 dc 1f fc e0 4c 39 f3 75 b7 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 40 00 <41> 0f b6 4d 00 4c 89 eb e9 5c ff ff ff 0f 1f 00 48 81 fa 00 10 [ 5388.523115] RIP [] xas_free_nodes+0xb8/0xf0 [libcfs] [ 5388.524266] RSP [ 5388.524876] CR2: 0000000000000000 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 9.59s (real) 4.94s (CPU), Child processes: 4.62s