************************ crashinfo ************************* /exports/testreports/45132/testresults/sanityn-zfs-centos7_x86_64-centos7_x86_64/oleg431-server-vmcore (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.XUnpe/vmlinux [TAINTED] DUMPFILE: /exports/testreports/45132/testresults/sanityn-zfs-centos7_x86_64-centos7_x86_64/oleg431-server-vmcore [PARTIAL DUMP] CPUS: 4 DATE: Thu Aug 22 03:00:25 EDT 2024 UPTIME: 01:33:45 LOAD AVERAGE: 0.74, 0.52, 0.49 TASKS: 364 NODENAME: oleg431-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=4540 CPU=0 CMD=umount #0 machine_kexec+0x19e #1 __crash_kexec+0x72 #2 crash_kexec+0x30 #3 oops_end+0xa8 #4 die+0x4b #5 do_general_protection+0xa2 #6 general_protection+0x28 #-1 __list_del_entry+0x29, 477 bytes of data #7 list_del+0xd #8 percpu_counter_destroy+0x44 #9 percpu_counter_fixup_free+0x1d #10 debug_check_no_obj_freed+0x250 #11 free_pages_prepare+0x1c6 #12 __free_pages_ok+0x19 #13 __free_pages+0x1d #14 kfree+0x1a7 #15 osd_device_free+0x16a #16 lu_object_fini+0x5e #17 dt_object_fini+0xe #18 osd_object_free+0x3f #19 lu_object_free+0x13d #20 lu_object_put+0xb1 #21 lu_object_put_nocache+0x16 #22 osd_scrub_cleanup+0x3d #23 osd_device_fini+0x101 #24 obd_precleanup+0x143 #25 class_cleanup+0x23d #26 class_process_config+0x547 #27 class_manual_cleanup+0x1ce #28 osd_obd_disconnect+0x185 #29 lustre_put_lsi_free+0x12d #30 lustre_put_lsi+0x125 #31 server_put_super+0x159 #32 generic_shutdown_super+0x6a #33 kill_anon_super+0x12 #34 lustre_kill_super+0x2b #35 deactivate_locked_super+0x49 #36 deactivate_super+0x46 #37 cleanup_mnt+0x3f #38 __cleanup_mnt+0x12 #39 task_work_run+0xb5 #40 do_notify_resume+0x92 #41 int_signal+0x12, 477 bytes of data -- CPU#1 -- PID=9980 CPU=1 CMD=kworker/1:0 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 smp_call_function_many+0x202, 507 bytes of data #4 smp_call_function_many+0x202 #5 smp_call_function+0x37 #6 on_each_cpu+0x2b #7 flush_tlb_kernel_range+0x59 #8 remove_vm_area+0xa6 #9 __vunmap+0x2e #10 vfree+0x48 #11 free_work+0x1f0b35fb #12 process_one_work+0x18d #13 worker_thread+0x126 #14 kthread+0xe4 #15 ret_from_fork_nospec_begin+0x7 -- CPU#2 -- PID=30016 CPU=2 CMD=kworker/2:0 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 smp_call_function_many+0x202, 507 bytes of data #4 smp_call_function_many+0x202 #5 smp_call_function+0x37 #6 on_each_cpu+0x2b #7 flush_tlb_kernel_range+0x59 #8 remove_vm_area+0xa6 #9 __vunmap+0x2e #10 vfree+0x48 #11 free_work+0x1f0b35fb #12 process_one_work+0x18d #13 worker_thread+0x126 #14 kthread+0xe4 #15 ret_from_fork_nospec_begin+0x7 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #0 crash_nmi_callback+0x31 #1 nmi_handle+0x97 #2 do_nmi+0x12c #3 end_repeat_nmi+0x1e #-1 native_safe_halt+0xb, 507 bytes of data #4 native_safe_halt+0xb #5 default_idle+0x1e #6 default_enter_idle+0x45 #7 cpuidle_enter_state+0x40 #8 cpuidle_idle_call+0xd8 #9 arch_cpu_idle+0xe #10 cpu_startup_entry+0x14a #11 start_secondary+0x1eb #12 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ *** Dump was triggered by kernel *** General Protection Fault +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 51 last 5s 133 last 60s 140 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 355 TASK_RUNNING 6 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 9 rcu_sched 0 ms (no user stack) 30 ksoftirqd/3 8 ms (no user stack) 4540 umount 149 ms (no user stack) 6797 txg_quiesce 149 ms (no user stack) 6798 txg_sync 149 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 685726 2.6 GB 71% of TOTAL MEM USED 269341 1 GB 28% of TOTAL MEM SHARED 11150 43.6 MB 1% of TOTAL MEM BUFFERS 5903 23.1 MB 0% of TOTAL MEM CACHED 75842 296.3 MB 7% of TOTAL MEM SLAB 32311 126.2 MB 3% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 65105 254.3 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=umount 24586 kworker/0:2 0.34155 ---+ CPU=1 ---- | CURRENT TASK , CMD=kworker/1:0 ---+ CPU=2 ---- | CURRENT TASK , CMD=kworker/2:0 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 10 LISTEN 3 NAGLE disabled (TCP_NODELAY): 8 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5622.9 eth0 n/a 0.2 RSS_TOTAL=63664 pages, %mem= 0.9 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012a32c000 ffff8801377ee800 sysfs sysfs /sys ffff88012a32c1c0 ffff880139944000 proc proc /proc ffff88012a32c380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012a32c540 ffff8800b5268000 securityfs securityfs /sys/kernel/security ffff88012a32c700 ffff8801377ef000 tmpfs tmpfs /dev/shm ffff88012a32c8c0 ffff88013734b000 devpts devpts /dev/pts ffff88012a32ca80 ffff8801377ef800 tmpfs tmpfs /run ffff88012a32cc40 ffff8801377cf800 tmpfs tmpfs /sys/fs/cgroup ffff88012a32ce00 ffff88012a368000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012a32cfc0 ffff88012a368800 pstore pstore /sys/fs/pstore ffff88012a32d180 ffff88012a36a800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a32d340 ffff88012a36a000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a32d500 ffff88012a369800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a32d6c0 ffff88012a369000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a32d880 ffff88012a36b000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012a32da40 ffff88012a36b800 cgroup cgroup /sys/fs/cgroup/pids ffff88012a32dc00 ffff88012a36c000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a32ddc0 ffff88012a36c800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a37c000 ffff88012a36d000 cgroup cgroup /sys/fs/cgroup/memory ffff88012a37c1c0 ffff88012a36d800 cgroup cgroup /sys/fs/cgroup/devices ffff88013701c380 ffff8800b51ee800 configfs configfs /sys/kernel/config ffff880129f6e380 ffff8800b5269000 ext4 /dev/nbd0 / ffff88013701c540 ffff8800b51eb800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff880129f6e540 ffff8800b4a54000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff88012a37c8c0 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff88012a37ca80 ffff88013734c800 mqueue mqueue /dev/mqueue ffff88013701c700 ffff8800b4147000 hugetlbfs hugetlbfs /dev/hugepages ffff88012a37cc40 ffff8800b4251800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff880129f6e700 ffff8800b4a52800 ramfs none /mnt ffff88013701ca80 ffff8800b4141800 tmpfs none /var/lib/stateless/writable ffff880138ccafc0 ffff8800b4092000 squashfs /dev/vda /home/green/git/lustre-release ffff880138ccb180 ffff8800b4141800 tmpfs none /var/cache/man ffff880129f6ec40 ffff8800b4141800 tmpfs none /var/log ffff880138ccb340 ffff8800b4141800 tmpfs none /var/lib/dbus ffff880129f6ee00 ffff8800b4141800 tmpfs none /tmp ffff88012a37ce00 ffff8800b4141800 tmpfs none /var/lib/dhclient ffff88012a37cfc0 ffff8800b4141800 tmpfs none /var/tmp ffff880129f6efc0 ffff8800b4141800 tmpfs none /var/lib/NetworkManager ffff880138ccb500 ffff8800b4141800 tmpfs none /var/lib/systemd/random-seed ffff88013701cc40 ffff8800b4141800 tmpfs none /var/spool ffff880129f6f180 ffff8800b4141800 tmpfs none /var/lib/nfs ffff880138ccb6c0 ffff8800b4141800 tmpfs none /var/lib/gssproxy ffff88013701ce00 ffff8800b4141800 tmpfs none /var/lib/logrotate ffff880129f6f340 ffff8800b4141800 tmpfs none /etc ffff880129f6f500 ffff8800b4141800 tmpfs none /var/lib/rsyslog ffff88013701cfc0 ffff8800b4141800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88013701d180 ffff8800b4140000 nfs4 192.168.200.253:/exports/state/oleg431-server.virtnet /var/lib/stateless/state ffff880129f6f6c0 ffff8800b4140000 nfs4 192.168.200.253:/exports/state/oleg431-server.virtnet /boot ffff880129f6f880 ffff8800b4140000 nfs4 192.168.200.253:/exports/state/oleg431-server.virtnet /etc/etc/kdump.conf ffff88012a37d500 ffff8800b51eb800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b1b0e000 ffff8800b4092000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b4b31500 ffff88009bf56800 lustre lustre-ost1/ost1 /mnt/lustre-ost1 ffff8800b1905180 ffff88009a50c000 lustre lustre-ost2/ost2 /mnt/lustre-ost2 ffff8800b1905c00 ffff8801315b3000 tmpfs tmpfs /run/user/0 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 5626.530393] [] percpu_counter_fixup_free+0x1d/0x40 [ 5626.533089] [] debug_check_no_obj_freed+0x250/0x260 [ 5626.536407] [] free_pages_prepare+0x1c6/0x290 [ 5626.538261] [] ? lprocfs_stats_free+0x276/0x440 [obdclass] [ 5626.541054] [] __free_pages_ok+0x19/0xa0 [ 5626.548006] [] __free_pages+0x1d/0x30 [ 5626.549911] [] kfree+0x1a7/0x1d0 [ 5626.554114] [] osd_device_free+0x16a/0x220 [osd_zfs] [ 5626.557298] [] lu_object_fini+0x5e/0xa0 [obdclass] [ 5626.560302] [] dt_object_fini+0xe/0x10 [obdclass] [ 5626.566956] [] osd_object_free+0x3f/0x180 [osd_zfs] [ 5626.571333] [] lu_object_free.isra.16+0x13d/0x1d0 [obdclass] [ 5626.575051] [] lu_object_put+0xb1/0x4c0 [obdclass] [ 5626.581683] [] lu_object_put_nocache+0x16/0x20 [obdclass] [ 5626.584010] [] osd_scrub_cleanup+0x3d/0xa0 [osd_zfs] [ 5626.593573] [] osd_device_fini+0x101/0x6b0 [osd_zfs] [ 5626.596266] [] obd_precleanup+0x143/0x2b0 [obdclass] [ 5626.599280] [] class_cleanup+0x23d/0x950 [obdclass] [ 5626.601813] [] ? do_raw_spin_unlock+0x49/0x90 [ 5626.604215] [] ? _raw_spin_unlock+0xe/0x20 [ 5626.606477] [] class_process_config+0x547/0x27d0 [obdclass] [ 5626.609517] [] ? __kmalloc+0x1e0/0x340 [ 5626.611618] [] class_manual_cleanup+0x1ce/0x770 [obdclass] [ 5626.614348] [] osd_obd_disconnect+0x185/0x200 [osd_zfs] [ 5626.617000] [] lustre_put_lsi_free+0x12d/0x410 [obdclass] [ 5626.619625] [] lustre_put_lsi+0x125/0x220 [obdclass] [ 5626.622285] [] server_put_super+0x159/0x1080 [ptlrpc] [ 5626.624616] [] generic_shutdown_super+0x6a/0xf0 [ 5626.627187] [] kill_anon_super+0x12/0x20 [ 5626.629859] [] lustre_kill_super+0x2b/0x30 [lustre] [ 5626.632604] [] deactivate_locked_super+0x49/0x60 [ 5626.641367] [] deactivate_super+0x46/0x60 [ 5626.642949] [] cleanup_mnt+0x3f/0x80 [ 5626.645216] [] __cleanup_mnt+0x12/0x20 [ 5626.650405] [] task_work_run+0xb5/0xf0 [ 5626.653424] [] do_notify_resume+0x92/0xb0 [ 5626.656291] [] int_signal+0x12/0x17 [ 5626.658460] Code: 00 00 55 48 8b 17 48 b9 00 01 00 00 00 00 ad de 48 8b 47 08 48 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a <4c> 8b 00 4c 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 [ 5626.667507] RIP [] __list_del_entry+0x29/0xd0 [ 5626.669921] RSP ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 11.54s (real) 6.51s (CPU), Child processes: 4.98s