************************ crashinfo ************************* /exports/testreports/42551/testresults/sanity-quota-zfs-centos7_x86_64-centos7_x86_64/oleg248-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.FTzRD/vmlinux [TAINTED] DUMPFILE: /exports/testreports/42551/testresults/sanity-quota-zfs-centos7_x86_64-centos7_x86_64/oleg248-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Wed May 8 19:03:01 EDT 2024 UPTIME: 01:02:13 LOAD AVERAGE: 0.00, 0.01, 0.05 TASKS: 396 NODENAME: oleg248-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 29 last 5s 64 last 60s 75 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 392 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 3207 monitor_thread 0 ms (no user stack) 3204 socknal_sd00_01 0 ms (no user stack) 6996 ldlm_bl_02 0 ms (no user stack) 3203 socknal_sd00_00 0 ms (no user stack) 3202 socknal_reaper 1 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 700229 2.7 GB 73% of TOTAL MEM USED 254838 995.5 MB 26% of TOTAL MEM SHARED 8412 32.9 MB 0% of TOTAL MEM BUFFERS 5096 19.9 MB 0% of TOTAL MEM CACHED 66045 258 MB 6% of TOTAL MEM SLAB 26202 102.4 MB 2% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 58984 230.4 MB 7% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 9 LISTEN 3 NAGLE disabled (TCP_NODELAY): 7 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- ESTABLISHED 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 3731.0 eth0 n/a 0.8 RSS_TOTAL=49324 pages, %mem= 0.8 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012a35e000 ffff88012a360000 sysfs sysfs /sys ffff88012a35e1c0 ffff880139944000 proc proc /proc ffff88012a35e380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012a35e540 ffff8800b5212000 securityfs securityfs /sys/kernel/security ffff88012a35e700 ffff88012a360800 tmpfs tmpfs /dev/shm ffff88012a35e8c0 ffff880137316000 devpts devpts /dev/pts ffff88012a35ea80 ffff88012a361000 tmpfs tmpfs /run ffff88012a35ec40 ffff88012a361800 tmpfs tmpfs /sys/fs/cgroup ffff88012a35ee00 ffff88012a362000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012a35efc0 ffff88012a362800 pstore pstore /sys/fs/pstore ffff88012a35f180 ffff88012a364800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a35f340 ffff88012a364000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012a35f500 ffff88012a363800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a35f6c0 ffff88012a363000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012a35f880 ffff88012a365000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a35fa40 ffff88012a365800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a35fc00 ffff88012a366000 cgroup cgroup /sys/fs/cgroup/devices ffff88012a35fdc0 ffff88012a366800 cgroup cgroup /sys/fs/cgroup/memory ffff88012a36e000 ffff88012a367000 cgroup cgroup /sys/fs/cgroup/pids ffff88012a36e1c0 ffff88012a367800 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a36e540 ffff8800b525d000 configfs configfs /sys/kernel/config ffff880138ccafc0 ffff880129e72800 ext4 /dev/nbd0 / ffff88012b285a40 ffff8800b525d800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012a36ec40 ffff8800b4db1800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880138ccb180 ffff880137317800 mqueue mqueue /dev/mqueue ffff88012a36ea80 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8801376688c0 ffff8800b4f60000 hugetlbfs hugetlbfs /dev/hugepages ffff88012a36e8c0 ffff8801373e6800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff88012b285c00 ffff880129e70800 ramfs none /mnt ffff88012b285dc0 ffff880129e75800 tmpfs none /var/lib/stateless/writable ffff880137668a80 ffff8800b428a800 squashfs /dev/vda /home/green/git/lustre-release ffff8800b4d30000 ffff880129e75800 tmpfs none /var/cache/man ffff88012a36efc0 ffff880129e75800 tmpfs none /var/log ffff88012a36f180 ffff880129e75800 tmpfs none /var/lib/dbus ffff88012a36f340 ffff880129e75800 tmpfs none /tmp ffff88012a36f500 ffff880129e75800 tmpfs none /var/lib/dhclient ffff880137668c40 ffff880129e75800 tmpfs none /var/tmp ffff880137668e00 ffff880129e75800 tmpfs none /var/lib/NetworkManager ffff880137668fc0 ffff880129e75800 tmpfs none /var/lib/systemd/random-seed ffff880137669180 ffff880129e75800 tmpfs none /var/spool ffff8800b4d301c0 ffff880129e75800 tmpfs none /var/lib/nfs ffff8800b4d30380 ffff880129e75800 tmpfs none /var/lib/gssproxy ffff880137669340 ffff880129e75800 tmpfs none /var/lib/logrotate ffff8800b4d30540 ffff880129e75800 tmpfs none /etc ffff8800b4d30700 ffff880129e75800 tmpfs none /var/lib/rsyslog ffff880137669500 ffff880129e75800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff8800b4d308c0 ffff8800b4db6800 nfs4 192.168.200.253:/exports/state/oleg248-server.virtnet /var/lib/stateless/state ffff88012a36f6c0 ffff8800b4db6800 nfs4 192.168.200.253:/exports/state/oleg248-server.virtnet /boot ffff8801376696c0 ffff8800b4db6800 nfs4 192.168.200.253:/exports/state/oleg248-server.virtnet /etc/etc/kdump.conf ffff8800b4d30e00 ffff8800b525d800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b28df180 ffff8800b428a800 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b299bc00 ffff88012bfa1800 lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 ffff8800b1c09500 ffff880136b30800 lustre lustre-ost1/ost1 /mnt/lustre-ost1 ffff8800b28dea80 ffff880094495000 lustre lustre-ost2/ost2 /mnt/lustre-ost2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 48.916517] Lustre: lustre-MDT0000: new disk, initializing [ 48.981135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 48.996284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 49.017518] random: crng init done [ 49.050564] mount.lustre (6910) used greatest stack depth: 10240 bytes left [ 49.708400] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 52.797544] Lustre: lustre-OST0000: new disk, initializing [ 52.799629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 52.801388] Lustre: Skipped 1 previous similar message [ 52.816334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 54.163524] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 56.177058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 56.181766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 56.211221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 57.265625] Lustre: lustre-OST0001: new disk, initializing [ 57.268315] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 57.286116] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 58.577925] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 63.400045] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 65.078547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 65.081094] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 65.105232] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 69.580448] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 75.218188] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing check_logdir /tmp/testlogs/ [ 76.036216] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing yml_node [ 77.021138] Lustre: DEBUG MARKER: Client: 2.15.63.5 [ 77.602724] Lustre: DEBUG MARKER: MDS: 2.15.63.5 [ 78.203003] Lustre: DEBUG MARKER: OSS: 2.15.63.5 [ 78.562058] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Wed May 8 18:02:05 EDT 2024 [ 82.056294] Lustre: DEBUG MARKER: excepting tests: 2 4a 63 65 [ 82.381511] Lustre: DEBUG MARKER: skipping tests SLOW=no: 61 12a 9 [ 83.080400] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing check_config_client /mnt/lustre [ 87.000148] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 87.838903] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 88.433930] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.858571] Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ===== 18:02:26 (1715205746) [ 131.228728] Lustre: DEBUG MARKER: == sanity-quota test 1a: Block hard limit (normal use and out of quota) ========================================================== 18:02:58 (1715205778) [ 138.682163] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 139.755316] Lustre: DEBUG MARKER: Write... [ 140.242993] Lustre: DEBUG MARKER: Write out of block quota ... ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 12.58s (real) 7.13s (CPU), Child processes: 5.43s