************************ crashinfo ************************* /exports/testreports/40120/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg302-client-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.3o1OE/vmlinux [TAINTED] DUMPFILE: /exports/testreports/40120/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64/oleg302-client-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Sun Feb 25 01:50:12 EST 2024 UPTIME: 01:23:38 LOAD AVERAGE: 0.35, 0.51, 0.56 TASKS: 148 NODENAME: oleg302-client.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 13 last 5s 35 last 60s 46 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 144 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 34 rcuos/3 0 ms (no user stack) 9 rcu_sched 0 ms (no user stack) 1 systemd 2 ms /usr/lib/systemd/systemd --switched-root --system --deserialize 24 11 rcuos/0 92 ms (no user stack) 83 kworker/0:2 92 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955079 3.6 GB ---- FREE 858229 3.3 GB 89% of TOTAL MEM USED 96850 378.3 MB 10% of TOTAL MEM SHARED 10262 40.1 MB 1% of TOTAL MEM BUFFERS 5159 20.2 MB 0% of TOTAL MEM CACHED 45569 178 MB 4% of TOTAL MEM SLAB 11863 46.3 MB 1% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739682 2.8 GB ---- COMMITTED 62792 245.3 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 6 TIME_WAIT 7 LISTEN 3 NAGLE disabled (TCP_NODELAY): 5 user_data set (NFS etc.): 4 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 18 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 5016.8 eth0 n/a 1.0 RSS_TOTAL=82080 pages, %mem= 1.3 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012aaba000 ffff88012aac0000 sysfs sysfs /sys ffff88012aaba1c0 ffff880139944000 proc proc /proc ffff88012aaba380 ffff880137668000 devtmpfs devtmpfs /dev ffff88012aaba540 ffff8800b6cd9800 securityfs securityfs /sys/kernel/security ffff88012aaba700 ffff88012aac0800 tmpfs tmpfs /dev/shm ffff88012aaba8c0 ffff880137315000 devpts devpts /dev/pts ffff88012aabaa80 ffff88012aac1000 tmpfs tmpfs /run ffff88012aabac40 ffff88012aac1800 tmpfs tmpfs /sys/fs/cgroup ffff88012aabae00 ffff88012aac2000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012aabafc0 ffff88012aac2800 pstore pstore /sys/fs/pstore ffff88012aabb180 ffff88012aac4800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012aabb340 ffff88012aac4000 cgroup cgroup /sys/fs/cgroup/pids ffff88012aabb500 ffff88012aac3800 cgroup cgroup /sys/fs/cgroup/memory ffff88012aabb6c0 ffff88012aac3000 cgroup cgroup /sys/fs/cgroup/freezer ffff88012aabb880 ffff88012aac5000 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012aabba40 ffff88012aac5800 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012aabbc00 ffff88012aac6000 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012aabbdc0 ffff88012aac6800 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012aace000 ffff88012aac7000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012aace1c0 ffff88012aac7800 cgroup cgroup /sys/fs/cgroup/devices ffff88012aace540 ffff8800b6e19000 configfs configfs /sys/kernel/config ffff88012aace700 ffff8800b6e18000 ext4 /dev/nbd0 / ffff880138ccb6c0 ffff8800b6e1a800 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff8801377de1c0 ffff8800b6378800 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880137652540 ffff8800b606a800 hugetlbfs hugetlbfs /dev/hugepages ffff880138ccb880 ffff880137316000 mqueue mqueue /dev/mqueue ffff8801377de380 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8801377de540 ffff88012a4a3800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff8801377de8c0 ffff88012e163000 ramfs none /mnt ffff88012aacee00 ffff8800b61ca000 tmpfs none /var/lib/stateless/writable ffff880138ccba40 ffff8800b6cda000 squashfs /dev/vda /home/green/git/lustre-release ffff88012aacefc0 ffff8800b61ca000 tmpfs none /var/cache/man ffff8801377dea80 ffff8800b61ca000 tmpfs none /var/log ffff88012aacf180 ffff8800b61ca000 tmpfs none /var/lib/dbus ffff880137652700 ffff8800b61ca000 tmpfs none /tmp ffff880138ccbc00 ffff8800b61ca000 tmpfs none /var/lib/dhclient ffff8801376528c0 ffff8800b61ca000 tmpfs none /var/tmp ffff880137652a80 ffff8800b61ca000 tmpfs none /var/lib/NetworkManager ffff880137652c40 ffff8800b61ca000 tmpfs none /var/lib/systemd/random-seed ffff880138ccbdc0 ffff8800b61ca000 tmpfs none /var/spool ffff880137652e00 ffff8800b61ca000 tmpfs none /var/lib/nfs ffff880137652fc0 ffff8800b61ca000 tmpfs none /var/lib/gssproxy ffff880137653180 ffff8800b61ca000 tmpfs none /var/lib/logrotate ffff880138ccb500 ffff8800b61ca000 tmpfs none /etc ffff88012aacf340 ffff8800b61ca000 tmpfs none /var/lib/rsyslog ffff88012aacf500 ffff8800b61ca000 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012aacf6c0 ffff88012b4bf000 nfs4 192.168.200.253:/exports/state/oleg302-client.virtnet /var/lib/stateless/state ffff88012aacfa40 ffff88012b4bf000 nfs4 192.168.200.253:/exports/state/oleg302-client.virtnet /boot ffff8801377defc0 ffff88012b4bf000 nfs4 192.168.200.253:/exports/state/oleg302-client.virtnet /etc/etc/kdump.conf ffff88012aacfc00 ffff8800b6e1a800 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff8800b59b6e00 ffff88012b494800 nfs4 192.168.200.253://exports/testreports/40120/testresults/conf-sanity-slow-106-zfs-centos7_x86_64-centos7_x86_64 /tmp/tmp/testlogs ffff8801377dfa40 ffff8800af91d000 tmpfs tmpfs /run/user/0 ffff8800b42fc540 ffff8800b6cda000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b59b7180 ffff88012e165000 lustre 192.168.203.102@tcp:/lustre /mnt/lustre +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 6.230855] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 6.596140] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 6.799043] SELinux: Disabled at runtime. [ 6.800121] SELinux: Unregistering netfilter hooks [ 6.854327] ip_tables: (C) 2000-2006 Netfilter Core Team [ 6.856473] systemd[1]: Inserted module 'ip_tables' [ 7.148729] systemd-journald[561]: Received request to flush runtime journal from PID 1 [ 7.266267] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 7.282208] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 7.353694] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 7.368809] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.599014] mount.nfs (728) used greatest stack depth: 10728 bytes left [ 22.815075] libcfs: loading out-of-tree module taints kernel. [ 22.816967] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 22.932604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 22.944457] alg: No test for adler32 (adler32-zlib) [ 23.201089] PCLMULQDQ-NI instructions are not detected. [ 23.821822] Lustre: Lustre: Build Version: 2.15.61_76_g3d34ee8 [ 23.979303] LNet: Added LNI 192.168.203.2@tcp [8/256/0/180] [ 23.981579] LNet: Accept secure, port 988 [ 25.523612] Key type lgssc registered [ 25.783686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 57.522201] Lustre: Mounted lustre-client [ 59.546232] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 69.012053] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing check_logdir /tmp/testlogs/ [ 70.157472] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing yml_node [ 71.150875] Lustre: DEBUG MARKER: Client: 2.15.61.76 [ 71.748275] Lustre: DEBUG MARKER: MDS: 2.15.61.76 [ 72.333164] Lustre: DEBUG MARKER: OSS: 2.15.61.76 [ 72.667086] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Feb 25 00:27:45 EST 2024 [ 74.021451] Lustre: DEBUG MARKER: excepting tests: 32newtarball 110 [ 74.221313] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 84.254550] Lustre: Unmounted lustre-client [ 112.303353] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 117.464121] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 124.351627] Lustre: DEBUG MARKER: == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:28:37 (1708838917) [ 140.587569] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 145.335803] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 145.438903] Lustre: Mounted lustre-client [ 146.942382] random: crng init done ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 10.26s (real) 5.24s (CPU), Child processes: 4.99s