************************ crashinfo ************************* /exports/testreports/43114/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg103-server-timeout-core (3.10.0-7.9-debug) +==========================+ | *** Crashinfo v1.3.7 *** | +==========================+ +++WARNING+++ PARTIAL DUMP with size(vmcore) < 25% size(RAM) KERNEL: /tmp/crash-anaysis.s2lOG/vmlinux [TAINTED] DUMPFILE: /exports/testreports/43114/testresults/racer-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/oleg103-server-timeout-core [PARTIAL DUMP] CPUS: 4 DATE: Tue Jun 4 23:20:14 EDT 2024 UPTIME: 00:17:04 LOAD AVERAGE: 0.35, 10.52, 9.08 TASKS: 538 NODENAME: oleg103-server.virtnet RELEASE: 3.10.0-7.9-debug VERSION: #1 SMP Sat Mar 26 23:28:42 EDT 2022 MACHINE: x86_64 (2399 Mhz) MEMORY: 4 GB PANIC: "" +--------------------------+ >------------------------| Per-cpu Stacks ('bt -a') |------------------------< +--------------------------+ -- CPU#0 -- PID=0 CPU=0 CMD=swapper/0 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 rest_init+0x8e #7 start_kernel+0x456 #8 x86_64_start_reservations+0x2a #9 x86_64_start_kernel+0x152 #10 start_cpu+0x5 -- CPU#1 -- PID=0 CPU=1 CMD=swapper/1 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#2 -- PID=0 CPU=2 CMD=swapper/2 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 -- CPU#3 -- PID=0 CPU=3 CMD=swapper/3 #-1 native_safe_halt+0xb, 449 bytes of data #0 default_idle+0x1e #1 default_enter_idle+0x45 #2 cpuidle_enter_state+0x40 #3 cpuidle_idle_call+0xd8 #4 arch_cpu_idle+0xe #5 cpu_startup_entry+0x14a #6 start_secondary+0x1eb #7 start_cpu+0x5 +--------------------------------+ >---------------------| How This Dump Has Been Created |---------------------< +--------------------------------+ Cannot identify the specific condition that triggered vmcore +---------------+ >------------------------------| Tasks Summary |------------------------------< +---------------+ Number of Threads That Ran Recently ----------------------------------- last second 31 last 5s 69 last 60s 79 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 534 TASK_RUNNING 1 +++WARNING+++ There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details +-----------------------+ >--------------------------| 5 Most Recent Threads |--------------------------< +-----------------------+ PID CMD Age ARGS ----- -------------- ------ ---------------------------- 46 kworker/0:1 0 ms (no user stack) 8265 ldlm_bl_03 0 ms (no user stack) 804 kworker/2:2 0 ms (no user stack) 907 in:imjournal 0 ms /usr/sbin/rsyslogd -n 3680 monitor_thread 0 ms (no user stack) +------------------------+ >-------------------------| Memory Usage (kmem -i) |-------------------------< +------------------------+ PAGES TOTAL PERCENTAGE TOTAL MEM 955067 3.6 GB ---- FREE 493020 1.9 GB 51% of TOTAL MEM USED 462047 1.8 GB 48% of TOTAL MEM SHARED 88616 346.2 MB 9% of TOTAL MEM BUFFERS 86237 336.9 MB 9% of TOTAL MEM CACHED 71892 280.8 MB 7% of TOTAL MEM SLAB 121265 473.7 MB 12% of TOTAL MEM TOTAL HUGE 0 0 ---- HUGE FREE 0 0 0% of TOTAL HUGE TOTAL SWAP 262143 1024 MB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 262143 1024 MB 100% of TOTAL SWAP COMMIT LIMIT 739676 2.8 GB ---- COMMITTED 59999 234.4 MB 8% of TOTAL LIMIT +-------------------------------+ >----------------------| Scheduler Runqueues (per CPU) |----------------------< +-------------------------------+ ---+ CPU=0 ---- | CURRENT TASK , CMD=swapper/0 ---+ CPU=1 ---- | CURRENT TASK , CMD=swapper/1 ---+ CPU=2 ---- | CURRENT TASK , CMD=swapper/2 ---+ CPU=3 ---- | CURRENT TASK , CMD=swapper/3 +------------------------+ >-------------------------| Network Status Summary |-------------------------< +------------------------+ TCP Connection Info ------------------- ESTABLISHED 9 LISTEN 3 NAGLE disabled (TCP_NODELAY): 7 user_data set (NFS etc.): 8 UDP Connection Info ------------------- 2 UDP sockets, 0 in ESTABLISHED Unix Connection Info ------------------------ ESTABLISHED 26 CLOSE 17 LISTEN 8 Raw sockets info -------------------- CLOSE 1 Interfaces Info --------------- How long ago (in seconds) interfaces transmitted/received? Name RX TX ---- ---------- --------- lo n/a 1022.1 eth0 n/a 2.1 RSS_TOTAL=54452 pages, %mem= 0.9 +------------+ >-------------------------------| Mounted FS |-------------------------------< +------------+ MOUNT SUPERBLK TYPE DEVNAME DIRNAME ffff880138cca000 ffff880139940800 rootfs rootfs / ffff88012a294000 ffff88012b385000 sysfs sysfs /sys ffff88012a2941c0 ffff880139944000 proc proc /proc ffff88012a294380 ffff880137678000 devtmpfs devtmpfs /dev ffff88012a294540 ffff8800b5200800 securityfs securityfs /sys/kernel/security ffff88012a294700 ffff88012b385800 tmpfs tmpfs /dev/shm ffff88012a2948c0 ffff88012d451000 devpts devpts /dev/pts ffff88012a294a80 ffff88012b386000 tmpfs tmpfs /run ffff88012a294c40 ffff88012b386800 tmpfs tmpfs /sys/fs/cgroup ffff88012a294e00 ffff88012b387000 cgroup cgroup /sys/fs/cgroup/systemd ffff88012a294fc0 ffff88012b387800 pstore pstore /sys/fs/pstore ffff88012a295180 ffff88012a339800 cgroup cgroup /sys/fs/cgroup/devices ffff88012a295340 ffff88012a339000 cgroup cgroup /sys/fs/cgroup/hugetlb ffff88012a295500 ffff88012a338800 cgroup cgroup /sys/fs/cgroup/cpuset ffff88012a2956c0 ffff88012a338000 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio ffff88012a295880 ffff88012a33a000 cgroup cgroup /sys/fs/cgroup/blkio ffff88012a295a40 ffff88012a33a800 cgroup cgroup /sys/fs/cgroup/freezer ffff88012a295c00 ffff88012a33b000 cgroup cgroup /sys/fs/cgroup/pids ffff88012a295dc0 ffff88012a33b800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct ffff88012b39e000 ffff88012a33c000 cgroup cgroup /sys/fs/cgroup/perf_event ffff88012b39e1c0 ffff88012a33c800 cgroup cgroup /sys/fs/cgroup/memory ffff88012b39e540 ffff88012a33f000 configfs configfs /sys/kernel/config ffff88012b39e700 ffff8800b4ebf800 ext4 /dev/nbd0 / ffff88012b39e8c0 ffff88012a33d000 rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs ffff88012b24aa80 ffff8800b5202000 autofs systemd-1 /proc/sys/fs/binfmt_misc ffff880137669500 ffff880139947800 debugfs debugfs /sys/kernel/debug ffff8801376696c0 ffff88012d452800 mqueue mqueue /dev/mqueue ffff880138ccb6c0 ffff8800b4c66800 hugetlbfs hugetlbfs /dev/hugepages ffff88012b24ac40 ffff88012e121800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ ffff880137669880 ffff880129cd8000 ramfs none /mnt ffff880137669a40 ffff8800b4eb9000 squashfs /dev/vda /home/green/git/lustre-release ffff880138ccba40 ffff8800b524d800 tmpfs none /var/lib/stateless/writable ffff88012b39ee00 ffff8800b524d800 tmpfs none /var/cache/man ffff88012b39efc0 ffff8800b524d800 tmpfs none /var/log ffff880137669dc0 ffff8800b524d800 tmpfs none /var/lib/dbus ffff880129d7c000 ffff8800b524d800 tmpfs none /tmp ffff88012b39f180 ffff8800b524d800 tmpfs none /var/lib/dhclient ffff88012b24ae00 ffff8800b524d800 tmpfs none /var/tmp ffff88012b39f340 ffff8800b524d800 tmpfs none /var/lib/NetworkManager ffff88012b24afc0 ffff8800b524d800 tmpfs none /var/lib/systemd/random-seed ffff880129d7c1c0 ffff8800b524d800 tmpfs none /var/spool ffff88012b39f500 ffff8800b524d800 tmpfs none /var/lib/nfs ffff88012b39f6c0 ffff8800b524d800 tmpfs none /var/lib/gssproxy ffff88012b39f880 ffff8800b524d800 tmpfs none /var/lib/logrotate ffff88012b24b180 ffff8800b524d800 tmpfs none /etc ffff880129d7c380 ffff8800b524d800 tmpfs none /var/lib/rsyslog ffff88012b39fa40 ffff8800b524d800 tmpfs none /var/lib/dhclient/var/lib/dhclient ffff88012b24b340 ffff88012d172000 nfs4 192.168.200.253:/exports/state/oleg103-server.virtnet /var/lib/stateless/state ffff88012b39fdc0 ffff88012d172000 nfs4 192.168.200.253:/exports/state/oleg103-server.virtnet /boot ffff880138ccbc00 ffff88012d172000 nfs4 192.168.200.253:/exports/state/oleg103-server.virtnet /etc/etc/kdump.conf ffff88012b39fc00 ffff88012a33d000 rpc_pipefs sunrpc /var/lib/nfs/var/lib/nfs/rpc_pipefs ffff880129d7ce00 ffff8800b4eb9000 squashfs /dev/vda /usr/sbin/mount.lustre ffff8800b26cf500 ffff880129e77000 lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 ffff8800b26cf340 ffff88012eb27000 lustre /dev/mapper/mds2_flakey /mnt/lustre-mds2 ffff8800b26cec40 ffff880129ded000 lustre /dev/mapper/ost1_flakey /mnt/lustre-ost1 ffff8800b26ce000 ffff88012c8e7000 lustre /dev/mapper/ost2_flakey /mnt/lustre-ost2 +-------------------------------+ >----------------------| Last 40 lines of dmesg buffer |----------------------< +-------------------------------+ [ 785.567828] LustreError: 7124:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 785.661083] LustreError: 15652:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 787.276972] LustreError: 15652:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 787.739603] LustreError: 15479:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 788.432939] Lustre: 15422:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880079eb2a00 x1800988873524928/t4295121679(0) o101->1dd7c598-43de-43ec-a645-6ba6bb99f1f1@192.168.201.3@tcp:469/0 lens 376/47888 e 0 to 0 dl 1717557519 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0 [ 788.646084] LustreError: 15482:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 788.707154] LustreError: 7124:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 788.759775] LustreError: 15601:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 788.881869] LustreError: 15411:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 789.041729] LustreError: 10617:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 789.089134] LustreError: 10617:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 789.137344] LustreError: 10617:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 789.697197] LustreError: 15409:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 790.248987] LustreError: 15411:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 791.074795] LustreError: 15601:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 792.746541] LustreError: 15417:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 793.390119] LustreError: 15422:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 793.643402] LustreError: 15424:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 793.971838] LustreError: 15424:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 794.090633] Lustre: 15409:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880083ff0e00 x1800988878297280/t4295112888(0) o101->1dd7c598-43de-43ec-a645-6ba6bb99f1f1@192.168.201.3@tcp:476/0 lens 376/48136 e 0 to 0 dl 1717557526 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0 [ 794.268907] LustreError: 15409:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 794.438606] LustreError: 15424:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 794.628908] LustreError: 15424:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 794.965880] LustreError: 15410:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 795.970444] LustreError: 15409:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 796.388106] LustreError: 15485:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 796.486714] LustreError: 15410:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 796.496073] LustreError: 15422:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 796.502501] LustreError: 15411:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 796.616057] LustreError: 15482:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000402:0x4393:0x0] doesn't exist!: rc = -14 [ 796.816936] LustreError: 15411:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 796.943871] LustreError: 15411:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 797.077161] LustreError: 10617:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 797.367613] LustreError: 10617:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 797.511600] LustreError: 8766:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 798.351675] Lustre: 15486:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88007e095500 x1800988881079936/t4295113487(0) o101->1dd7c598-43de-43ec-a645-6ba6bb99f1f1@192.168.201.3@tcp:479/0 lens 376/48136 e 0 to 0 dl 1717557529 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0 [ 798.381842] LustreError: 15650:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 798.409431] LustreError: 15601:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 798.696695] LustreError: 15650:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 [ 798.988953] LustreError: 15601:0:(/home/green/git/lustre-release/lustre/mdt/mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x470d:0x0] doesn't exist!: rc = -14 ****************************************************************************** ************************ A Summary Of Problems Found ************************* ****************************************************************************** -------------------- A list of all +++WARNING+++ messages -------------------- PARTIAL DUMP with size(vmcore) < 25% size(RAM) There are 3 threads running in their own namespaces Use 'taskinfo --ns' to get more details ------------------------------------------------------------------------------ ** Execution took 12.60s (real) 6.96s (CPU), Child processes: 5.58s