== replay-dual test 26: dbench and tar with mds failover ========================================================== 20:58:22 (1768010302) Starting client oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre Started clients oleg421-client.virtnet: 192.168.204.121@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44499 Started dbench loop with 44500 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jan 9 20:58:24 EST 2026 waiting for dbench pid 44534 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 150 6.28 MB/sec warmup 1 sec latency 43.814 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 27648 3710976 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3734528 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 30720 7445504 1% /mnt/lustre 1 275 5.12 MB/sec warmup 2 sec latency 43.786 ms 1 424 4.95 MB/sec warmup 3 sec latency 27.682 ms 1 562 4.78 MB/sec warmup 4 sec latency 34.732 ms test_26 fail mds1 1 times Failing mds1 on oleg421-server 1 695 4.24 MB/sec warmup 5 sec latency 107.567 ms Stopping /mnt/lustre-mds1 (opts:) on oleg421-server 1 731 3.55 MB/sec warmup 6 sec latency 555.493 ms 20:58:30 (1768010310) shut down facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server 1 731 3.04 MB/sec warmup 7 sec latency 1555.723 ms 1 731 2.66 MB/sec warmup 8 sec latency 2556.018 ms 1 731 2.37 MB/sec warmup 9 sec latency 3556.191 ms 1 731 2.13 MB/sec warmup 10 sec latency 4556.953 ms 1 731 1.94 MB/sec warmup 11 sec latency 5557.116 ms 1 731 1.78 MB/sec warmup 12 sec latency 6557.264 ms 1 731 1.64 MB/sec warmup 13 sec latency 7557.404 ms 1 731 1.52 MB/sec warmup 14 sec latency 8557.580 ms 1 731 1.42 MB/sec warmup 15 sec latency 9557.844 ms 1 731 1.33 MB/sec warmup 16 sec latency 10558.056 ms Failover mds1 to oleg421-server mount facets: mds1 1 731 1.25 MB/sec warmup 17 sec latency 11558.238 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 731 1.18 MB/sec warmup 18 sec latency 12558.410 ms 1 731 1.12 MB/sec warmup 19 sec latency 13558.585 ms oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 20:58:44 (1768010324) targets are mounted 20:58:44 (1768010324) facet_failover done 1 731 0.00 MB/sec execute 1 sec latency 15558.898 ms 1 731 0.00 MB/sec execute 2 sec latency 16559.117 ms 1 731 0.00 MB/sec execute 3 sec latency 17559.322 ms 1 731 0.00 MB/sec execute 4 sec latency 18559.468 ms 1 735 0.00 MB/sec execute 5 sec latency 19506.625 ms oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 865 0.04 MB/sec execute 6 sec latency 79.989 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 988 0.08 MB/sec execute 7 sec latency 53.241 ms 1 1159 0.21 MB/sec execute 8 sec latency 68.094 ms 1 1387 0.52 MB/sec execute 9 sec latency 32.020 ms 1 1511 0.47 MB/sec execute 10 sec latency 84.379 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3610624 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3631104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 59392 7241728 1% /mnt/lustre 1 1616 0.44 MB/sec execute 11 sec latency 61.770 ms 1 1808 0.42 MB/sec execute 12 sec latency 64.942 ms 1 2008 0.40 MB/sec execute 13 sec latency 47.624 ms test_26 fail mds1 2 times 1 2199 0.39 MB/sec execute 14 sec latency 25.862 ms Failing mds1 on oleg421-server Stopping /mnt/lustre-mds1 (opts:) on oleg421-server 1 2388 0.44 MB/sec execute 15 sec latency 85.415 ms 1 2389 0.42 MB/sec execute 16 sec latency 991.672 ms 20:59:00 (1768010340) shut down facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server 1 2389 0.39 MB/sec execute 17 sec latency 1991.920 ms 1 2389 0.37 MB/sec execute 18 sec latency 2992.134 ms 1 2389 0.35 MB/sec execute 19 sec latency 3992.372 ms 1 2389 0.33 MB/sec execute 20 sec latency 4992.583 ms 1 2389 0.32 MB/sec execute 21 sec latency 5992.857 ms 1 2389 0.30 MB/sec execute 22 sec latency 6993.077 ms 1 2389 0.29 MB/sec execute 23 sec latency 7993.255 ms 1 2389 0.28 MB/sec execute 24 sec latency 8993.401 ms 1 2389 0.27 MB/sec execute 25 sec latency 9993.577 ms 1 2389 0.26 MB/sec execute 26 sec latency 10993.742 ms Failover mds1 to oleg421-server mount facets: mds1 1 2389 0.25 MB/sec execute 27 sec latency 11993.857 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2389 0.24 MB/sec execute 28 sec latency 12993.986 ms oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all 1 2389 0.23 MB/sec execute 29 sec latency 13994.163 ms pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 20:59:13 (1768010353) targets are mounted 20:59:13 (1768010353) facet_failover done 1 2389 0.22 MB/sec execute 30 sec latency 14994.352 ms 1 2389 0.21 MB/sec execute 31 sec latency 15994.562 ms 1 2389 0.21 MB/sec execute 32 sec latency 16994.731 ms 1 2389 0.20 MB/sec execute 33 sec latency 17994.911 ms 1 2389 0.20 MB/sec execute 34 sec latency 18995.043 ms 1 2389 0.19 MB/sec execute 35 sec latency 19995.189 ms oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2477 0.21 MB/sec execute 36 sec latency 20116.176 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2566 0.21 MB/sec execute 37 sec latency 581.503 ms 1 2781 0.28 MB/sec execute 38 sec latency 784.939 ms 1 3010 0.31 MB/sec execute 39 sec latency 67.120 ms 1 3305 0.35 MB/sec execute 40 sec latency 43.490 ms 1 3584 0.42 MB/sec execute 41 sec latency 23.810 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3567616 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 35840 3601408 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 75776 7169024 2% /mnt/lustre 1 3748 0.43 MB/sec execute 42 sec latency 52.283 ms 1 3887 0.43 MB/sec execute 43 sec latency 34.611 ms 1 4055 0.43 MB/sec execute 44 sec latency 27.129 ms test_26 fail mds1 3 times Failing mds1 on oleg421-server 1 4236 0.42 MB/sec execute 45 sec latency 36.463 ms Stopping /mnt/lustre-mds1 (opts:) on oleg421-server 1 4302 0.42 MB/sec execute 46 sec latency 264.487 ms 20:59:31 (1768010371) shut down facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server 1 4302 0.41 MB/sec execute 47 sec latency 1264.607 ms 1 4302 0.40 MB/sec execute 48 sec latency 2264.809 ms 1 4302 0.39 MB/sec execute 49 sec latency 3264.978 ms 1 4302 0.38 MB/sec execute 50 sec latency 4265.149 ms 1 4302 0.37 MB/sec execute 51 sec latency 5265.315 ms 1 4302 0.37 MB/sec execute 52 sec latency 6265.519 ms 1 4302 0.36 MB/sec execute 53 sec latency 7265.708 ms 1 4302 0.35 MB/sec execute 54 sec latency 8265.885 ms 1 4302 0.35 MB/sec execute 55 sec latency 9266.046 ms 1 4302 0.34 MB/sec execute 56 sec latency 10266.191 ms Failover mds1 to oleg421-server mount facets: mds1 1 4302 0.34 MB/sec execute 57 sec latency 11266.357 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4302 0.33 MB/sec execute 58 sec latency 12266.512 ms 1 4302 0.32 MB/sec execute 59 sec latency 13266.657 ms oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all 1 4302 0.32 MB/sec execute 60 sec latency 14266.858 ms pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 20:59:44 (1768010384) targets are mounted 20:59:44 (1768010384) facet_failover done 1 4302 0.31 MB/sec execute 61 sec latency 15267.067 ms 1 4302 0.31 MB/sec execute 62 sec latency 16267.298 ms 1 4302 0.30 MB/sec execute 63 sec latency 17267.488 ms 1 4302 0.30 MB/sec execute 64 sec latency 18267.637 ms 1 4302 0.29 MB/sec execute 65 sec latency 19267.776 ms 1 4306 0.29 MB/sec execute 66 sec latency 20200.986 ms oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4440 0.29 MB/sec execute 67 sec latency 56.225 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4549 0.29 MB/sec execute 68 sec latency 84.277 ms 1 4756 0.30 MB/sec execute 69 sec latency 60.790 ms 1 4955 0.34 MB/sec execute 70 sec latency 272.997 ms 1 5118 0.34 MB/sec execute 71 sec latency 104.452 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6400 2202112 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 45056 3665920 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 37888 3706880 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 82944 7372800 2% /mnt/lustre 1 5256 0.33 MB/sec execute 72 sec latency 64.807 ms 1 5422 0.33 MB/sec execute 73 sec latency 24.104 ms 1 5613 0.33 MB/sec execute 74 sec latency 57.498 ms test_26 fail mds1 4 times 1 5834 0.33 MB/sec execute 75 sec latency 22.763 ms Failing mds1 on oleg421-server Stopping /mnt/lustre-mds1 (opts:) on oleg421-server 1 5965 0.34 MB/sec execute 76 sec latency 131.003 ms 21:00:01 (1768010401) shut down facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server 1 5965 0.34 MB/sec execute 77 sec latency 1117.522 ms 1 5965 0.33 MB/sec execute 78 sec latency 2117.668 ms 1 5965 0.33 MB/sec execute 79 sec latency 3117.833 ms 1 5965 0.32 MB/sec execute 80 sec latency 4118.141 ms 1 5965 0.32 MB/sec execute 81 sec latency 5118.388 ms 1 5965 0.32 MB/sec execute 82 sec latency 6118.569 ms 1 5965 0.31 MB/sec execute 83 sec latency 7118.717 ms 1 5965 0.31 MB/sec execute 84 sec latency 8118.876 ms 1 5965 0.30 MB/sec execute 85 sec latency 9119.062 ms 1 5965 0.30 MB/sec execute 86 sec latency 10119.237 ms Failover mds1 to oleg421-server mount facets: mds1 1 5965 0.30 MB/sec execute 87 sec latency 11119.359 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5965 0.29 MB/sec execute 88 sec latency 12119.575 ms 1 5965 0.29 MB/sec execute 89 sec latency 13119.772 ms oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all 1 5965 0.29 MB/sec execute 90 sec latency 14119.905 ms pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 21:00:14 (1768010414) targets are mounted 21:00:14 (1768010414) facet_failover done 1 5965 0.28 MB/sec execute 91 sec latency 15120.068 ms 1 5965 0.28 MB/sec execute 92 sec latency 16120.251 ms 1 5965 0.28 MB/sec execute 93 sec latency 17120.381 ms 1 5965 0.27 MB/sec execute 94 sec latency 18120.547 ms 1 5965 0.27 MB/sec execute 95 sec latency 19120.710 ms 1 5965 0.27 MB/sec execute 96 sec latency 20120.856 ms 1 6045 0.27 MB/sec execute 97 sec latency 20414.257 ms oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6429 0.31 MB/sec execute 98 sec latency 52.609 ms 1 6726 0.32 MB/sec execute 99 sec latency 110.390 ms 1 cleanup 100 sec 1 cleanup 101 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1023 34.568 20116.161 Close 789 28.443 20414.238 Rename 47 25.025 39.764 Unlink 175 8.298 32.453 Qpathinfo 979 4.691 90.649 Qfileinfo 185 0.916 4.782 Qfsinfo 172 0.545 3.018 Sfileinfo 96 428.218 20200.960 Find 392 2.684 80.840 WriteX 553 3.746 20.723 ReadX 1827 0.468 68.079 LockX 4 2.214 2.909 UnlockX 4 2.310 2.598 Flush 75 40.948 151.167 Throughput 0.322486 MB/sec 1 clients 1 procs max_latency=20414.257 ms stopping dbench on /mnt/lustre at Fri Jan 9 21:00:25 EST 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6528 2201984 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 45056 3663872 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 37888 3679232 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 82944 7343104 2% /mnt/lustre looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jan 9 21:00:28 EST 2026 waiting for dbench pid 48526 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 163 6.74 MB/sec warmup 1 sec latency 42.021 ms test_26 fail mds1 5 times 1 309 5.64 MB/sec warmup 2 sec latency 59.915 ms Failing mds1 on oleg421-server Stopping /mnt/lustre-mds1 (opts:) on oleg421-server 1 436 5.13 MB/sec warmup 3 sec latency 55.624 ms 21:00:32 (1768010432) shut down facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server 1 436 3.85 MB/sec warmup 4 sec latency 1055.732 ms 1 436 3.08 MB/sec warmup 5 sec latency 2055.895 ms 1 436 2.56 MB/sec warmup 6 sec latency 3056.097 ms 1 436 2.20 MB/sec warmup 7 sec latency 4056.347 ms 1 436 1.92 MB/sec warmup 8 sec latency 5056.532 ms 1 436 1.71 MB/sec warmup 9 sec latency 6056.715 ms 1 436 1.54 MB/sec warmup 10 sec latency 7056.940 ms 1 436 1.40 MB/sec warmup 11 sec latency 8057.177 ms 1 436 1.28 MB/sec warmup 12 sec latency 9057.333 ms 1 436 1.18 MB/sec warmup 13 sec latency 10057.497 ms 1 436 1.10 MB/sec warmup 14 sec latency 11057.609 ms Failover mds1 to oleg421-server mount facets: mds1 1 436 1.03 MB/sec warmup 15 sec latency 12057.755 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 436 0.96 MB/sec warmup 16 sec latency 13057.896 ms 1 436 0.90 MB/sec warmup 17 sec latency 14058.071 ms oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 21:00:45 (1768010445) targets are mounted 21:00:45 (1768010445) facet_failover done 1 436 0.85 MB/sec warmup 18 sec latency 15058.224 ms 1 436 0.81 MB/sec warmup 19 sec latency 16058.317 ms 1 436 0.00 MB/sec execute 1 sec latency 18058.560 ms 1 436 0.00 MB/sec execute 2 sec latency 19058.681 ms 1 501 0.69 MB/sec execute 3 sec latency 19269.304 ms oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 664 1.44 MB/sec execute 4 sec latency 43.157 ms 1 740 1.19 MB/sec execute 5 sec latency 189.944 ms 1 863 1.03 MB/sec execute 6 sec latency 84.198 ms 1 974 0.92 MB/sec execute 7 sec latency 73.681 ms 1 1145 0.95 MB/sec execute 8 sec latency 43.440 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 35840 3688448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 9216 3715072 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 45056 7403520 1% /mnt/lustre 1 1263 0.86 MB/sec execute 9 sec latency 84.134 ms 1 1484 1.07 MB/sec execute 10 sec latency 103.594 ms 1 1632 0.98 MB/sec execute 11 sec latency 35.426 ms test_26 fail mds1 6 times Failing mds1 on oleg421-server 1 1803 0.91 MB/sec execute 12 sec latency 57.109 ms Stopping /mnt/lustre-mds1 (opts:) on oleg421-server 1 1905 0.85 MB/sec execute 13 sec latency 325.732 ms 21:01:01 (1768010461) shut down facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server 1 1905 0.79 MB/sec execute 14 sec latency 1325.849 ms 1 1905 0.74 MB/sec execute 15 sec latency 2325.989 ms 1 1905 0.69 MB/sec execute 16 sec latency 3326.180 ms 1 1905 0.65 MB/sec execute 17 sec latency 4326.418 ms 1 1905 0.61 MB/sec execute 18 sec latency 5326.641 ms 1 1905 0.58 MB/sec execute 19 sec latency 6326.844 ms 1 1905 0.55 MB/sec execute 20 sec latency 7327.041 ms 1 1905 0.53 MB/sec execute 21 sec latency 8327.233 ms 1 1905 0.50 MB/sec execute 22 sec latency 9327.460 ms 1 1905 0.48 MB/sec execute 23 sec latency 10327.684 ms Failover mds1 to oleg421-server mount facets: mds1 1 1905 0.46 MB/sec execute 24 sec latency 11327.800 ms 1 1905 0.44 MB/sec execute 25 sec latency 12327.957 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1905 0.42 MB/sec execute 26 sec latency 13328.117 ms oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all 1 1905 0.41 MB/sec execute 27 sec latency 14328.286 ms pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 21:01:15 (1768010475) targets are mounted 21:01:15 (1768010475) facet_failover done 1 1905 0.39 MB/sec execute 28 sec latency 15328.466 ms 1 1905 0.38 MB/sec execute 29 sec latency 16328.620 ms 1 1905 0.37 MB/sec execute 30 sec latency 17328.758 ms 1 1905 0.36 MB/sec execute 31 sec latency 18328.912 ms 1 1905 0.35 MB/sec execute 32 sec latency 19329.060 ms 1 1905 0.33 MB/sec execute 33 sec latency 20329.175 ms oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2043 0.33 MB/sec execute 34 sec latency 20377.020 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2210 0.33 MB/sec execute 35 sec latency 37.938 ms 1 2376 0.35 MB/sec execute 36 sec latency 75.689 ms 1 2466 0.36 MB/sec execute 37 sec latency 47.231 ms 1 2566 0.36 MB/sec execute 38 sec latency 493.770 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 38912 3691520 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3718144 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 63488 7409664 1% /mnt/lustre 1 2783 0.43 MB/sec execute 39 sec latency 674.297 ms 1 3176 0.47 MB/sec execute 40 sec latency 51.788 ms 1 3399 0.49 MB/sec execute 41 sec latency 36.563 ms test_26 fail mds1 7 times 1 3592 0.55 MB/sec execute 42 sec latency 35.606 ms Failing mds1 on oleg421-server Stopping /mnt/lustre-mds1 (opts:) on oleg421-server 1 3724 0.54 MB/sec execute 43 sec latency 45.753 ms 21:01:32 (1768010492) shut down 1 3724 0.53 MB/sec execute 44 sec latency 1039.460 ms facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server 1 3724 0.52 MB/sec execute 45 sec latency 2039.615 ms 1 3724 0.51 MB/sec execute 46 sec latency 3039.748 ms 1 3724 0.50 MB/sec execute 47 sec latency 4039.956 ms 1 3724 0.49 MB/sec execute 48 sec latency 5040.094 ms 1 3724 0.48 MB/sec execute 49 sec latency 6040.233 ms 1 3724 0.47 MB/sec execute 50 sec latency 7040.382 ms 1 3724 0.46 MB/sec execute 51 sec latency 8040.511 ms 1 3724 0.45 MB/sec execute 52 sec latency 9040.645 ms 1 3724 0.44 MB/sec execute 53 sec latency 10040.786 ms 1 3724 0.43 MB/sec execute 54 sec latency 11040.945 ms Failover mds1 to oleg421-server mount facets: mds1 1 3724 0.43 MB/sec execute 55 sec latency 12041.067 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3724 0.42 MB/sec execute 56 sec latency 13041.214 ms 1 3724 0.41 MB/sec execute 57 sec latency 14041.358 ms oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 21:01:46 (1768010506) targets are mounted 21:01:46 (1768010506) facet_failover done 1 3724 0.40 MB/sec execute 58 sec latency 15041.470 ms 1 3724 0.40 MB/sec execute 59 sec latency 16041.599 ms 1 3724 0.39 MB/sec execute 60 sec latency 17041.768 ms 1 3724 0.38 MB/sec execute 61 sec latency 18041.951 ms 1 3724 0.38 MB/sec execute 62 sec latency 19042.120 ms 1 3724 0.37 MB/sec execute 63 sec latency 20042.259 ms 1 3821 0.38 MB/sec execute 64 sec latency 20307.508 ms oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3939 0.38 MB/sec execute 65 sec latency 32.585 ms 1 4076 0.37 MB/sec execute 66 sec latency 32.526 ms 1 4221 0.37 MB/sec execute 67 sec latency 33.343 ms 1 4299 0.37 MB/sec execute 68 sec latency 354.823 ms 1 4443 0.37 MB/sec execute 69 sec latency 43.329 ms 1 4557 0.37 MB/sec execute 70 sec latency 72.754 ms 1 4743 0.38 MB/sec execute 71 sec latency 34.569 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Jan 9 21:01:59 EST 2026 with return code 0 48526 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48526 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished