== replay-dual test 26: dbench and tar with mds failover ========================================================== 19:05:53 (1787871953) Starting client oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre Started clients oleg413-client.virtnet: 192.168.204.113@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45192 Started dbench loop with 45195 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3738624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7500800 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Aug 27 19:05:59 EDT 2026 waiting for dbench pid 45262 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 117.721 ms 1 71 1.57 MB/sec warmup 2 sec latency 138.699 ms 1 113 1.75 MB/sec warmup 3 sec latency 223.990 ms 1 139 1.50 MB/sec warmup 4 sec latency 160.671 ms test_26 fail mds1 1 times 1 166 1.36 MB/sec warmup 5 sec latency 182.001 ms 1 189 1.26 MB/sec warmup 6 sec latency 290.996 ms 1 207 1.15 MB/sec warmup 7 sec latency 313.562 ms Failing mds1 on oleg413-server 1 238 1.14 MB/sec warmup 8 sec latency 234.413 ms Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 1 264 1.10 MB/sec warmup 9 sec latency 165.730 ms 1 278 1.04 MB/sec warmup 10 sec latency 577.682 ms 1 278 0.95 MB/sec warmup 11 sec latency 1579.652 ms 1 278 0.87 MB/sec warmup 12 sec latency 2580.409 ms 1 278 0.80 MB/sec warmup 13 sec latency 3580.926 ms 19:06:13 (1787871973) shut down facet: mds1 facet_host: oleg413-server facet_failover_host: oleg413-server 1 278 0.74 MB/sec warmup 14 sec latency 4581.218 ms 1 278 0.69 MB/sec warmup 15 sec latency 5582.960 ms 1 278 0.65 MB/sec warmup 16 sec latency 6585.863 ms 1 278 0.61 MB/sec warmup 17 sec latency 7586.289 ms 1 278 0.58 MB/sec warmup 18 sec latency 8592.131 ms 1 278 0.55 MB/sec warmup 19 sec latency 9592.634 ms 1 278 0.00 MB/sec execute 1 sec latency 11594.148 ms 1 278 0.00 MB/sec execute 2 sec latency 12595.111 ms 1 278 0.00 MB/sec execute 3 sec latency 13598.953 ms 1 278 0.00 MB/sec execute 4 sec latency 14599.928 ms Failover mds1 to oleg413-server mount facets: mds1 1 278 0.00 MB/sec execute 5 sec latency 15601.071 ms 1 278 0.00 MB/sec execute 6 sec latency 16601.677 ms 1 278 0.00 MB/sec execute 7 sec latency 17602.505 ms 1 278 0.00 MB/sec execute 8 sec latency 18602.779 ms 1 278 0.00 MB/sec execute 9 sec latency 19603.000 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 278 0.00 MB/sec execute 10 sec latency 20603.214 ms 1 278 0.00 MB/sec execute 11 sec latency 21603.423 ms 1 278 0.00 MB/sec execute 12 sec latency 22605.687 ms 1 278 0.00 MB/sec execute 13 sec latency 23605.958 ms 1 278 0.00 MB/sec execute 14 sec latency 24606.999 ms 1 278 0.00 MB/sec execute 15 sec latency 25608.856 ms 1 278 0.00 MB/sec execute 16 sec latency 26609.167 ms 1 278 0.00 MB/sec execute 17 sec latency 27618.123 ms 1 278 0.00 MB/sec execute 18 sec latency 28620.070 ms 1 278 0.00 MB/sec execute 19 sec latency 29620.506 ms 1 278 0.00 MB/sec execute 20 sec latency 30620.861 ms 1 278 0.00 MB/sec execute 21 sec latency 31621.401 ms 1 278 0.00 MB/sec execute 22 sec latency 32625.360 ms 1 278 0.00 MB/sec execute 23 sec latency 33625.693 ms oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all 1 278 0.00 MB/sec execute 24 sec latency 34627.052 ms pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 1 278 0.00 MB/sec execute 25 sec latency 35627.532 ms Started lustre-MDT0000 19:06:45 (1787872005) targets are mounted 19:06:45 (1787872005) facet_failover done 1 278 0.00 MB/sec execute 26 sec latency 36630.729 ms 1 278 0.00 MB/sec execute 27 sec latency 37636.765 ms 1 278 0.00 MB/sec execute 28 sec latency 38637.277 ms 1 278 0.00 MB/sec execute 29 sec latency 39637.458 ms 1 278 0.00 MB/sec execute 30 sec latency 40637.780 ms 1 279 0.00 MB/sec execute 31 sec latency 41608.418 ms 1 302 0.02 MB/sec execute 32 sec latency 297.316 ms 1 334 0.05 MB/sec execute 33 sec latency 157.806 ms 1 372 0.08 MB/sec execute 34 sec latency 125.843 ms 1 406 0.11 MB/sec execute 35 sec latency 151.218 ms 1 438 0.14 MB/sec execute 36 sec latency 159.343 ms oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 478 0.17 MB/sec execute 37 sec latency 126.181 ms 1 511 0.19 MB/sec execute 38 sec latency 118.739 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 599 0.27 MB/sec execute 39 sec latency 130.009 ms 1 650 0.27 MB/sec execute 40 sec latency 88.418 ms 1 671 0.26 MB/sec execute 41 sec latency 346.544 ms 1 699 0.26 MB/sec execute 42 sec latency 118.875 ms 1 723 0.25 MB/sec execute 43 sec latency 320.186 ms 1 738 0.25 MB/sec execute 44 sec latency 233.952 ms 1 774 0.24 MB/sec execute 45 sec latency 130.062 ms 1 791 0.24 MB/sec execute 46 sec latency 386.666 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 34816 3708928 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 12288 3695616 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 47104 7404544 1% /mnt/lustre 1 806 0.23 MB/sec execute 47 sec latency 162.785 ms 1 851 0.23 MB/sec execute 48 sec latency 120.118 ms 1 876 0.23 MB/sec execute 49 sec latency 251.301 ms 1 905 0.22 MB/sec execute 50 sec latency 169.926 ms 1 924 0.22 MB/sec execute 51 sec latency 201.966 ms 1 974 0.22 MB/sec execute 52 sec latency 71.370 ms test_26 fail mds1 2 times 1 996 0.22 MB/sec execute 53 sec latency 223.751 ms 1 1060 0.23 MB/sec execute 54 sec latency 98.721 ms 1 1101 0.23 MB/sec execute 55 sec latency 118.832 ms Failing mds1 on oleg413-server 1 1138 0.22 MB/sec execute 56 sec latency 108.247 ms Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 1 1173 0.22 MB/sec execute 57 sec latency 136.656 ms 1 1188 0.22 MB/sec execute 58 sec latency 456.474 ms 1 1188 0.21 MB/sec execute 59 sec latency 1456.825 ms 1 1188 0.21 MB/sec execute 60 sec latency 2457.706 ms 19:07:20 (1787872040) shut down facet: mds1 facet_host: oleg413-server facet_failover_host: oleg413-server 1 1188 0.21 MB/sec execute 61 sec latency 3457.947 ms 1 1188 0.20 MB/sec execute 62 sec latency 4458.795 ms 1 1188 0.20 MB/sec execute 63 sec latency 5459.158 ms 1 1188 0.20 MB/sec execute 64 sec latency 6462.302 ms 1 1188 0.19 MB/sec execute 65 sec latency 7464.691 ms 1 1188 0.19 MB/sec execute 66 sec latency 8466.083 ms 1 1188 0.19 MB/sec execute 67 sec latency 9466.422 ms 1 1188 0.18 MB/sec execute 68 sec latency 10466.698 ms 1 1188 0.18 MB/sec execute 69 sec latency 11467.030 ms 1 1188 0.18 MB/sec execute 70 sec latency 12470.177 ms 1 1188 0.18 MB/sec execute 71 sec latency 13470.374 ms Failover mds1 to oleg413-server mount facets: mds1 1 1188 0.17 MB/sec execute 72 sec latency 14472.615 ms 1 1188 0.17 MB/sec execute 73 sec latency 15473.132 ms 1 1188 0.17 MB/sec execute 74 sec latency 16473.743 ms 1 1188 0.17 MB/sec execute 75 sec latency 17474.026 ms 1 1188 0.17 MB/sec execute 76 sec latency 18475.130 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1188 0.16 MB/sec execute 77 sec latency 19475.716 ms 1 1188 0.16 MB/sec execute 78 sec latency 20477.096 ms 1 1188 0.16 MB/sec execute 79 sec latency 21478.534 ms 1 1188 0.16 MB/sec execute 80 sec latency 22478.783 ms 1 1188 0.16 MB/sec execute 81 sec latency 23479.082 ms 1 1188 0.15 MB/sec execute 82 sec latency 24479.388 ms 1 1188 0.15 MB/sec execute 83 sec latency 25479.616 ms 1 1188 0.15 MB/sec execute 84 sec latency 26479.883 ms 1 1188 0.15 MB/sec execute 85 sec latency 27482.501 ms 1 1188 0.15 MB/sec execute 86 sec latency 28482.904 ms 1 1188 0.14 MB/sec execute 87 sec latency 29483.107 ms 1 1188 0.14 MB/sec execute 88 sec latency 30483.446 ms 1 1188 0.14 MB/sec execute 89 sec latency 31483.651 ms 1 1188 0.14 MB/sec execute 90 sec latency 32484.257 ms 1 1188 0.14 MB/sec execute 91 sec latency 33486.889 ms 1 1188 0.14 MB/sec execute 92 sec latency 34487.112 ms oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all 1 1188 0.14 MB/sec execute 93 sec latency 35487.381 ms pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 1 1188 0.13 MB/sec execute 94 sec latency 36487.592 ms 1 1188 0.13 MB/sec execute 95 sec latency 37490.189 ms Started lustre-MDT0000 19:07:55 (1787872075) targets are mounted 19:07:55 (1787872075) facet_failover done 1 1188 0.13 MB/sec execute 96 sec latency 38491.834 ms 1 1188 0.13 MB/sec execute 97 sec latency 39495.939 ms 1 1188 0.13 MB/sec execute 98 sec latency 40496.788 ms 1 1212 0.13 MB/sec execute 99 sec latency 40589.990 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 139 82.961 297.297 Close 129 652.739 41608.393 Rename 9 107.137 159.324 Unlink 14 36.263 91.275 Qpathinfo 84 26.439 138.996 Qfileinfo 59 2.408 11.506 Qfsinfo 28 4.574 31.661 Sfileinfo 30 66.344 133.248 Find 37 9.403 59.764 WriteX 284 14.257 76.934 ReadX 128 5.917 101.564 Flush 24 161.977 386.646 Throughput 0.127163 MB/sec 1 clients 1 procs max_latency=41608.418 ms stopping dbench on /mnt/lustre at Thu Aug 27 19:08:04 EDT 2026 with return code 0 oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Aug 27 19:08:11 EDT 2026 waiting for dbench pid 47218 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 15 0.07 MB/sec warmup 1 sec latency 18.124 ms 1 64 1.37 MB/sec warmup 2 sec latency 129.166 ms 1 106 1.66 MB/sec warmup 3 sec latency 139.193 ms 1 133 1.45 MB/sec warmup 4 sec latency 190.576 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3725312 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3731456 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 47104 7456768 1% /mnt/lustre 1 155 1.31 MB/sec warmup 5 sec latency 230.487 ms 1 184 1.23 MB/sec warmup 6 sec latency 145.168 ms 1 215 1.19 MB/sec warmup 7 sec latency 170.330 ms 1 252 1.19 MB/sec warmup 8 sec latency 140.036 ms 1 290 1.19 MB/sec warmup 9 sec latency 124.138 ms 1 321 1.17 MB/sec warmup 10 sec latency 151.187 ms test_26 fail mds1 3 times 1 357 1.16 MB/sec warmup 11 sec latency 147.836 ms 1 389 1.15 MB/sec warmup 12 sec latency 178.687 ms Failing mds1 on oleg413-server 1 421 1.14 MB/sec warmup 13 sec latency 229.749 ms Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 1 453 1.13 MB/sec warmup 14 sec latency 161.591 ms 1 482 1.12 MB/sec warmup 15 sec latency 194.280 ms 1 485 1.06 MB/sec warmup 16 sec latency 953.675 ms 1 485 0.99 MB/sec warmup 17 sec latency 1953.964 ms 19:08:29 (1787872109) shut down facet: mds1 facet_host: oleg413-server facet_failover_host: oleg413-server 1 485 0.94 MB/sec warmup 18 sec latency 2954.540 ms 1 485 0.89 MB/sec warmup 19 sec latency 3955.140 ms 1 485 0.00 MB/sec execute 1 sec latency 5962.218 ms 1 485 0.00 MB/sec execute 2 sec latency 6966.576 ms 1 485 0.00 MB/sec execute 3 sec latency 7967.380 ms 1 485 0.00 MB/sec execute 4 sec latency 8969.201 ms 1 485 0.00 MB/sec execute 5 sec latency 9969.521 ms 1 485 0.00 MB/sec execute 6 sec latency 10971.205 ms 1 485 0.00 MB/sec execute 7 sec latency 11971.531 ms 1 485 0.00 MB/sec execute 8 sec latency 12972.681 ms Failover mds1 to oleg413-server mount facets: mds1 1 485 0.00 MB/sec execute 9 sec latency 13972.839 ms 1 485 0.00 MB/sec execute 10 sec latency 14973.162 ms 1 485 0.00 MB/sec execute 11 sec latency 15973.517 ms 1 485 0.00 MB/sec execute 12 sec latency 16976.574 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 485 0.00 MB/sec execute 13 sec latency 17977.406 ms 1 485 0.00 MB/sec execute 14 sec latency 18979.400 ms 1 485 0.00 MB/sec execute 15 sec latency 19980.033 ms 1 485 0.00 MB/sec execute 16 sec latency 20980.679 ms 1 485 0.00 MB/sec execute 17 sec latency 21981.213 ms 1 485 0.00 MB/sec execute 18 sec latency 22982.351 ms 1 485 0.00 MB/sec execute 19 sec latency 23982.994 ms 1 485 0.00 MB/sec execute 20 sec latency 24983.310 ms 1 485 0.00 MB/sec execute 21 sec latency 25983.656 ms 1 485 0.00 MB/sec execute 22 sec latency 26984.376 ms 1 485 0.00 MB/sec execute 23 sec latency 27985.133 ms 1 485 0.00 MB/sec execute 24 sec latency 28986.934 ms 1 485 0.00 MB/sec execute 25 sec latency 29987.237 ms 1 485 0.00 MB/sec execute 26 sec latency 30988.450 ms 1 485 0.00 MB/sec execute 27 sec latency 31989.578 ms oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all 1 485 0.00 MB/sec execute 28 sec latency 32989.870 ms pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 1 485 0.00 MB/sec execute 29 sec latency 33990.653 ms 1 485 0.00 MB/sec execute 30 sec latency 34995.073 ms Started lustre-MDT0000 19:09:02 (1787872142) targets are mounted 19:09:02 (1787872142) facet_failover done 1 485 0.00 MB/sec execute 31 sec latency 35995.556 ms 1 485 0.00 MB/sec execute 32 sec latency 36996.407 ms 1 485 0.00 MB/sec execute 33 sec latency 37997.124 ms 1 485 0.00 MB/sec execute 34 sec latency 38997.432 ms 1 485 0.00 MB/sec execute 35 sec latency 39998.251 ms 1 493 0.01 MB/sec execute 36 sec latency 40775.627 ms 1 519 0.03 MB/sec execute 37 sec latency 266.321 ms 1 599 0.10 MB/sec execute 38 sec latency 143.842 ms 1 652 0.11 MB/sec execute 39 sec latency 152.164 ms 1 674 0.11 MB/sec execute 40 sec latency 288.320 ms oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 711 0.10 MB/sec execute 41 sec latency 134.838 ms 1 731 0.10 MB/sec execute 42 sec latency 195.494 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 764 0.10 MB/sec execute 43 sec latency 142.051 ms 1 789 0.10 MB/sec execute 44 sec latency 193.772 ms 1 803 0.10 MB/sec execute 45 sec latency 410.232 ms 1 858 0.10 MB/sec execute 46 sec latency 128.993 ms 1 885 0.10 MB/sec execute 47 sec latency 216.660 ms 1 916 0.10 MB/sec execute 48 sec latency 167.828 ms 1 969 0.10 MB/sec execute 49 sec latency 85.758 ms 1 1021 0.11 MB/sec execute 50 sec latency 182.677 ms 1 1082 0.12 MB/sec execute 51 sec latency 101.464 ms 1 1136 0.12 MB/sec execute 52 sec latency 89.694 ms 1 1175 0.11 MB/sec execute 53 sec latency 135.326 ms 1 1215 0.11 MB/sec execute 54 sec latency 134.678 ms 1 1242 0.11 MB/sec execute 55 sec latency 151.836 ms 1 1274 0.11 MB/sec execute 56 sec latency 159.347 ms 1 1410 0.16 MB/sec execute 57 sec latency 93.951 ms 1 1459 0.16 MB/sec execute 58 sec latency 95.951 ms 1 1493 0.15 MB/sec execute 59 sec latency 344.467 ms 1 1545 0.15 MB/sec execute 60 sec latency 127.682 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Aug 27 19:09:33 EDT 2026 with return code 0 1 1580 0.15 MB/sec execute 61 sec latency 92.483 ms 47218 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47218 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished