== replay-dual test 26: dbench and tar with mds failover ========================================================== 04:06:24 (1784189184) Starting client oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre Started clients oleg257-client.virtnet: 192.168.202.157@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44983 Started dbench loop with 44984 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3734528 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7496704 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jul 16 04:06:30 EDT 2026 waiting for dbench pid 45053 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 1 of 1 processes prepared for launch 0 sec releasing clients 1 34 1.05 MB/sec warmup 1 sec latency 116.717 ms 1 91 2.09 MB/sec warmup 2 sec latency 139.980 ms 1 129 1.91 MB/sec warmup 3 sec latency 154.346 ms 1 165 1.70 MB/sec warmup 4 sec latency 147.850 ms test_26 fail mds1 1 times 1 190 1.51 MB/sec warmup 5 sec latency 222.851 ms 1 224 1.43 MB/sec warmup 6 sec latency 142.753 ms Failing mds1 on oleg257-server 1 263 1.42 MB/sec warmup 7 sec latency 170.383 ms Stopping /mnt/lustre-mds1 (opts:) on oleg257-server 1 300 1.37 MB/sec warmup 8 sec latency 132.745 ms 1 331 1.33 MB/sec warmup 9 sec latency 350.097 ms 1 331 1.20 MB/sec warmup 10 sec latency 1350.331 ms 1 331 1.09 MB/sec warmup 11 sec latency 2350.591 ms 04:06:41 (1784189201) shut down facet: mds1 facet_host: oleg257-server facet_failover_host: oleg257-server 1 331 1.00 MB/sec warmup 12 sec latency 3353.130 ms 1 331 0.92 MB/sec warmup 13 sec latency 4353.392 ms 1 331 0.86 MB/sec warmup 14 sec latency 5353.675 ms 1 331 0.80 MB/sec warmup 15 sec latency 6353.959 ms 1 331 0.75 MB/sec warmup 16 sec latency 7354.298 ms 1 331 0.71 MB/sec warmup 17 sec latency 8354.553 ms 1 331 0.67 MB/sec warmup 18 sec latency 9354.962 ms 1 331 0.63 MB/sec warmup 19 sec latency 10355.467 ms 1 331 0.00 MB/sec execute 1 sec latency 12357.188 ms Failover mds1 to oleg257-server mount facets: mds1 1 331 0.00 MB/sec execute 2 sec latency 13358.246 ms 1 331 0.00 MB/sec execute 3 sec latency 14358.522 ms 1 331 0.00 MB/sec execute 4 sec latency 15361.165 ms 1 331 0.00 MB/sec execute 5 sec latency 16361.784 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 331 0.00 MB/sec execute 6 sec latency 17362.113 ms 1 331 0.00 MB/sec execute 7 sec latency 18362.918 ms 1 331 0.00 MB/sec execute 8 sec latency 19363.261 ms 1 331 0.00 MB/sec execute 9 sec latency 20363.525 ms 1 331 0.00 MB/sec execute 10 sec latency 21363.848 ms 1 331 0.00 MB/sec execute 11 sec latency 22364.202 ms oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all 1 331 0.00 MB/sec execute 12 sec latency 23364.496 ms 1 334 0.00 MB/sec execute 13 sec latency 24331.481 ms pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 1 360 0.06 MB/sec execute 14 sec latency 174.262 ms Started lustre-MDT0000 04:07:04 (1784189224) targets are mounted 04:07:04 (1784189224) facet_failover done 1 397 0.14 MB/sec execute 15 sec latency 134.989 ms 1 428 0.19 MB/sec execute 16 sec latency 149.809 ms 1 462 0.24 MB/sec execute 17 sec latency 163.513 ms 1 500 0.30 MB/sec execute 18 sec latency 132.477 ms 1 524 0.32 MB/sec execute 19 sec latency 174.538 ms 1 606 0.45 MB/sec execute 20 sec latency 143.204 ms 1 649 0.43 MB/sec execute 21 sec latency 161.076 ms oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 673 0.42 MB/sec execute 22 sec latency 203.703 ms 1 708 0.40 MB/sec execute 23 sec latency 130.983 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 730 0.39 MB/sec execute 24 sec latency 270.591 ms 1 762 0.37 MB/sec execute 25 sec latency 112.366 ms 1 789 0.36 MB/sec execute 26 sec latency 168.061 ms 1 820 0.35 MB/sec execute 27 sec latency 217.567 ms 1 882 0.34 MB/sec execute 28 sec latency 193.603 ms 1 918 0.34 MB/sec execute 29 sec latency 182.762 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 44032 3686400 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7433216 1% /mnt/lustre 1 955 0.33 MB/sec execute 30 sec latency 139.611 ms 1 983 0.32 MB/sec execute 31 sec latency 255.011 ms 1 1050 0.34 MB/sec execute 32 sec latency 132.446 ms 1 1095 0.33 MB/sec execute 33 sec latency 94.896 ms 1 1147 0.32 MB/sec execute 34 sec latency 89.534 ms 1 1200 0.32 MB/sec execute 35 sec latency 91.113 ms test_26 fail mds1 2 times 1 1242 0.31 MB/sec execute 36 sec latency 74.708 ms 1 1274 0.30 MB/sec execute 37 sec latency 125.458 ms Failing mds1 on oleg257-server Stopping /mnt/lustre-mds1 (opts:) on oleg257-server 1 1421 0.37 MB/sec execute 38 sec latency 113.536 ms 1 1473 0.36 MB/sec execute 39 sec latency 201.480 ms 1 1473 0.35 MB/sec execute 40 sec latency 1201.683 ms 1 1473 0.34 MB/sec execute 41 sec latency 2201.873 ms 04:07:31 (1784189251) shut down facet: mds1 facet_host: oleg257-server facet_failover_host: oleg257-server 1 1473 0.33 MB/sec execute 42 sec latency 3202.769 ms 1 1473 0.33 MB/sec execute 43 sec latency 4202.959 ms 1 1473 0.32 MB/sec execute 44 sec latency 5204.755 ms 1 1473 0.31 MB/sec execute 45 sec latency 6204.994 ms 1 1473 0.30 MB/sec execute 46 sec latency 7205.267 ms 1 1473 0.30 MB/sec execute 47 sec latency 8206.605 ms 1 1473 0.29 MB/sec execute 48 sec latency 9206.825 ms 1 1473 0.29 MB/sec execute 49 sec latency 10207.114 ms 1 1473 0.28 MB/sec execute 50 sec latency 11207.984 ms 1 1473 0.27 MB/sec execute 51 sec latency 12208.574 ms Failover mds1 to oleg257-server mount facets: mds1 1 1473 0.27 MB/sec execute 52 sec latency 13213.160 ms 1 1473 0.26 MB/sec execute 53 sec latency 14214.890 ms 1 1473 0.26 MB/sec execute 54 sec latency 15215.266 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1473 0.25 MB/sec execute 55 sec latency 16216.894 ms 1 1473 0.25 MB/sec execute 56 sec latency 17217.090 ms 1 1473 0.25 MB/sec execute 57 sec latency 18217.884 ms 1 1473 0.24 MB/sec execute 58 sec latency 19218.458 ms 1 1473 0.24 MB/sec execute 59 sec latency 20219.122 ms 1 1473 0.23 MB/sec execute 60 sec latency 21219.305 ms 1 1473 0.23 MB/sec execute 61 sec latency 22220.128 ms 1 1473 0.23 MB/sec execute 62 sec latency 23220.426 ms oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all 1 1474 0.22 MB/sec execute 63 sec latency 24102.142 ms pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 1 1502 0.22 MB/sec execute 64 sec latency 207.940 ms 1 1551 0.22 MB/sec execute 65 sec latency 137.932 ms Started lustre-MDT0000 04:07:55 (1784189275) targets are mounted 04:07:55 (1784189275) facet_failover done 1 1603 0.21 MB/sec execute 66 sec latency 163.752 ms 1 1634 0.21 MB/sec execute 67 sec latency 277.626 ms 1 1660 0.21 MB/sec execute 68 sec latency 179.913 ms 1 1691 0.21 MB/sec execute 69 sec latency 192.239 ms 1 1729 0.20 MB/sec execute 70 sec latency 153.100 ms 1 1773 0.20 MB/sec execute 71 sec latency 93.076 ms 1 1820 0.20 MB/sec execute 72 sec latency 165.551 ms 1 1851 0.20 MB/sec execute 73 sec latency 205.332 ms 1 1884 0.19 MB/sec execute 74 sec latency 186.618 ms oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1923 0.19 MB/sec execute 75 sec latency 282.055 ms 1 1965 0.19 MB/sec execute 76 sec latency 135.893 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1986 0.19 MB/sec execute 77 sec latency 170.141 ms 1 2016 0.19 MB/sec execute 78 sec latency 150.227 ms 1 2076 0.18 MB/sec execute 79 sec latency 91.807 ms 1 2145 0.18 MB/sec execute 80 sec latency 112.519 ms 1 2170 0.18 MB/sec execute 81 sec latency 102.794 ms 1 2243 0.18 MB/sec execute 82 sec latency 108.848 ms 1 2327 0.19 MB/sec execute 83 sec latency 94.308 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3731456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 50176 3712000 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 72704 7443456 1% /mnt/lustre 1 2347 0.19 MB/sec execute 84 sec latency 467.856 ms 1 2373 0.19 MB/sec execute 85 sec latency 175.247 ms 1 2393 0.19 MB/sec execute 86 sec latency 303.888 ms 1 2418 0.18 MB/sec execute 87 sec latency 323.165 ms 1 2450 0.18 MB/sec execute 88 sec latency 124.763 ms 1 2482 0.19 MB/sec execute 89 sec latency 289.124 ms 1 2566 0.19 MB/sec execute 90 sec latency 259.101 ms test_26 fail mds1 3 times 1 2567 0.19 MB/sec execute 91 sec latency 1178.716 ms 1 2604 0.18 MB/sec execute 92 sec latency 116.975 ms Failing mds1 on oleg257-server 1 2759 0.21 MB/sec execute 93 sec latency 103.249 ms Stopping /mnt/lustre-mds1 (opts:) on oleg257-server 1 2781 0.21 MB/sec execute 94 sec latency 306.709 ms 1 2880 0.21 MB/sec execute 95 sec latency 520.979 ms 1 2880 0.21 MB/sec execute 96 sec latency 1521.227 ms 1 2880 0.21 MB/sec execute 97 sec latency 2523.281 ms 04:08:27 (1784189307) shut down facet: mds1 facet_host: oleg257-server facet_failover_host: oleg257-server 1 2880 0.21 MB/sec execute 98 sec latency 3526.526 ms 1 2880 0.20 MB/sec execute 99 sec latency 4526.916 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec Failover mds1 to oleg257-server 1 cleanup 108 sec mount facets: mds1 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec 1 cleanup 116 sec 1 cleanup 117 sec 1 cleanup 118 sec 1 cleanup 119 sec 1 cleanup 120 sec 1 cleanup 121 sec oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all 1 cleanup 122 sec 1 cleanup 123 sec pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 1 cleanup 124 sec Started lustre-MDT0000 04:08:54 (1784189334) targets are mounted 04:08:54 (1784189334) facet_failover done 1 cleanup 125 sec 0 cleanup 126 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 402 190.870 26941.485 Close 344 83.171 24102.120 Rename 18 83.554 132.740 Unlink 42 32.885 73.888 Qpathinfo 364 18.237 98.141 Qfileinfo 113 2.563 14.921 Qfsinfo 54 4.091 22.648 Sfileinfo 50 66.011 175.229 Find 128 5.979 43.616 WriteX 382 14.231 84.235 ReadX 614 1.685 94.293 LockX 2 18.429 21.396 UnlockX 2 17.942 23.813 Flush 35 162.364 467.766 Throughput 0.203695 MB/sec 1 clients 1 procs max_latency=24331.481 ms stopping dbench on /mnt/lustre at Thu Jul 16 04:08:56 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jul 16 04:09:04 EDT 2026 waiting for dbench pid 47778 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 31 1.04 MB/sec warmup 1 sec latency 119.629 ms 1 91 2.09 MB/sec warmup 2 sec latency 180.401 ms 1 139 2.00 MB/sec warmup 3 sec latency 170.678 ms 1 173 1.76 MB/sec warmup 4 sec latency 154.615 ms 1 206 1.62 MB/sec warmup 5 sec latency 135.506 ms 1 263 1.65 MB/sec warmup 6 sec latency 122.224 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3697664 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3748864 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7446528 1% /mnt/lustre 1 297 1.56 MB/sec warmup 7 sec latency 135.567 ms 1 323 1.46 MB/sec warmup 8 sec latency 239.433 ms 1 355 1.41 MB/sec warmup 9 sec latency 169.797 ms 1 406 1.43 MB/sec warmup 10 sec latency 127.214 ms 1 453 1.44 MB/sec warmup 11 sec latency 101.318 ms 1 503 1.45 MB/sec warmup 12 sec latency 113.904 ms test_26 fail mds1 4 times 1 606 1.61 MB/sec warmup 13 sec latency 106.117 ms 1 664 1.51 MB/sec warmup 14 sec latency 100.420 ms Failing mds1 on oleg257-server 1 697 1.41 MB/sec warmup 15 sec latency 256.481 ms Stopping /mnt/lustre-mds1 (opts:) on oleg257-server 1 728 1.33 MB/sec warmup 16 sec latency 168.340 ms 1 736 1.25 MB/sec warmup 17 sec latency 727.155 ms 1 736 1.18 MB/sec warmup 18 sec latency 1731.976 ms 04:09:23 (1784189363) shut down facet: mds1 facet_host: oleg257-server facet_failover_host: oleg257-server 1 736 1.12 MB/sec warmup 19 sec latency 2732.159 ms 1 736 0.00 MB/sec execute 1 sec latency 4732.456 ms 1 736 0.00 MB/sec execute 2 sec latency 5735.455 ms 1 736 0.00 MB/sec execute 3 sec latency 6735.892 ms 1 736 0.00 MB/sec execute 4 sec latency 7737.720 ms 1 736 0.00 MB/sec execute 5 sec latency 8738.655 ms 1 736 0.00 MB/sec execute 6 sec latency 9738.826 ms 1 736 0.00 MB/sec execute 7 sec latency 10739.235 ms 1 736 0.00 MB/sec execute 8 sec latency 11739.627 ms 1 736 0.00 MB/sec execute 9 sec latency 12740.633 ms Failover mds1 to oleg257-server mount facets: mds1 1 736 0.00 MB/sec execute 10 sec latency 13740.897 ms 1 736 0.00 MB/sec execute 11 sec latency 14741.173 ms 1 736 0.00 MB/sec execute 12 sec latency 15741.783 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 736 0.00 MB/sec execute 13 sec latency 16744.089 ms 1 736 0.00 MB/sec execute 14 sec latency 17744.321 ms 1 736 0.00 MB/sec execute 15 sec latency 18744.582 ms 1 736 0.00 MB/sec execute 16 sec latency 19744.782 ms 1 736 0.00 MB/sec execute 17 sec latency 20745.560 ms 1 736 0.00 MB/sec execute 18 sec latency 21747.040 ms 1 736 0.00 MB/sec execute 19 sec latency 22747.405 ms 1 736 0.00 MB/sec execute 20 sec latency 23747.782 ms 1 736 0.00 MB/sec execute 21 sec latency 24748.106 ms 1 736 0.00 MB/sec execute 22 sec latency 25748.382 ms 1 736 0.00 MB/sec execute 23 sec latency 26748.641 ms 1 736 0.00 MB/sec execute 24 sec latency 27750.723 ms 1 736 0.00 MB/sec execute 25 sec latency 28750.960 ms 1 736 0.00 MB/sec execute 26 sec latency 29751.094 ms 1 736 0.00 MB/sec execute 27 sec latency 30751.294 ms 1 736 0.00 MB/sec execute 28 sec latency 31751.664 ms 1 736 0.00 MB/sec execute 29 sec latency 32753.396 ms oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all 1 736 0.00 MB/sec execute 30 sec latency 33753.664 ms pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 1 736 0.00 MB/sec execute 31 sec latency 34754.618 ms Started lustre-MDT0000 04:09:55 (1784189395) targets are mounted 04:09:55 (1784189395) facet_failover done 1 736 0.00 MB/sec execute 32 sec latency 35754.873 ms 1 736 0.00 MB/sec execute 33 sec latency 36756.625 ms 1 736 0.00 MB/sec execute 34 sec latency 37757.033 ms 1 736 0.00 MB/sec execute 35 sec latency 38757.890 ms 1 736 0.00 MB/sec execute 36 sec latency 39762.803 ms 1 736 0.00 MB/sec execute 37 sec latency 40763.148 ms 1 736 0.00 MB/sec execute 38 sec latency 41763.303 ms 1 736 0.00 MB/sec execute 39 sec latency 42763.495 ms 1 740 0.00 MB/sec execute 40 sec latency 43526.585 ms 1 779 0.00 MB/sec execute 41 sec latency 131.591 ms 1 799 0.00 MB/sec execute 42 sec latency 323.672 ms 1 845 0.01 MB/sec execute 43 sec latency 162.147 ms 1 876 0.01 MB/sec execute 44 sec latency 287.497 ms oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 902 0.01 MB/sec execute 45 sec latency 138.992 ms 1 917 0.01 MB/sec execute 46 sec latency 332.949 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 964 0.01 MB/sec execute 47 sec latency 122.593 ms 1 980 0.01 MB/sec execute 48 sec latency 214.941 ms 1 1009 0.02 MB/sec execute 49 sec latency 301.937 ms 1 1064 0.03 MB/sec execute 50 sec latency 138.664 ms 1 1134 0.03 MB/sec execute 51 sec latency 86.853 ms 1 1193 0.03 MB/sec execute 52 sec latency 66.869 ms 1 1231 0.03 MB/sec execute 53 sec latency 84.059 ms 1 1274 0.03 MB/sec execute 54 sec latency 151.894 ms 1 1428 0.09 MB/sec execute 55 sec latency 136.993 ms 1 1481 0.08 MB/sec execute 56 sec latency 268.094 ms 1 1525 0.08 MB/sec execute 57 sec latency 299.061 ms 1 1576 0.08 MB/sec execute 58 sec latency 91.817 ms 1 1619 0.08 MB/sec execute 59 sec latency 144.518 ms 1 1643 0.08 MB/sec execute 60 sec latency 443.298 ms 1 1684 0.08 MB/sec execute 61 sec latency 122.005 ms 1 1755 0.08 MB/sec execute 62 sec latency 128.201 ms 1 1814 0.08 MB/sec execute 63 sec latency 90.587 ms 1 1862 0.08 MB/sec execute 64 sec latency 108.296 ms 1 1895 0.08 MB/sec execute 65 sec latency 326.882 ms 1 1974 0.08 MB/sec execute 66 sec latency 74.471 ms 1 2039 0.08 MB/sec execute 67 sec latency 110.825 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Jul 16 04:10:31 EDT 2026 with return code 0 47778 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47778 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished