== replay-dual test 26: dbench and tar with mds failover ========================================================== 15:03:21 (1777575801) Starting client oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre Started clients oleg402-client.virtnet: 192.168.204.102@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 64829 Started dbench loop with 64831 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1756 1269548 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3588940 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3108 7194396 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Apr 30 15:03:28 EDT 2026 waiting for dbench pid 64910 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 10 0.00 MB/sec warmup 1 sec latency 3.015 ms 1 51 1.05 MB/sec warmup 2 sec latency 169.688 ms 1 110 1.75 MB/sec warmup 3 sec latency 112.409 ms 1 130 1.44 MB/sec warmup 4 sec latency 201.427 ms 1 137 1.20 MB/sec warmup 5 sec latency 937.690 ms 1 143 1.01 MB/sec warmup 6 sec latency 1626.475 ms 1 181 1.05 MB/sec warmup 7 sec latency 134.450 ms 1 199 0.98 MB/sec warmup 8 sec latency 414.780 ms test_26 fail mds1 1 times 1 239 1.02 MB/sec warmup 9 sec latency 136.249 ms 1 269 1.01 MB/sec warmup 10 sec latency 138.317 ms 1 294 0.98 MB/sec warmup 11 sec latency 180.702 ms Failing mds1 on oleg402-server 1 316 0.96 MB/sec warmup 12 sec latency 196.301 ms Stopping /mnt/lustre-mds1 (opts:) on oleg402-server 1 341 0.94 MB/sec warmup 13 sec latency 144.549 ms 1 373 0.95 MB/sec warmup 14 sec latency 195.622 ms 1 405 0.96 MB/sec warmup 15 sec latency 147.520 ms 1 444 0.98 MB/sec warmup 16 sec latency 174.049 ms 15:03:45 (1777575825) shut down facet: mds1 facet_host: oleg402-server facet_failover_host: oleg402-server 1 482 0.99 MB/sec warmup 17 sec latency 127.880 ms 1 501 0.97 MB/sec warmup 18 sec latency 533.252 ms 1 501 0.92 MB/sec warmup 19 sec latency 1533.645 ms 1 501 0.00 MB/sec execute 1 sec latency 3536.956 ms 1 501 0.00 MB/sec execute 2 sec latency 4537.343 ms 1 501 0.00 MB/sec execute 3 sec latency 5538.883 ms 1 501 0.00 MB/sec execute 4 sec latency 6539.745 ms 1 501 0.00 MB/sec execute 5 sec latency 7543.580 ms 1 501 0.00 MB/sec execute 6 sec latency 8543.945 ms Failover mds1 to oleg402-server 1 501 0.00 MB/sec execute 7 sec latency 9548.338 ms mount facets: mds1 1 501 0.00 MB/sec execute 8 sec latency 10548.582 ms 1 501 0.00 MB/sec execute 9 sec latency 11549.010 ms 1 501 0.00 MB/sec execute 10 sec latency 12551.365 ms 1 501 0.00 MB/sec execute 11 sec latency 13551.673 ms 1 501 0.00 MB/sec execute 12 sec latency 14553.147 ms 1 501 0.00 MB/sec execute 13 sec latency 15553.929 ms 1 501 0.00 MB/sec execute 14 sec latency 16554.521 ms 1 501 0.00 MB/sec execute 15 sec latency 17555.312 ms 1 501 0.00 MB/sec execute 16 sec latency 18558.677 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 501 0.00 MB/sec execute 17 sec latency 19559.386 ms 1 501 0.00 MB/sec execute 18 sec latency 20559.617 ms 1 501 0.00 MB/sec execute 19 sec latency 21559.889 ms 1 501 0.00 MB/sec execute 20 sec latency 22560.464 ms 1 501 0.00 MB/sec execute 21 sec latency 23560.903 ms 1 501 0.00 MB/sec execute 22 sec latency 24561.331 ms 1 501 0.00 MB/sec execute 23 sec latency 25561.703 ms 1 501 0.00 MB/sec execute 24 sec latency 26562.059 ms oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all 1 501 0.00 MB/sec execute 25 sec latency 27562.286 ms 1 501 0.00 MB/sec execute 26 sec latency 28562.616 ms pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0000 1 501 0.00 MB/sec execute 27 sec latency 29564.087 ms 15:04:15 (1777575855) targets are mounted 15:04:15 (1777575855) facet_failover done 1 510 0.01 MB/sec execute 28 sec latency 30307.953 ms 1 542 0.06 MB/sec execute 29 sec latency 215.726 ms 1 586 0.09 MB/sec execute 30 sec latency 597.140 ms 1 644 0.12 MB/sec execute 31 sec latency 108.530 ms oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 678 0.12 MB/sec execute 32 sec latency 107.665 ms 1 710 0.11 MB/sec execute 33 sec latency 127.751 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 735 0.11 MB/sec execute 34 sec latency 158.898 ms 1 767 0.11 MB/sec execute 35 sec latency 147.031 ms 1 789 0.11 MB/sec execute 36 sec latency 157.440 ms 1 821 0.11 MB/sec execute 37 sec latency 150.243 ms 1 870 0.11 MB/sec execute 38 sec latency 130.673 ms 1 902 0.11 MB/sec execute 39 sec latency 119.934 ms 1 940 0.11 MB/sec execute 40 sec latency 108.170 ms 1 972 0.11 MB/sec execute 41 sec latency 188.255 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2100 1269204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1872 1269432 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9852 3592168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37296 3566120 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 47148 7158288 1% /mnt/lustre 1 1004 0.11 MB/sec execute 42 sec latency 134.477 ms 1 1063 0.13 MB/sec execute 43 sec latency 84.656 ms 1 1095 0.12 MB/sec execute 44 sec latency 117.915 ms 1 1134 0.12 MB/sec execute 45 sec latency 118.978 ms 1 1172 0.12 MB/sec execute 46 sec latency 104.674 ms 1 1172 0.12 MB/sec execute 47 sec latency 1011.812 ms 1 1173 0.12 MB/sec execute 48 sec latency 1013.649 ms 1 1195 0.11 MB/sec execute 49 sec latency 1408.955 ms 1 1227 0.11 MB/sec execute 50 sec latency 118.718 ms 1 1263 0.11 MB/sec execute 51 sec latency 112.028 ms test_26 fail mds2 2 times 1 1387 0.16 MB/sec execute 52 sec latency 94.084 ms 1 1419 0.16 MB/sec execute 53 sec latency 157.160 ms 1 1451 0.16 MB/sec execute 54 sec latency 101.439 ms Failing mds2 on oleg402-server 1 1493 0.16 MB/sec execute 55 sec latency 107.227 ms Stopping /mnt/lustre-mds2 (opts:) on oleg402-server 1 1523 0.15 MB/sec execute 56 sec latency 110.824 ms 1 1547 0.15 MB/sec execute 57 sec latency 458.023 ms 1 1547 0.15 MB/sec execute 58 sec latency 1458.408 ms 1 1547 0.15 MB/sec execute 59 sec latency 2458.695 ms 15:04:48 (1777575888) shut down facet: mds2 facet_host: oleg402-server facet_failover_host: oleg402-server 1 1547 0.14 MB/sec execute 60 sec latency 3458.965 ms 1 1547 0.14 MB/sec execute 61 sec latency 4459.647 ms 1 1547 0.14 MB/sec execute 62 sec latency 5461.725 ms 1 1547 0.14 MB/sec execute 63 sec latency 6462.005 ms 1 1547 0.13 MB/sec execute 64 sec latency 7462.342 ms 1 1547 0.13 MB/sec execute 65 sec latency 8465.829 ms 1 1547 0.13 MB/sec execute 66 sec latency 9466.560 ms 1 1547 0.13 MB/sec execute 67 sec latency 10466.844 ms 1 1547 0.13 MB/sec execute 68 sec latency 11467.177 ms 1 1547 0.12 MB/sec execute 69 sec latency 12467.480 ms Failover mds2 to oleg402-server 1 1547 0.12 MB/sec execute 70 sec latency 13468.096 ms mount facets: mds2 1 1547 0.12 MB/sec execute 71 sec latency 14468.378 ms 1 1547 0.12 MB/sec execute 72 sec latency 15469.540 ms 1 1547 0.12 MB/sec execute 73 sec latency 16470.262 ms 1 1547 0.12 MB/sec execute 74 sec latency 17470.827 ms 1 1547 0.11 MB/sec execute 75 sec latency 18471.583 ms 1 1547 0.11 MB/sec execute 76 sec latency 19471.833 ms 1 1547 0.11 MB/sec execute 77 sec latency 20472.354 ms 1 1547 0.11 MB/sec execute 78 sec latency 21472.631 ms 1 1547 0.11 MB/sec execute 79 sec latency 22472.846 ms 1 1547 0.11 MB/sec execute 80 sec latency 23474.737 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1547 0.11 MB/sec execute 81 sec latency 24474.937 ms 1 1547 0.11 MB/sec execute 82 sec latency 25475.258 ms 1 1547 0.10 MB/sec execute 83 sec latency 26475.586 ms 1 1547 0.10 MB/sec execute 84 sec latency 27477.216 ms 1 1547 0.10 MB/sec execute 85 sec latency 28477.525 ms 1 1547 0.10 MB/sec execute 86 sec latency 29478.616 ms 1 1547 0.10 MB/sec execute 87 sec latency 30479.881 ms oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all 1 1547 0.10 MB/sec execute 88 sec latency 31480.164 ms pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 1 1547 0.10 MB/sec execute 89 sec latency 32484.749 ms Started lustre-MDT0001 15:05:18 (1777575918) targets are mounted 15:05:18 (1777575918) facet_failover done 1 1547 0.10 MB/sec execute 90 sec latency 33485.025 ms 1 1564 0.09 MB/sec execute 91 sec latency 33943.651 ms 1 1605 0.09 MB/sec execute 92 sec latency 82.486 ms 1 1643 0.09 MB/sec execute 93 sec latency 145.930 ms 1 1665 0.09 MB/sec execute 94 sec latency 133.772 ms oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1696 0.09 MB/sec execute 95 sec latency 142.911 ms 1 1750 0.09 MB/sec execute 96 sec latency 117.702 ms 1 1779 0.09 MB/sec execute 97 sec latency 153.667 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1820 0.09 MB/sec execute 98 sec latency 108.980 ms 1 1855 0.09 MB/sec execute 99 sec latency 119.975 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 254 192.303 30307.929 Close 198 188.986 33943.615 Rename 11 107.328 153.738 Unlink 24 32.475 71.051 Qpathinfo 198 20.677 117.234 Qfileinfo 57 3.064 15.110 Qfsinfo 33 3.233 17.710 Sfileinfo 34 71.542 134.463 Find 73 15.121 75.639 WriteX 197 14.991 107.183 ReadX 296 0.244 15.131 Flush 30 66.084 158.879 Throughput 0.089706 MB/sec 1 clients 1 procs max_latency=33943.651 ms stopping dbench on /mnt/lustre at Thu Apr 30 15:05:32 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2376 1268928 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23456 3582408 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 40832 3566104 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 64288 7148512 1% /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Apr 30 15:05:42 EDT 2026 waiting for dbench pid 67089 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 17.955 ms test_26 fail mds1 3 times 1 102 2.36 MB/sec warmup 2 sec latency 105.257 ms Failing mds1 on oleg402-server 1 137 2.00 MB/sec warmup 3 sec latency 197.323 ms Stopping /mnt/lustre-mds1 (opts:) on oleg402-server 1 165 1.70 MB/sec warmup 4 sec latency 222.750 ms 1 188 1.51 MB/sec warmup 5 sec latency 346.252 ms 1 188 1.26 MB/sec warmup 6 sec latency 1095.032 ms 1 188 1.08 MB/sec warmup 7 sec latency 2095.273 ms 15:05:49 (1777575949) shut down facet: mds1 facet_host: oleg402-server facet_failover_host: oleg402-server 1 188 0.95 MB/sec warmup 8 sec latency 3100.386 ms 1 188 0.84 MB/sec warmup 9 sec latency 4100.633 ms 1 188 0.76 MB/sec warmup 10 sec latency 5100.891 ms 1 188 0.69 MB/sec warmup 11 sec latency 6101.172 ms 1 188 0.63 MB/sec warmup 12 sec latency 7101.628 ms 1 188 0.58 MB/sec warmup 13 sec latency 8102.337 ms 1 188 0.54 MB/sec warmup 14 sec latency 9104.339 ms 1 188 0.50 MB/sec warmup 15 sec latency 10105.215 ms 1 188 0.47 MB/sec warmup 16 sec latency 11105.499 ms 1 188 0.45 MB/sec warmup 17 sec latency 12105.787 ms Failover mds1 to oleg402-server mount facets: mds1 1 188 0.42 MB/sec warmup 18 sec latency 13108.852 ms 1 188 0.40 MB/sec warmup 19 sec latency 14110.063 ms 1 188 0.00 MB/sec execute 1 sec latency 16115.840 ms 1 188 0.00 MB/sec execute 2 sec latency 17116.039 ms 1 188 0.00 MB/sec execute 3 sec latency 18117.037 ms 1 188 0.00 MB/sec execute 4 sec latency 19119.147 ms 1 188 0.00 MB/sec execute 5 sec latency 20120.391 ms 1 188 0.00 MB/sec execute 6 sec latency 21120.766 ms 1 188 0.00 MB/sec execute 7 sec latency 22122.744 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 188 0.00 MB/sec execute 8 sec latency 23124.345 ms 1 188 0.00 MB/sec execute 9 sec latency 24124.613 ms 1 188 0.00 MB/sec execute 10 sec latency 25127.178 ms 1 188 0.00 MB/sec execute 11 sec latency 26127.515 ms 1 188 0.00 MB/sec execute 12 sec latency 27127.776 ms 1 188 0.00 MB/sec execute 13 sec latency 28128.333 ms 1 188 0.00 MB/sec execute 14 sec latency 29128.955 ms 1 188 0.00 MB/sec execute 15 sec latency 30129.222 ms 1 188 0.00 MB/sec execute 16 sec latency 31131.928 ms 1 188 0.00 MB/sec execute 17 sec latency 32132.208 ms 1 188 0.00 MB/sec execute 18 sec latency 33132.518 ms 1 188 0.00 MB/sec execute 19 sec latency 34132.774 ms 1 188 0.00 MB/sec execute 20 sec latency 35133.346 ms oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all 1 188 0.00 MB/sec execute 21 sec latency 36135.683 ms pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 1 188 0.00 MB/sec execute 22 sec latency 37135.958 ms Started lustre-MDT0000 15:06:25 (1777575985) targets are mounted 15:06:25 (1777575985) facet_failover done 1 188 0.00 MB/sec execute 23 sec latency 38136.721 ms 1 188 0.00 MB/sec execute 24 sec latency 39137.278 ms 1 199 0.01 MB/sec execute 25 sec latency 39570.711 ms 1 230 0.05 MB/sec execute 26 sec latency 176.596 ms 1 263 0.09 MB/sec execute 27 sec latency 124.328 ms 1 290 0.11 MB/sec execute 28 sec latency 171.628 ms oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 315 0.13 MB/sec execute 29 sec latency 206.022 ms 1 341 0.16 MB/sec execute 30 sec latency 161.160 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 371 0.18 MB/sec execute 31 sec latency 221.878 ms 1 405 0.21 MB/sec execute 32 sec latency 129.920 ms 1 442 0.24 MB/sec execute 33 sec latency 120.952 ms 1 476 0.27 MB/sec execute 34 sec latency 159.251 ms 1 504 0.28 MB/sec execute 35 sec latency 132.149 ms 1 558 0.32 MB/sec execute 36 sec latency 144.381 ms 1 639 0.36 MB/sec execute 37 sec latency 98.246 ms 1 674 0.36 MB/sec execute 38 sec latency 117.647 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2680 1268624 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2400 1268904 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 49112 3551536 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8764 3590140 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 57876 7141676 1% /mnt/lustre 1 699 0.35 MB/sec execute 39 sec latency 139.730 ms 1 723 0.34 MB/sec execute 40 sec latency 169.376 ms 1 755 0.33 MB/sec execute 41 sec latency 147.308 ms 1 779 0.33 MB/sec execute 42 sec latency 145.811 ms 1 801 0.32 MB/sec execute 43 sec latency 156.552 ms 1 827 0.32 MB/sec execute 44 sec latency 102.307 ms 1 874 0.31 MB/sec execute 45 sec latency 109.568 ms 1 884 0.31 MB/sec execute 46 sec latency 685.913 ms 1 890 0.30 MB/sec execute 47 sec latency 1261.947 ms 1 918 0.29 MB/sec execute 48 sec latency 110.025 ms 1 959 0.29 MB/sec execute 49 sec latency 117.781 ms test_26 fail mds2 4 times 1 983 0.29 MB/sec execute 50 sec latency 154.731 ms 1 1033 0.30 MB/sec execute 51 sec latency 124.949 ms Failing mds2 on oleg402-server 1 1074 0.29 MB/sec execute 52 sec latency 110.060 ms Stopping /mnt/lustre-mds2 (opts:) on oleg402-server 1 1124 0.29 MB/sec execute 53 sec latency 94.400 ms 1 1158 0.29 MB/sec execute 54 sec latency 110.454 ms 1 1202 0.28 MB/sec execute 55 sec latency 70.350 ms 1 1251 0.28 MB/sec execute 56 sec latency 97.598 ms 1 1278 0.27 MB/sec execute 57 sec latency 334.956 ms 1 1278 0.27 MB/sec execute 58 sec latency 1335.408 ms 1 1278 0.26 MB/sec execute 59 sec latency 2335.720 ms 1 1278 0.26 MB/sec execute 60 sec latency 3335.949 ms 1 1278 0.25 MB/sec execute 61 sec latency 4336.634 ms 1 1278 0.25 MB/sec execute 62 sec latency 5336.831 ms 15:07:04 (1777576024) shut down facet: mds2 facet_host: oleg402-server facet_failover_host: oleg402-server 1 1278 0.25 MB/sec execute 63 sec latency 6337.110 ms 1 1278 0.24 MB/sec execute 64 sec latency 7338.133 ms 1 1278 0.24 MB/sec execute 65 sec latency 8339.400 ms 1 1278 0.24 MB/sec execute 66 sec latency 9343.635 ms 1 1278 0.23 MB/sec execute 67 sec latency 10345.196 ms 1 1278 0.23 MB/sec execute 68 sec latency 11345.665 ms 1 1278 0.23 MB/sec execute 69 sec latency 12346.144 ms 1 1278 0.22 MB/sec execute 70 sec latency 13346.435 ms 1 1278 0.22 MB/sec execute 71 sec latency 14346.694 ms 1 1278 0.22 MB/sec execute 72 sec latency 15346.932 ms Failover mds2 to oleg402-server mount facets: mds2 1 1278 0.21 MB/sec execute 73 sec latency 16347.256 ms 1 1278 0.21 MB/sec execute 74 sec latency 17347.560 ms 1 1278 0.21 MB/sec execute 75 sec latency 18352.643 ms 1 1278 0.20 MB/sec execute 76 sec latency 19353.575 ms 1 1278 0.20 MB/sec execute 77 sec latency 20355.457 ms 1 1278 0.20 MB/sec execute 78 sec latency 21367.741 ms 1 1278 0.20 MB/sec execute 79 sec latency 22368.631 ms 1 1278 0.19 MB/sec execute 80 sec latency 23372.023 ms 1 1278 0.19 MB/sec execute 81 sec latency 24372.263 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1278 0.19 MB/sec execute 82 sec latency 25372.557 ms 1 1278 0.19 MB/sec execute 83 sec latency 26372.920 ms 1 1278 0.18 MB/sec execute 84 sec latency 27374.773 ms 1 1278 0.18 MB/sec execute 85 sec latency 28375.029 ms 1 1278 0.18 MB/sec execute 86 sec latency 29375.310 ms 1 1278 0.18 MB/sec execute 87 sec latency 30375.625 ms oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all 1 1278 0.18 MB/sec execute 88 sec latency 31375.881 ms pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 1 1278 0.17 MB/sec execute 89 sec latency 32377.160 ms Started lustre-MDT0001 15:07:32 (1777576052) targets are mounted 15:07:32 (1777576052) facet_failover done 1 1278 0.17 MB/sec execute 90 sec latency 33379.518 ms 1 1406 0.20 MB/sec execute 91 sec latency 33494.737 ms 1 1449 0.20 MB/sec execute 92 sec latency 103.867 ms 1 1486 0.20 MB/sec execute 93 sec latency 351.119 ms oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1523 0.20 MB/sec execute 94 sec latency 134.731 ms 1 1574 0.19 MB/sec execute 95 sec latency 104.222 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1610 0.19 MB/sec execute 96 sec latency 121.245 ms 1 1651 0.19 MB/sec execute 97 sec latency 104.042 ms 1 1686 0.19 MB/sec execute 98 sec latency 140.246 ms 1 1740 0.19 MB/sec execute 99 sec latency 145.518 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 265 72.739 276.440 Close 231 186.316 39570.577 Rename 11 105.007 169.338 Unlink 24 29.439 66.046 Qpathinfo 173 18.761 104.182 Qfileinfo 89 3.602 17.274 Qfsinfo 33 4.524 33.098 Sfileinfo 34 71.582 156.529 Find 64 535.802 33494.716 WriteX 375 17.838 92.313 ReadX 280 2.163 110.036 Flush 27 65.890 125.242 Throughput 0.188543 MB/sec 1 clients 1 procs max_latency=39570.711 ms stopping dbench on /mnt/lustre at Thu Apr 30 15:07:45 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench tar: Unexpected EOF in archive tar: Unexpected EOF in archive looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Apr 30 15:07:53 EDT 2026 waiting for dbench pid 69123 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 12 0.00 MB/sec warmup 1 sec latency 26.138 ms 1 71 1.57 MB/sec warmup 2 sec latency 98.351 ms 1 121 1.83 MB/sec warmup 3 sec latency 173.616 ms 1 155 1.63 MB/sec warmup 4 sec latency 147.033 ms 1 184 1.48 MB/sec warmup 5 sec latency 173.432 ms 1 207 1.35 MB/sec warmup 6 sec latency 479.377 ms 1 256 1.38 MB/sec warmup 7 sec latency 84.822 ms tar: Error is not recoverable: exiting now 1 298 1.37 MB/sec warmup 8 sec latency 128.354 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Apr 30 15:08:01 EDT 2026 with return code 0 69123 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 69123 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished