== replay-dual test 26: dbench and tar with mds failover ========================================================== 09:52:40 (1783950760) Starting client oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre Started clients oleg121-client.virtnet: 192.168.201.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65681 Started dbench loop with 65683 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1940 1269364 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1756 1269548 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3588892 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3108 7194368 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Jul 13 09:52:47 EDT 2026 waiting for dbench pid 65761 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 10 0.00 MB/sec warmup 1 sec latency 46.439 ms 1 56 1.11 MB/sec warmup 2 sec latency 123.378 ms 1 111 1.74 MB/sec warmup 3 sec latency 100.433 ms 1 138 1.50 MB/sec warmup 4 sec latency 176.689 ms 1 166 1.36 MB/sec warmup 5 sec latency 156.186 ms 1 184 1.23 MB/sec warmup 6 sec latency 428.058 ms 1 215 1.19 MB/sec warmup 7 sec latency 611.134 ms 1 262 1.24 MB/sec warmup 8 sec latency 121.822 ms test_26 fail mds1 1 times 1 297 1.21 MB/sec warmup 9 sec latency 167.109 ms 1 326 1.17 MB/sec warmup 10 sec latency 184.543 ms Failing mds1 on oleg121-server 1 349 1.13 MB/sec warmup 11 sec latency 218.683 ms 1 388 1.15 MB/sec warmup 12 sec latency 259.882 ms Stopping /mnt/lustre-mds1 (opts:) on oleg121-server 1 414 1.12 MB/sec warmup 13 sec latency 210.640 ms 1 454 1.13 MB/sec warmup 14 sec latency 170.620 ms 1 500 1.16 MB/sec warmup 15 sec latency 114.244 ms 09:53:02 (1783950782) shut down facet: mds1 facet_host: oleg121-server facet_failover_host: oleg121-server 1 501 1.09 MB/sec warmup 16 sec latency 984.569 ms 1 501 1.02 MB/sec warmup 17 sec latency 1986.394 ms 1 501 0.97 MB/sec warmup 18 sec latency 2986.780 ms 1 501 0.92 MB/sec warmup 19 sec latency 3987.515 ms 1 501 0.00 MB/sec execute 1 sec latency 5987.900 ms 1 501 0.00 MB/sec execute 2 sec latency 6988.321 ms 1 501 0.00 MB/sec execute 3 sec latency 7988.561 ms 1 501 0.00 MB/sec execute 4 sec latency 8988.843 ms 1 501 0.00 MB/sec execute 5 sec latency 9991.777 ms Failover mds1 to oleg121-server mount facets: mds1 1 501 0.00 MB/sec execute 6 sec latency 10992.793 ms 1 501 0.00 MB/sec execute 7 sec latency 12002.449 ms 1 501 0.00 MB/sec execute 8 sec latency 13015.316 ms 1 501 0.00 MB/sec execute 9 sec latency 14034.173 ms 1 501 0.00 MB/sec execute 10 sec latency 15042.226 ms 1 501 0.00 MB/sec execute 11 sec latency 16043.818 ms 1 501 0.00 MB/sec execute 12 sec latency 17044.079 ms 1 501 0.00 MB/sec execute 13 sec latency 18044.342 ms 1 501 0.00 MB/sec execute 14 sec latency 19044.969 ms 1 501 0.00 MB/sec execute 15 sec latency 20045.194 ms 1 501 0.00 MB/sec execute 16 sec latency 21045.515 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 501 0.00 MB/sec execute 17 sec latency 22046.931 ms 1 501 0.00 MB/sec execute 18 sec latency 23048.307 ms 1 501 0.00 MB/sec execute 19 sec latency 24048.535 ms 1 501 0.00 MB/sec execute 20 sec latency 25048.766 ms 1 501 0.00 MB/sec execute 21 sec latency 26049.216 ms 1 501 0.00 MB/sec execute 22 sec latency 27049.379 ms 1 501 0.00 MB/sec execute 23 sec latency 28050.178 ms 1 501 0.00 MB/sec execute 24 sec latency 29059.317 ms 1 501 0.00 MB/sec execute 25 sec latency 30060.240 ms 1 501 0.00 MB/sec execute 26 sec latency 31060.460 ms 1 501 0.00 MB/sec execute 27 sec latency 32061.184 ms oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all 1 501 0.00 MB/sec execute 28 sec latency 33061.411 ms pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 1 501 0.00 MB/sec execute 29 sec latency 34062.958 ms Started lustre-MDT0000 09:53:36 (1783950816) targets are mounted 09:53:36 (1783950816) facet_failover done 1 501 0.00 MB/sec execute 30 sec latency 35063.244 ms 1 511 0.01 MB/sec execute 31 sec latency 35656.378 ms 1 552 0.05 MB/sec execute 32 sec latency 161.532 ms 1 569 0.05 MB/sec execute 33 sec latency 940.537 ms 1 636 0.11 MB/sec execute 34 sec latency 86.465 ms 1 674 0.11 MB/sec execute 35 sec latency 149.186 ms 1 708 0.10 MB/sec execute 36 sec latency 127.774 ms oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 731 0.10 MB/sec execute 37 sec latency 154.315 ms 1 759 0.10 MB/sec execute 38 sec latency 163.846 ms 1 778 0.10 MB/sec execute 39 sec latency 160.298 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 801 0.10 MB/sec execute 40 sec latency 207.007 ms 1 845 0.10 MB/sec execute 41 sec latency 140.457 ms 1 874 0.10 MB/sec execute 42 sec latency 109.827 ms 1 906 0.10 MB/sec execute 43 sec latency 135.827 ms 1 933 0.10 MB/sec execute 44 sec latency 158.393 ms 1 971 0.10 MB/sec execute 45 sec latency 109.095 ms 1 1000 0.10 MB/sec execute 46 sec latency 123.943 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2060 1269244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1872 1269432 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 42152 3556832 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8996 3597456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 51148 7154288 1% /mnt/lustre 1 1061 0.12 MB/sec execute 47 sec latency 92.667 ms 1 1124 0.11 MB/sec execute 48 sec latency 78.027 ms 1 1157 0.11 MB/sec execute 49 sec latency 156.236 ms 1 1173 0.11 MB/sec execute 50 sec latency 733.765 ms 1 1208 0.11 MB/sec execute 51 sec latency 121.710 ms 1 1223 0.11 MB/sec execute 52 sec latency 602.035 ms 1 1251 0.11 MB/sec execute 53 sec latency 1110.742 ms 1 1394 0.16 MB/sec execute 54 sec latency 99.231 ms 1 1428 0.16 MB/sec execute 55 sec latency 105.059 ms test_26 fail mds2 2 times 1 1463 0.15 MB/sec execute 56 sec latency 140.788 ms 1 1500 0.15 MB/sec execute 57 sec latency 112.592 ms Failing mds2 on oleg121-server 1 1538 0.15 MB/sec execute 58 sec latency 139.255 ms Stopping /mnt/lustre-mds2 (opts:) on oleg121-server 1 1588 0.15 MB/sec execute 59 sec latency 132.040 ms 1 1604 0.14 MB/sec execute 60 sec latency 557.258 ms 1 1604 0.14 MB/sec execute 61 sec latency 1557.557 ms 09:54:09 (1783950849) shut down facet: mds2 facet_host: oleg121-server facet_failover_host: oleg121-server 1 1604 0.14 MB/sec execute 62 sec latency 2561.750 ms 1 1604 0.14 MB/sec execute 63 sec latency 3561.954 ms 1 1604 0.14 MB/sec execute 64 sec latency 4562.492 ms 1 1604 0.13 MB/sec execute 65 sec latency 5562.738 ms 1 1604 0.13 MB/sec execute 66 sec latency 6565.517 ms 1 1604 0.13 MB/sec execute 67 sec latency 7565.759 ms 1 1604 0.13 MB/sec execute 68 sec latency 8565.916 ms 1 1604 0.13 MB/sec execute 69 sec latency 9570.017 ms 1 1604 0.12 MB/sec execute 70 sec latency 10571.205 ms 1 1604 0.12 MB/sec execute 71 sec latency 11572.493 ms 1 1604 0.12 MB/sec execute 72 sec latency 12573.342 ms Failover mds2 to oleg121-server mount facets: mds2 1 1604 0.12 MB/sec execute 73 sec latency 13575.052 ms 1 1604 0.12 MB/sec execute 74 sec latency 14585.052 ms 1 1604 0.12 MB/sec execute 75 sec latency 15585.222 ms 1 1604 0.11 MB/sec execute 76 sec latency 16585.516 ms 1 1604 0.11 MB/sec execute 77 sec latency 17585.776 ms 1 1604 0.11 MB/sec execute 78 sec latency 18586.128 ms 1 1604 0.11 MB/sec execute 79 sec latency 19586.510 ms 1 1604 0.11 MB/sec execute 80 sec latency 20593.574 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1604 0.11 MB/sec execute 81 sec latency 21593.829 ms 1 1604 0.11 MB/sec execute 82 sec latency 22594.024 ms 1 1604 0.10 MB/sec execute 83 sec latency 23594.301 ms 1 1604 0.10 MB/sec execute 84 sec latency 24594.586 ms 1 1604 0.10 MB/sec execute 85 sec latency 25594.985 ms 1 1604 0.10 MB/sec execute 86 sec latency 26595.212 ms 1 1604 0.10 MB/sec execute 87 sec latency 27595.685 ms oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all 1 1604 0.10 MB/sec execute 88 sec latency 28596.350 ms pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 1 1604 0.10 MB/sec execute 89 sec latency 29596.604 ms Started lustre-MDT0001 09:54:37 (1783950877) targets are mounted 09:54:37 (1783950877) facet_failover done 1 1604 0.10 MB/sec execute 90 sec latency 30596.726 ms 1 1607 0.10 MB/sec execute 91 sec latency 31476.282 ms 1 1643 0.09 MB/sec execute 92 sec latency 153.042 ms 1 1660 0.09 MB/sec execute 93 sec latency 210.086 ms 1 1691 0.09 MB/sec execute 94 sec latency 130.692 ms 1 1727 0.09 MB/sec execute 95 sec latency 119.085 ms 1 1779 0.09 MB/sec execute 96 sec latency 83.675 ms 1 1825 0.09 MB/sec execute 97 sec latency 183.170 ms oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 1855 0.09 MB/sec execute 98 sec latency 156.081 ms 1 1877 0.09 MB/sec execute 99 sec latency 175.853 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 0 cleanup 105 sec 0 cleanup 105 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 254 205.572 35656.363 Close 199 174.939 31476.259 Rename 11 98.725 206.981 Unlink 24 30.716 55.683 Qpathinfo 211 21.182 157.885 Qfileinfo 57 4.258 45.712 Qfsinfo 33 3.839 30.959 Sfileinfo 34 102.230 1110.726 Find 75 15.120 47.339 WriteX 198 16.740 87.455 ReadX 297 0.183 20.355 Flush 30 60.919 154.303 Throughput 0.0897049 MB/sec 1 clients 1 procs max_latency=35656.378 ms stopping dbench on /mnt/lustre at Mon Jul 13 09:54:52 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2340 1268964 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1269208 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 50608 3555352 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5048 3601716 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 55656 7157068 1% /mnt/lustre striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Jul 13 09:55:02 EDT 2026 waiting for dbench pid 67974 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 21 0.46 MB/sec warmup 1 sec latency 24.201 ms 1 110 2.62 MB/sec warmup 2 sec latency 82.405 ms test_26 fail mds1 3 times 1 139 2.01 MB/sec warmup 3 sec latency 213.976 ms 1 172 1.76 MB/sec warmup 4 sec latency 115.592 ms Failing mds1 on oleg121-server 1 189 1.52 MB/sec warmup 5 sec latency 473.060 ms Stopping /mnt/lustre-mds1 (opts:) on oleg121-server 1 223 1.44 MB/sec warmup 6 sec latency 149.656 ms 1 262 1.42 MB/sec warmup 7 sec latency 115.433 ms 1 262 1.24 MB/sec warmup 8 sec latency 1053.977 ms 1 262 1.10 MB/sec warmup 9 sec latency 2054.534 ms 09:55:11 (1783950911) shut down facet: mds1 facet_host: oleg121-server facet_failover_host: oleg121-server 1 262 0.99 MB/sec warmup 10 sec latency 3054.815 ms 1 262 0.90 MB/sec warmup 11 sec latency 4055.013 ms 1 262 0.83 MB/sec warmup 12 sec latency 5055.317 ms 1 262 0.76 MB/sec warmup 13 sec latency 6056.314 ms 1 262 0.71 MB/sec warmup 14 sec latency 7056.538 ms 1 262 0.66 MB/sec warmup 15 sec latency 8058.778 ms 1 262 0.62 MB/sec warmup 16 sec latency 9059.041 ms 1 262 0.58 MB/sec warmup 17 sec latency 10059.845 ms 1 262 0.55 MB/sec warmup 18 sec latency 11060.218 ms 1 262 0.52 MB/sec warmup 19 sec latency 12060.670 ms Failover mds1 to oleg121-server mount facets: mds1 1 262 0.00 MB/sec execute 1 sec latency 14061.002 ms 1 262 0.00 MB/sec execute 2 sec latency 15061.339 ms 1 262 0.00 MB/sec execute 3 sec latency 16063.227 ms 1 262 0.00 MB/sec execute 4 sec latency 17063.797 ms 1 262 0.00 MB/sec execute 5 sec latency 18064.003 ms 1 262 0.00 MB/sec execute 6 sec latency 19065.415 ms 1 262 0.00 MB/sec execute 7 sec latency 20067.594 ms 1 262 0.00 MB/sec execute 8 sec latency 21069.109 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 262 0.00 MB/sec execute 9 sec latency 22069.287 ms 1 262 0.00 MB/sec execute 10 sec latency 23077.804 ms 1 262 0.00 MB/sec execute 11 sec latency 24078.021 ms 1 262 0.00 MB/sec execute 12 sec latency 25078.346 ms 1 262 0.00 MB/sec execute 13 sec latency 26080.307 ms 1 262 0.00 MB/sec execute 14 sec latency 27080.503 ms 1 262 0.00 MB/sec execute 15 sec latency 28080.807 ms 1 262 0.00 MB/sec execute 16 sec latency 29080.998 ms 1 262 0.00 MB/sec execute 17 sec latency 30081.293 ms 1 262 0.00 MB/sec execute 18 sec latency 31081.499 ms 1 262 0.00 MB/sec execute 19 sec latency 32082.429 ms 1 262 0.00 MB/sec execute 20 sec latency 33083.009 ms oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all 1 262 0.00 MB/sec execute 21 sec latency 34083.513 ms pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 1 262 0.00 MB/sec execute 22 sec latency 35086.828 ms 1 262 0.00 MB/sec execute 23 sec latency 36087.857 ms Started lustre-MDT0000 09:55:45 (1783950945) targets are mounted 09:55:45 (1783950945) facet_failover done 1 262 0.00 MB/sec execute 24 sec latency 37092.406 ms 1 274 0.01 MB/sec execute 25 sec latency 37473.587 ms 1 301 0.04 MB/sec execute 26 sec latency 165.154 ms 1 344 0.09 MB/sec execute 27 sec latency 110.755 ms 1 381 0.13 MB/sec execute 28 sec latency 152.499 ms 1 409 0.15 MB/sec execute 29 sec latency 211.025 ms 1 437 0.18 MB/sec execute 30 sec latency 192.560 ms oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 469 0.21 MB/sec execute 31 sec latency 122.934 ms 1 500 0.24 MB/sec execute 32 sec latency 138.194 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 522 0.25 MB/sec execute 33 sec latency 191.633 ms 1 613 0.32 MB/sec execute 34 sec latency 169.927 ms 1 669 0.32 MB/sec execute 35 sec latency 93.661 ms 1 711 0.31 MB/sec execute 36 sec latency 103.710 ms 1 765 0.31 MB/sec execute 37 sec latency 94.777 ms 1 806 0.30 MB/sec execute 38 sec latency 127.393 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2676 1268628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2412 1268892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 51276 3553848 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 8264 3593480 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 59540 7147328 1% /mnt/lustre 1 876 0.30 MB/sec execute 39 sec latency 108.775 ms 1 884 0.29 MB/sec execute 40 sec latency 748.075 ms 1 885 0.29 MB/sec execute 41 sec latency 1618.672 ms 1 903 0.28 MB/sec execute 42 sec latency 244.895 ms 1 928 0.27 MB/sec execute 43 sec latency 110.351 ms 1 959 0.27 MB/sec execute 44 sec latency 147.134 ms 1 980 0.27 MB/sec execute 45 sec latency 130.081 ms 1 980 0.26 MB/sec execute 46 sec latency 1143.789 ms 1 980 0.25 MB/sec execute 47 sec latency 2144.274 ms 1 1006 0.25 MB/sec execute 48 sec latency 2227.307 ms 1 1051 0.26 MB/sec execute 49 sec latency 148.554 ms 1 1095 0.26 MB/sec execute 50 sec latency 108.137 ms 1 1136 0.26 MB/sec execute 51 sec latency 88.469 ms test_26 fail mds2 4 times 1 1163 0.25 MB/sec execute 52 sec latency 199.349 ms 1 1190 0.25 MB/sec execute 53 sec latency 117.182 ms Failing mds2 on oleg121-server 1 1216 0.24 MB/sec execute 54 sec latency 189.384 ms 1 1244 0.24 MB/sec execute 55 sec latency 105.701 ms Stopping /mnt/lustre-mds2 (opts:) on oleg121-server 1 1269 0.24 MB/sec execute 56 sec latency 164.577 ms 1 1378 0.28 MB/sec execute 57 sec latency 396.542 ms 1 1378 0.28 MB/sec execute 58 sec latency 1396.744 ms 1 1378 0.27 MB/sec execute 59 sec latency 2397.112 ms 1 1378 0.27 MB/sec execute 60 sec latency 3398.629 ms 09:56:22 (1783950982) shut down facet: mds2 facet_host: oleg121-server facet_failover_host: oleg121-server 1 1378 0.26 MB/sec execute 61 sec latency 4401.048 ms 1 1378 0.26 MB/sec execute 62 sec latency 5401.292 ms 1 1378 0.25 MB/sec execute 63 sec latency 6402.181 ms 1 1378 0.25 MB/sec execute 64 sec latency 7403.417 ms 1 1378 0.25 MB/sec execute 65 sec latency 8403.780 ms 1 1378 0.24 MB/sec execute 66 sec latency 9404.001 ms 1 1378 0.24 MB/sec execute 67 sec latency 10404.807 ms 1 1378 0.24 MB/sec execute 68 sec latency 11405.635 ms 1 1378 0.23 MB/sec execute 69 sec latency 12405.826 ms 1 1378 0.23 MB/sec execute 70 sec latency 13406.036 ms Failover mds2 to oleg121-server mount facets: mds2 1 1378 0.23 MB/sec execute 71 sec latency 14406.171 ms 1 1378 0.22 MB/sec execute 72 sec latency 15406.390 ms 1 1378 0.22 MB/sec execute 73 sec latency 16406.802 ms 1 1378 0.22 MB/sec execute 74 sec latency 17408.569 ms 1 1378 0.21 MB/sec execute 75 sec latency 18411.571 ms 1 1378 0.21 MB/sec execute 76 sec latency 19411.779 ms 1 1378 0.21 MB/sec execute 77 sec latency 20412.114 ms 1 1378 0.21 MB/sec execute 78 sec latency 21412.534 ms 1 1378 0.20 MB/sec execute 79 sec latency 22415.341 ms 1 1378 0.20 MB/sec execute 80 sec latency 23415.554 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1378 0.20 MB/sec execute 81 sec latency 24416.101 ms 1 1378 0.20 MB/sec execute 82 sec latency 25416.746 ms 1 1378 0.19 MB/sec execute 83 sec latency 26419.326 ms 1 1378 0.19 MB/sec execute 84 sec latency 27428.642 ms 1 1378 0.19 MB/sec execute 85 sec latency 28431.918 ms 1 1378 0.19 MB/sec execute 86 sec latency 29432.536 ms oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all 1 1378 0.18 MB/sec execute 87 sec latency 30438.259 ms 1 1378 0.18 MB/sec execute 88 sec latency 31439.222 ms pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 1 1378 0.18 MB/sec execute 89 sec latency 32442.086 ms Started lustre-MDT0001 09:56:51 (1783951011) targets are mounted 09:56:51 (1783951011) facet_failover done 1 1378 0.18 MB/sec execute 90 sec latency 33443.759 ms 1 1393 0.18 MB/sec execute 91 sec latency 34081.004 ms 1 1421 0.17 MB/sec execute 92 sec latency 132.667 ms 1 1457 0.17 MB/sec execute 93 sec latency 114.252 ms 1 1474 0.17 MB/sec execute 94 sec latency 477.203 ms 1 1505 0.17 MB/sec execute 95 sec latency 167.177 ms 1 1553 0.17 MB/sec execute 96 sec latency 128.473 ms oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 1595 0.17 MB/sec execute 97 sec latency 111.018 ms 1 1634 0.17 MB/sec execute 98 sec latency 140.053 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1660 0.16 MB/sec execute 99 sec latency 143.951 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 228 78.537 252.612 Close 200 203.321 37473.564 Rename 11 126.420 199.327 Unlink 22 32.799 65.309 Qpathinfo 152 246.950 34080.959 Qfileinfo 77 2.645 22.921 Qfsinfo 33 6.370 56.524 Sfileinfo 34 72.168 143.993 Find 61 15.158 65.392 WriteX 328 17.575 126.157 ReadX 257 0.137 5.661 Flush 27 126.989 2227.290 Throughput 0.164101 MB/sec 1 clients 1 procs max_latency=37473.587 ms stopping dbench on /mnt/lustre at Mon Jul 13 09:57:06 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Jul 13 09:57:12 EDT 2026 waiting for dbench pid 70059 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 28 0.92 MB/sec warmup 1 sec latency 26.492 ms 1 92 2.10 MB/sec warmup 2 sec latency 110.055 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 139 2.00 MB/sec warmup 3 sec latency 123.770 ms 1 173 1.76 MB/sec warmup 4 sec latency 137.907 ms 1 197 1.57 MB/sec warmup 5 sec latency 458.073 ms 1 227 1.45 MB/sec warmup 6 sec latency 224.109 ms 1 271 1.45 MB/sec warmup 7 sec latency 138.266 ms 1 303 1.39 MB/sec warmup 8 sec latency 180.227 ms 1 341 1.36 MB/sec warmup 9 sec latency 111.982 ms 1 391 1.38 MB/sec warmup 10 sec latency 91.988 ms 1 437 1.40 MB/sec warmup 11 sec latency 198.100 ms 1 485 1.41 MB/sec warmup 12 sec latency 217.844 ms 1 573 1.48 MB/sec warmup 13 sec latency 127.662 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Jul 13 09:57:26 EDT 2026 with return code 0 1 653 1.50 MB/sec warmup 14 sec latency 128.423 ms 70059 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 70059 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished