== replay-dual test 26: dbench and tar with mds failover ========================================================== 17:25:48 (1790976348) Starting client oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre Started clients oleg144-client.virtnet: 192.168.201.144@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65608 Started dbench loop with 65610 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on '/usr/share/dbench/client.txt' -> 'client.txt' lustre-MDT0000_UUID 1414116 1944 1269360 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3580680 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3108 7186136 1% /mnt/lustre running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Oct 2 17:25:53 EDT 2026 waiting for dbench pid 65665 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 43.292 ms 1 90 2.09 MB/sec warmup 2 sec latency 167.334 ms 1 129 1.92 MB/sec warmup 3 sec latency 156.302 ms 1 165 1.70 MB/sec warmup 4 sec latency 165.802 ms 1 190 1.51 MB/sec warmup 5 sec latency 519.207 ms 1 234 1.48 MB/sec warmup 6 sec latency 123.160 ms 1 289 1.53 MB/sec warmup 7 sec latency 77.888 ms 1 331 1.50 MB/sec warmup 8 sec latency 107.059 ms test_26 fail mds1 1 times 1 373 1.48 MB/sec warmup 9 sec latency 145.695 ms 1 405 1.43 MB/sec warmup 10 sec latency 141.692 ms Failing mds1 on oleg144-server 1 440 1.40 MB/sec warmup 11 sec latency 151.187 ms Stopping /mnt/lustre-mds1 (opts:) on oleg144-server 1 469 1.37 MB/sec warmup 12 sec latency 151.065 ms 1 501 1.34 MB/sec warmup 13 sec latency 146.248 ms 1 501 1.25 MB/sec warmup 14 sec latency 1018.741 ms 1 501 1.16 MB/sec warmup 15 sec latency 2019.007 ms 17:26:08 (1790976368) shut down facet: mds1 facet_host: oleg144-server facet_failover_host: oleg144-server 1 501 1.09 MB/sec warmup 16 sec latency 3020.737 ms 1 501 1.03 MB/sec warmup 17 sec latency 4020.972 ms 1 501 0.97 MB/sec warmup 18 sec latency 5021.135 ms 1 501 0.92 MB/sec warmup 19 sec latency 6021.329 ms 1 501 0.00 MB/sec execute 1 sec latency 8026.831 ms 1 501 0.00 MB/sec execute 2 sec latency 9027.023 ms 1 501 0.00 MB/sec execute 3 sec latency 10027.658 ms 1 501 0.00 MB/sec execute 4 sec latency 11029.415 ms 1 501 0.00 MB/sec execute 5 sec latency 12029.629 ms Failover mds1 to oleg144-server mount facets: mds1 1 501 0.00 MB/sec execute 6 sec latency 13031.774 ms 1 501 0.00 MB/sec execute 7 sec latency 14034.645 ms 1 501 0.00 MB/sec execute 8 sec latency 15034.977 ms 1 501 0.00 MB/sec execute 9 sec latency 16035.461 ms 1 501 0.00 MB/sec execute 10 sec latency 17035.710 ms 1 501 0.00 MB/sec execute 11 sec latency 18042.261 ms 1 501 0.00 MB/sec execute 12 sec latency 19042.470 ms 1 501 0.00 MB/sec execute 13 sec latency 20042.793 ms 1 501 0.00 MB/sec execute 14 sec latency 21043.495 ms 1 501 0.00 MB/sec execute 15 sec latency 22044.465 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 501 0.00 MB/sec execute 16 sec latency 23044.769 ms 1 501 0.00 MB/sec execute 17 sec latency 24048.223 ms 1 501 0.00 MB/sec execute 18 sec latency 25049.044 ms 1 501 0.00 MB/sec execute 19 sec latency 26055.239 ms 1 501 0.00 MB/sec execute 20 sec latency 27056.554 ms 1 501 0.00 MB/sec execute 21 sec latency 28056.769 ms 1 501 0.00 MB/sec execute 22 sec latency 29056.976 ms 1 501 0.00 MB/sec execute 23 sec latency 30057.177 ms 1 501 0.00 MB/sec execute 24 sec latency 31057.586 ms 1 501 0.00 MB/sec execute 25 sec latency 32057.769 ms 1 501 0.00 MB/sec execute 26 sec latency 33059.022 ms oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all 1 501 0.00 MB/sec execute 27 sec latency 34059.498 ms pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 1 501 0.00 MB/sec execute 28 sec latency 35060.033 ms Started lustre-MDT0000 17:26:41 (1790976401) targets are mounted 17:26:41 (1790976401) facet_failover done 1 501 0.00 MB/sec execute 29 sec latency 36060.651 ms 1 522 0.02 MB/sec execute 30 sec latency 36264.087 ms 1 606 0.11 MB/sec execute 31 sec latency 132.918 ms 1 654 0.11 MB/sec execute 32 sec latency 122.482 ms 1 695 0.11 MB/sec execute 33 sec latency 99.789 ms 1 731 0.11 MB/sec execute 34 sec latency 97.292 ms 1 762 0.11 MB/sec execute 35 sec latency 200.071 ms oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 789 0.11 MB/sec execute 36 sec latency 178.241 ms 1 815 0.11 MB/sec execute 37 sec latency 132.048 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 863 0.11 MB/sec execute 38 sec latency 145.447 ms 1 884 0.11 MB/sec execute 39 sec latency 168.335 ms 1 884 0.11 MB/sec execute 40 sec latency 1169.968 ms 1 902 0.10 MB/sec execute 41 sec latency 1516.863 ms 1 940 0.10 MB/sec execute 42 sec latency 115.811 ms 1 981 0.10 MB/sec execute 43 sec latency 126.144 ms 1 1051 0.12 MB/sec execute 44 sec latency 100.093 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2120 1269184 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1916 1269388 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 10500 3595024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 45060 3559456 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 55560 7154480 1% /mnt/lustre 1 1112 0.12 MB/sec execute 45 sec latency 73.173 ms 1 1147 0.12 MB/sec execute 46 sec latency 122.771 ms 1 1198 0.12 MB/sec execute 47 sec latency 151.406 ms 1 1242 0.12 MB/sec execute 48 sec latency 80.197 ms 1 1285 0.12 MB/sec execute 49 sec latency 113.791 ms 1 1404 0.17 MB/sec execute 50 sec latency 544.522 ms 1 1428 0.17 MB/sec execute 51 sec latency 796.756 ms 1 1468 0.16 MB/sec execute 52 sec latency 248.524 ms 1 1509 0.16 MB/sec execute 53 sec latency 96.078 ms test_26 fail mds2 2 times 1 1566 0.16 MB/sec execute 54 sec latency 90.105 ms 1 1621 0.16 MB/sec execute 55 sec latency 113.260 ms Failing mds2 on oleg144-server 1 1659 0.16 MB/sec execute 56 sec latency 118.843 ms Stopping /mnt/lustre-mds2 (opts:) on oleg144-server 1 1696 0.15 MB/sec execute 57 sec latency 136.078 ms 1 1713 0.15 MB/sec execute 58 sec latency 772.885 ms 1 1713 0.15 MB/sec execute 59 sec latency 1773.703 ms 17:27:13 (1790976433) shut down facet: mds2 facet_host: oleg144-server facet_failover_host: oleg144-server 1 1713 0.15 MB/sec execute 60 sec latency 2777.056 ms 1 1713 0.14 MB/sec execute 61 sec latency 3777.230 ms 1 1713 0.14 MB/sec execute 62 sec latency 4777.409 ms 1 1713 0.14 MB/sec execute 63 sec latency 5777.598 ms 1 1713 0.14 MB/sec execute 64 sec latency 6778.175 ms 1 1713 0.13 MB/sec execute 65 sec latency 7778.369 ms 1 1713 0.13 MB/sec execute 66 sec latency 8778.577 ms 1 1713 0.13 MB/sec execute 67 sec latency 9778.923 ms 1 1713 0.13 MB/sec execute 68 sec latency 10779.659 ms 1 1713 0.13 MB/sec execute 69 sec latency 11780.002 ms Failover mds2 to oleg144-server mount facets: mds2 1 1713 0.13 MB/sec execute 70 sec latency 12782.576 ms 1 1713 0.12 MB/sec execute 71 sec latency 13782.764 ms 1 1713 0.12 MB/sec execute 72 sec latency 14782.978 ms 1 1713 0.12 MB/sec execute 73 sec latency 15784.978 ms 1 1713 0.12 MB/sec execute 74 sec latency 16785.162 ms 1 1713 0.12 MB/sec execute 75 sec latency 17785.513 ms 1 1713 0.12 MB/sec execute 76 sec latency 18791.171 ms 1 1713 0.11 MB/sec execute 77 sec latency 19791.514 ms 1 1713 0.11 MB/sec execute 78 sec latency 20791.784 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1713 0.11 MB/sec execute 79 sec latency 21792.024 ms 1 1713 0.11 MB/sec execute 80 sec latency 22792.505 ms 1 1713 0.11 MB/sec execute 81 sec latency 23796.972 ms 1 1713 0.11 MB/sec execute 82 sec latency 24802.690 ms 1 1713 0.11 MB/sec execute 83 sec latency 25807.712 ms 1 1713 0.10 MB/sec execute 84 sec latency 26808.426 ms 1 1713 0.10 MB/sec execute 85 sec latency 27808.661 ms oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all 1 1713 0.10 MB/sec execute 86 sec latency 28819.746 ms 1 1713 0.10 MB/sec execute 87 sec latency 29820.014 ms pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 1 1713 0.10 MB/sec execute 88 sec latency 30821.502 ms Started lustre-MDT0001 17:27:42 (1790976462) targets are mounted 17:27:42 (1790976462) facet_failover done 1 1715 0.10 MB/sec execute 89 sec latency 31803.475 ms 1 1770 0.10 MB/sec execute 90 sec latency 99.513 ms 1 1820 0.10 MB/sec execute 91 sec latency 114.903 ms 1 1857 0.10 MB/sec execute 92 sec latency 98.994 ms 1 1907 0.10 MB/sec execute 93 sec latency 123.618 ms 1 1965 0.10 MB/sec execute 94 sec latency 107.489 ms 1 2004 0.10 MB/sec execute 95 sec latency 117.326 ms oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 2069 0.10 MB/sec execute 96 sec latency 96.192 ms 1 2128 0.09 MB/sec execute 97 sec latency 88.097 ms 1 2158 0.09 MB/sec execute 98 sec latency 137.438 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2194 0.09 MB/sec execute 99 sec latency 159.558 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 318 270.948 36264.073 Close 272 17.075 796.735 Rename 11 98.670 178.218 Unlink 28 31.904 57.295 Qpathinfo 289 15.447 88.359 Qfileinfo 68 1.979 14.477 Qfsinfo 33 3.901 29.511 Sfileinfo 34 60.694 127.662 Find 92 13.497 55.307 WriteX 221 12.929 96.744 ReadX 348 0.258 26.541 Flush 30 53.522 97.276 Throughput 0.0937105 MB/sec 1 clients 1 procs max_latency=36264.087 ms stopping dbench on /mnt/lustre at Fri Oct 2 17:27:56 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2352 1268952 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 5908 3601056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 47472 3559332 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 53380 7160388 1% /mnt/lustre striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Oct 2 17:28:06 EDT 2026 waiting for dbench pid 67894 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 8 0.00 MB/sec warmup 1 sec latency 151.731 ms 1 70 1.57 MB/sec warmup 2 sec latency 79.801 ms 1 113 1.74 MB/sec warmup 3 sec latency 187.779 ms test_26 fail mds1 3 times 1 139 1.50 MB/sec warmup 4 sec latency 210.132 ms 1 158 1.30 MB/sec warmup 5 sec latency 236.133 ms Failing mds1 on oleg144-server 1 184 1.23 MB/sec warmup 6 sec latency 171.234 ms 1 198 1.12 MB/sec warmup 7 sec latency 610.349 ms Stopping /mnt/lustre-mds1 (opts:) on oleg144-server 1 221 1.07 MB/sec warmup 8 sec latency 213.976 ms 1 252 1.06 MB/sec warmup 9 sec latency 131.154 ms 1 284 1.06 MB/sec warmup 10 sec latency 156.963 ms 1 330 1.09 MB/sec warmup 11 sec latency 88.420 ms 1 339 1.02 MB/sec warmup 12 sec latency 799.916 ms 1 339 0.94 MB/sec warmup 13 sec latency 1800.118 ms 1 339 0.87 MB/sec warmup 14 sec latency 2800.370 ms 1 339 0.82 MB/sec warmup 15 sec latency 3800.635 ms 1 339 0.77 MB/sec warmup 16 sec latency 4800.861 ms 1 339 0.72 MB/sec warmup 17 sec latency 5801.108 ms 17:28:24 (1790976504) shut down facet: mds1 facet_host: oleg144-server facet_failover_host: oleg144-server 1 339 0.68 MB/sec warmup 18 sec latency 6801.550 ms 1 339 0.64 MB/sec warmup 19 sec latency 7801.747 ms 1 339 0.00 MB/sec execute 1 sec latency 9802.076 ms 1 339 0.00 MB/sec execute 2 sec latency 10802.277 ms 1 339 0.00 MB/sec execute 3 sec latency 11803.993 ms 1 339 0.00 MB/sec execute 4 sec latency 12804.348 ms 1 339 0.00 MB/sec execute 5 sec latency 13805.404 ms 1 339 0.00 MB/sec execute 6 sec latency 14805.657 ms 1 339 0.00 MB/sec execute 7 sec latency 15805.843 ms Failover mds1 to oleg144-server mount facets: mds1 1 339 0.00 MB/sec execute 8 sec latency 16805.985 ms 1 339 0.00 MB/sec execute 9 sec latency 17808.471 ms 1 339 0.00 MB/sec execute 10 sec latency 18808.837 ms 1 339 0.00 MB/sec execute 11 sec latency 19833.439 ms 1 339 0.00 MB/sec execute 12 sec latency 20833.638 ms 1 339 0.00 MB/sec execute 13 sec latency 21833.798 ms 1 339 0.00 MB/sec execute 14 sec latency 22833.979 ms 1 339 0.00 MB/sec execute 15 sec latency 23834.178 ms 1 339 0.00 MB/sec execute 16 sec latency 24855.951 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 339 0.00 MB/sec execute 17 sec latency 25856.284 ms 1 339 0.00 MB/sec execute 18 sec latency 26857.132 ms 1 339 0.00 MB/sec execute 19 sec latency 27857.354 ms 1 339 0.00 MB/sec execute 20 sec latency 28857.640 ms 1 339 0.00 MB/sec execute 21 sec latency 29861.502 ms 1 339 0.00 MB/sec execute 22 sec latency 30861.707 ms 1 339 0.00 MB/sec execute 23 sec latency 31863.476 ms 1 339 0.00 MB/sec execute 24 sec latency 32871.185 ms 1 339 0.00 MB/sec execute 25 sec latency 33871.406 ms 1 339 0.00 MB/sec execute 26 sec latency 34871.664 ms 1 339 0.00 MB/sec execute 27 sec latency 35874.077 ms 1 339 0.00 MB/sec execute 28 sec latency 36874.545 ms oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all 1 339 0.00 MB/sec execute 29 sec latency 37874.750 ms pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 17:28:57 (1790976537) targets are mounted 17:28:57 (1790976537) facet_failover done 1 339 0.00 MB/sec execute 30 sec latency 38874.914 ms 1 357 0.02 MB/sec execute 31 sec latency 39584.192 ms 1 389 0.05 MB/sec execute 32 sec latency 303.879 ms 1 438 0.09 MB/sec execute 33 sec latency 123.291 ms 1 490 0.14 MB/sec execute 34 sec latency 120.558 ms 1 564 0.19 MB/sec execute 35 sec latency 111.373 ms oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 639 0.24 MB/sec execute 36 sec latency 85.778 ms 1 674 0.24 MB/sec execute 37 sec latency 116.771 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 710 0.23 MB/sec execute 38 sec latency 129.731 ms 1 736 0.23 MB/sec execute 39 sec latency 198.762 ms 1 776 0.23 MB/sec execute 40 sec latency 122.698 ms 1 825 0.22 MB/sec execute 41 sec latency 86.323 ms 1 873 0.22 MB/sec execute 42 sec latency 101.646 ms 1 905 0.22 MB/sec execute 43 sec latency 176.815 ms 1 936 0.22 MB/sec execute 44 sec latency 123.346 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2852 1268452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2612 1268692 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38880 3568084 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 31896 3575068 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 70776 7143152 1% /mnt/lustre 1 967 0.21 MB/sec execute 45 sec latency 167.892 ms 1 988 0.21 MB/sec execute 46 sec latency 161.602 ms 1 1035 0.23 MB/sec execute 47 sec latency 145.133 ms 1 1035 0.22 MB/sec execute 48 sec latency 1087.528 ms 1 1039 0.22 MB/sec execute 49 sec latency 1832.819 ms 1 1067 0.21 MB/sec execute 50 sec latency 247.829 ms 1 1126 0.21 MB/sec execute 51 sec latency 79.363 ms 1 1160 0.21 MB/sec execute 52 sec latency 122.389 ms 1 1193 0.20 MB/sec execute 53 sec latency 79.037 ms 1 1233 0.20 MB/sec execute 54 sec latency 108.311 ms 1 1384 0.25 MB/sec execute 55 sec latency 100.498 ms test_26 fail mds2 4 times 1 1421 0.24 MB/sec execute 56 sec latency 105.007 ms 1 1468 0.24 MB/sec execute 57 sec latency 90.776 ms Failing mds2 on oleg144-server 1 1497 0.24 MB/sec execute 58 sec latency 172.382 ms Stopping /mnt/lustre-mds2 (opts:) on oleg144-server 1 1558 0.23 MB/sec execute 59 sec latency 106.614 ms 1 1640 0.23 MB/sec execute 60 sec latency 73.989 ms 1 1707 0.23 MB/sec execute 61 sec latency 83.882 ms 1 1775 0.23 MB/sec execute 62 sec latency 173.706 ms 1 1855 0.22 MB/sec execute 63 sec latency 104.330 ms 17:29:30 (1790976570) shut down facet: mds2 facet_host: oleg144-server facet_failover_host: oleg144-server 1 1928 0.22 MB/sec execute 64 sec latency 103.189 ms 1 2020 0.22 MB/sec execute 65 sec latency 62.639 ms 1 2148 0.22 MB/sec execute 66 sec latency 55.789 ms 1 2218 0.22 MB/sec execute 67 sec latency 62.608 ms 1 2259 0.21 MB/sec execute 68 sec latency 734.856 ms 1 2259 0.21 MB/sec execute 69 sec latency 1735.046 ms 1 2259 0.21 MB/sec execute 70 sec latency 2735.312 ms 1 2259 0.20 MB/sec execute 71 sec latency 3736.205 ms 1 2259 0.20 MB/sec execute 72 sec latency 4737.717 ms 1 2259 0.20 MB/sec execute 73 sec latency 5737.897 ms Failover mds2 to oleg144-server mount facets: mds2 1 2259 0.20 MB/sec execute 74 sec latency 6746.395 ms 1 2259 0.19 MB/sec execute 75 sec latency 7747.488 ms 1 2259 0.19 MB/sec execute 76 sec latency 8755.183 ms 1 2259 0.19 MB/sec execute 77 sec latency 9755.391 ms 1 2259 0.19 MB/sec execute 78 sec latency 10758.541 ms 1 2259 0.18 MB/sec execute 79 sec latency 11759.517 ms 1 2259 0.18 MB/sec execute 80 sec latency 12760.136 ms 1 2259 0.18 MB/sec execute 81 sec latency 13760.368 ms 1 2259 0.18 MB/sec execute 82 sec latency 14760.733 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.17 MB/sec execute 83 sec latency 15761.478 ms 1 2259 0.17 MB/sec execute 84 sec latency 16761.668 ms 1 2259 0.17 MB/sec execute 85 sec latency 17761.879 ms 1 2259 0.17 MB/sec execute 86 sec latency 18762.073 ms 1 2259 0.17 MB/sec execute 87 sec latency 19762.398 ms 1 2259 0.16 MB/sec execute 88 sec latency 20762.709 ms 1 2259 0.16 MB/sec execute 89 sec latency 21762.989 ms oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all 1 2259 0.16 MB/sec execute 90 sec latency 22763.233 ms pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 1 2259 0.16 MB/sec execute 91 sec latency 23763.448 ms Started lustre-MDT0001 17:29:58 (1790976598) targets are mounted 17:29:58 (1790976598) facet_failover done 1 2327 0.17 MB/sec execute 92 sec latency 23998.252 ms 1 2367 0.17 MB/sec execute 93 sec latency 101.225 ms 1 2400 0.17 MB/sec execute 94 sec latency 134.627 ms 1 2429 0.17 MB/sec execute 95 sec latency 127.814 ms 1 2455 0.16 MB/sec execute 96 sec latency 143.072 ms 1 2503 0.17 MB/sec execute 97 sec latency 126.516 ms 1 2586 0.17 MB/sec execute 98 sec latency 77.411 ms oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 2608 0.17 MB/sec execute 99 sec latency 135.134 ms 1 cleanup 100 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 392 210.475 39584.169 Close 336 10.111 41.806 Rename 17 89.592 198.727 Unlink 41 25.116 65.961 Qpathinfo 357 15.738 125.256 Qfileinfo 106 1.759 9.199 Qfsinfo 52 4.363 29.834 Sfileinfo 50 63.299 145.466 Find 126 11.805 85.765 WriteX 376 13.019 101.210 ReadX 540 1.550 247.814 Flush 34 56.681 176.795 Throughput 0.168793 MB/sec 1 clients 1 procs max_latency=39584.192 ms stopping dbench on /mnt/lustre at Fri Oct 2 17:30:10 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench tar: Unexpected EOF in archive tar: Unexpected EOF in archive looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Oct 2 17:30:20 EDT 2026 waiting for dbench pid 69974 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 3 0.00 MB/sec warmup 1 sec latency 647.384 ms 1 31 0.52 MB/sec warmup 2 sec latency 31.371 ms 1 80 1.18 MB/sec warmup 3 sec latency 135.327 ms 1 111 1.31 MB/sec warmup 4 sec latency 131.098 ms 1 131 1.15 MB/sec warmup 5 sec latency 240.835 ms 1 155 1.09 MB/sec warmup 6 sec latency 232.642 ms 1 184 1.05 MB/sec warmup 7 sec latency 343.750 ms 1 215 1.04 MB/sec warmup 8 sec latency 417.498 ms 1 255 1.07 MB/sec warmup 9 sec latency 114.412 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Oct 2 17:30:30 EDT 2026 with return code 0 1 290 1.07 MB/sec warmup 10 sec latency 117.416 ms 69974 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 69974 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished