== replay-dual test 26: dbench and tar with mds failover ========================================================== 12:24:39 (1788625479) Starting client oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Started clients oleg317-client.virtnet: 192.168.203.117@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65597 Started dbench loop with 65598 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1944 1269360 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1756 1269548 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5640 3580676 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 7204 7186132 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Sep 5 12:24:44 EDT 2026 waiting for dbench pid 65668 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 12 0.00 MB/sec warmup 1 sec latency 37.226 ms 1 82 1.83 MB/sec warmup 2 sec latency 116.000 ms 1 122 1.83 MB/sec warmup 3 sec latency 167.883 ms 1 164 1.70 MB/sec warmup 4 sec latency 101.218 ms 1 188 1.51 MB/sec warmup 5 sec latency 451.791 ms 1 216 1.39 MB/sec warmup 6 sec latency 153.490 ms 1 258 1.38 MB/sec warmup 7 sec latency 106.679 ms 1 292 1.33 MB/sec warmup 8 sec latency 145.687 ms test_26 fail mds1 1 times 1 323 1.30 MB/sec warmup 9 sec latency 146.729 ms 1 352 1.25 MB/sec warmup 10 sec latency 184.586 ms Failing mds1 on oleg317-server 1 389 1.25 MB/sec warmup 11 sec latency 120.364 ms Stopping /mnt/lustre-mds1 (opts:) on oleg317-server 1 428 1.26 MB/sec warmup 12 sec latency 193.286 ms 1 467 1.26 MB/sec warmup 13 sec latency 150.928 ms 1 501 1.24 MB/sec warmup 14 sec latency 268.816 ms 12:24:59 (1788625499) shut down 1 501 1.16 MB/sec warmup 15 sec latency 1269.198 ms facet: mds1 facet_host: oleg317-server facet_failover_host: oleg317-server 1 501 1.09 MB/sec warmup 16 sec latency 2269.940 ms 1 501 1.03 MB/sec warmup 17 sec latency 3270.123 ms 1 501 0.97 MB/sec warmup 18 sec latency 4270.337 ms 1 501 0.92 MB/sec warmup 19 sec latency 5270.554 ms 1 501 0.00 MB/sec execute 1 sec latency 7272.013 ms 1 501 0.00 MB/sec execute 2 sec latency 8274.213 ms 1 501 0.00 MB/sec execute 3 sec latency 9274.660 ms 1 501 0.00 MB/sec execute 4 sec latency 10275.492 ms 1 501 0.00 MB/sec execute 5 sec latency 11276.906 ms Failover mds1 to oleg317-server mount facets: mds1 1 501 0.00 MB/sec execute 6 sec latency 12277.545 ms 1 501 0.00 MB/sec execute 7 sec latency 13277.767 ms 1 501 0.00 MB/sec execute 8 sec latency 14279.294 ms 1 501 0.00 MB/sec execute 9 sec latency 15279.812 ms 1 501 0.00 MB/sec execute 10 sec latency 16280.030 ms 1 501 0.00 MB/sec execute 11 sec latency 17280.288 ms 1 501 0.00 MB/sec execute 12 sec latency 18280.551 ms 1 501 0.00 MB/sec execute 13 sec latency 19283.276 ms 1 501 0.00 MB/sec execute 14 sec latency 20288.106 ms 1 501 0.00 MB/sec execute 15 sec latency 21289.071 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 501 0.00 MB/sec execute 16 sec latency 22289.269 ms 1 501 0.00 MB/sec execute 17 sec latency 23289.631 ms 1 501 0.00 MB/sec execute 18 sec latency 24290.025 ms 1 501 0.00 MB/sec execute 19 sec latency 25292.488 ms 1 501 0.00 MB/sec execute 20 sec latency 26295.249 ms 1 501 0.00 MB/sec execute 21 sec latency 27295.584 ms 1 501 0.00 MB/sec execute 22 sec latency 28295.924 ms 1 501 0.00 MB/sec execute 23 sec latency 29298.210 ms 1 501 0.00 MB/sec execute 24 sec latency 30299.756 ms 1 501 0.00 MB/sec execute 25 sec latency 31305.963 ms 1 501 0.00 MB/sec execute 26 sec latency 32309.223 ms 1 501 0.00 MB/sec execute 27 sec latency 33309.529 ms 1 501 0.00 MB/sec execute 28 sec latency 34309.860 ms 1 501 0.00 MB/sec execute 29 sec latency 35310.310 ms oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all 1 501 0.00 MB/sec execute 30 sec latency 36310.542 ms 1 501 0.00 MB/sec execute 31 sec latency 37310.859 ms pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 1 501 0.00 MB/sec execute 32 sec latency 38312.970 ms Started lustre-MDT0000 12:25:37 (1788625537) targets are mounted 12:25:37 (1788625537) facet_failover done 1 513 0.01 MB/sec execute 33 sec latency 38979.694 ms 1 552 0.05 MB/sec execute 34 sec latency 220.646 ms 1 582 0.07 MB/sec execute 35 sec latency 911.999 ms 1 642 0.10 MB/sec execute 36 sec latency 113.037 ms 1 674 0.10 MB/sec execute 37 sec latency 122.800 ms 1 710 0.10 MB/sec execute 38 sec latency 177.765 ms 1 738 0.10 MB/sec execute 39 sec latency 118.929 ms oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 778 0.10 MB/sec execute 40 sec latency 141.549 ms 1 808 0.10 MB/sec execute 41 sec latency 155.554 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 857 0.10 MB/sec execute 42 sec latency 105.914 ms 1 885 0.10 MB/sec execute 43 sec latency 131.690 ms 1 916 0.10 MB/sec execute 44 sec latency 139.616 ms 1 957 0.10 MB/sec execute 45 sec latency 135.438 ms 1 988 0.10 MB/sec execute 46 sec latency 123.994 ms 1 1064 0.12 MB/sec execute 47 sec latency 75.383 ms 1 1124 0.11 MB/sec execute 48 sec latency 88.654 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1269152 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1944 1269360 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9616 3593688 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 45664 3560324 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 55280 7154012 1% /mnt/lustre 1 1148 0.11 MB/sec execute 49 sec latency 164.935 ms 1 1172 0.11 MB/sec execute 50 sec latency 647.743 ms 1 1193 0.11 MB/sec execute 51 sec latency 957.102 ms 1 1231 0.11 MB/sec execute 52 sec latency 85.493 ms 1 1274 0.11 MB/sec execute 53 sec latency 123.050 ms 1 1410 0.16 MB/sec execute 54 sec latency 90.903 ms 1 1448 0.16 MB/sec execute 55 sec latency 114.148 ms 1 1499 0.15 MB/sec execute 56 sec latency 104.055 ms 1 1538 0.15 MB/sec execute 57 sec latency 117.994 ms 1 1597 0.15 MB/sec execute 58 sec latency 66.884 ms test_26 fail mds2 2 times 1 1634 0.15 MB/sec execute 59 sec latency 110.218 ms 1 1660 0.15 MB/sec execute 60 sec latency 136.031 ms Failing mds2 on oleg317-server 1 1692 0.14 MB/sec execute 61 sec latency 173.879 ms Stopping /mnt/lustre-mds2 (opts:) on oleg317-server 1 1752 0.14 MB/sec execute 62 sec latency 80.462 ms 1 1752 0.14 MB/sec execute 63 sec latency 1023.602 ms 1 1752 0.14 MB/sec execute 64 sec latency 2025.075 ms 12:26:09 (1788625569) shut down facet: mds2 facet_host: oleg317-server facet_failover_host: oleg317-server 1 1752 0.14 MB/sec execute 65 sec latency 3026.840 ms 1 1752 0.13 MB/sec execute 66 sec latency 4027.092 ms 1 1752 0.13 MB/sec execute 67 sec latency 5027.547 ms 1 1752 0.13 MB/sec execute 68 sec latency 6027.991 ms 1 1752 0.13 MB/sec execute 69 sec latency 7028.254 ms 1 1752 0.13 MB/sec execute 70 sec latency 8028.724 ms 1 1752 0.12 MB/sec execute 71 sec latency 9029.435 ms 1 1752 0.12 MB/sec execute 72 sec latency 10029.663 ms 1 1752 0.12 MB/sec execute 73 sec latency 11029.892 ms 1 1752 0.12 MB/sec execute 74 sec latency 12030.305 ms Failover mds2 to oleg317-server mount facets: mds2 1 1752 0.12 MB/sec execute 75 sec latency 13036.608 ms 1 1752 0.12 MB/sec execute 76 sec latency 14037.326 ms 1 1752 0.11 MB/sec execute 77 sec latency 15037.813 ms 1 1752 0.11 MB/sec execute 78 sec latency 16038.091 ms 1 1752 0.11 MB/sec execute 79 sec latency 17038.756 ms 1 1752 0.11 MB/sec execute 80 sec latency 18038.925 ms 1 1752 0.11 MB/sec execute 81 sec latency 19039.571 ms 1 1752 0.11 MB/sec execute 82 sec latency 20043.534 ms 1 1752 0.11 MB/sec execute 83 sec latency 21047.024 ms 1 1752 0.10 MB/sec execute 84 sec latency 22047.425 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1752 0.10 MB/sec execute 85 sec latency 23050.818 ms 1 1752 0.10 MB/sec execute 86 sec latency 24056.037 ms 1 1752 0.10 MB/sec execute 87 sec latency 25056.846 ms 1 1752 0.10 MB/sec execute 88 sec latency 26057.494 ms 1 1752 0.10 MB/sec execute 89 sec latency 27062.840 ms 1 1752 0.10 MB/sec execute 90 sec latency 28064.808 ms 1 1752 0.10 MB/sec execute 91 sec latency 29065.031 ms 1 1752 0.10 MB/sec execute 92 sec latency 30065.263 ms oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all 1 1752 0.09 MB/sec execute 93 sec latency 31065.938 ms 1 1752 0.09 MB/sec execute 94 sec latency 32067.574 ms pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 1 1769 0.09 MB/sec execute 95 sec latency 32670.835 ms Started lustre-MDT0001 12:26:40 (1788625600) targets are mounted 12:26:40 (1788625600) facet_failover done 1 1805 0.09 MB/sec execute 96 sec latency 129.991 ms 1 1840 0.09 MB/sec execute 97 sec latency 122.159 ms 1 1871 0.09 MB/sec execute 98 sec latency 112.942 ms 1 1927 0.09 MB/sec execute 99 sec latency 119.044 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 263 208.170 38979.650 Close 224 160.344 32670.813 Rename 11 115.007 164.911 Unlink 27 30.132 75.699 Qpathinfo 229 16.679 96.835 Qfileinfo 60 2.873 17.503 Qfsinfo 33 4.332 34.028 Sfileinfo 34 63.238 120.551 Find 76 14.025 68.193 WriteX 198 13.116 87.488 ReadX 301 0.255 37.351 Flush 30 59.264 122.775 Throughput 0.0906066 MB/sec 1 clients 1 procs max_latency=38979.694 ms stopping dbench on /mnt/lustre at Sat Sep 5 12:26:48 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1268996 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2592 3582736 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 20044 3586732 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 22636 7169468 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Sep 5 12:26:57 EDT 2026 waiting for dbench pid 67837 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 10 0.00 MB/sec warmup 1 sec latency 84.265 ms 1 71 1.57 MB/sec warmup 2 sec latency 132.509 ms 1 113 1.75 MB/sec warmup 3 sec latency 183.469 ms 1 148 1.57 MB/sec warmup 4 sec latency 214.814 ms 1 155 1.31 MB/sec warmup 5 sec latency 963.006 ms 1 174 1.18 MB/sec warmup 6 sec latency 1390.528 ms 1 199 1.12 MB/sec warmup 7 sec latency 363.456 ms 1 239 1.14 MB/sec warmup 8 sec latency 138.317 ms test_26 fail mds1 3 times 1 283 1.17 MB/sec warmup 9 sec latency 111.034 ms 1 310 1.13 MB/sec warmup 10 sec latency 166.372 ms Failing mds1 on oleg317-server 1 339 1.11 MB/sec warmup 11 sec latency 164.599 ms Stopping /mnt/lustre-mds1 (opts:) on oleg317-server 1 365 1.08 MB/sec warmup 12 sec latency 140.178 ms 1 404 1.10 MB/sec warmup 13 sec latency 139.112 ms 1 404 1.02 MB/sec warmup 14 sec latency 1020.878 ms 1 404 0.95 MB/sec warmup 15 sec latency 2021.341 ms 12:27:12 (1788625632) shut down facet: mds1 facet_host: oleg317-server facet_failover_host: oleg317-server 1 404 0.90 MB/sec warmup 16 sec latency 3022.552 ms 1 404 0.84 MB/sec warmup 17 sec latency 4022.789 ms 1 404 0.80 MB/sec warmup 18 sec latency 5023.175 ms 1 404 0.75 MB/sec warmup 19 sec latency 6024.047 ms 1 404 0.00 MB/sec execute 1 sec latency 8028.181 ms 1 404 0.00 MB/sec execute 2 sec latency 9028.449 ms 1 404 0.00 MB/sec execute 3 sec latency 10028.973 ms 1 404 0.00 MB/sec execute 4 sec latency 11029.301 ms 1 404 0.00 MB/sec execute 5 sec latency 12029.913 ms Failover mds1 to oleg317-server mount facets: mds1 1 404 0.00 MB/sec execute 6 sec latency 13031.679 ms 1 404 0.00 MB/sec execute 7 sec latency 14032.070 ms 1 404 0.00 MB/sec execute 8 sec latency 15032.923 ms 1 404 0.00 MB/sec execute 9 sec latency 16033.575 ms 1 404 0.00 MB/sec execute 10 sec latency 17034.227 ms 1 404 0.00 MB/sec execute 11 sec latency 18038.014 ms 1 404 0.00 MB/sec execute 12 sec latency 19038.629 ms 1 404 0.00 MB/sec execute 13 sec latency 20038.909 ms 1 404 0.00 MB/sec execute 14 sec latency 21039.143 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 404 0.00 MB/sec execute 15 sec latency 22039.376 ms 1 404 0.00 MB/sec execute 16 sec latency 23040.544 ms 1 404 0.00 MB/sec execute 17 sec latency 24041.319 ms 1 404 0.00 MB/sec execute 18 sec latency 25041.573 ms 1 404 0.00 MB/sec execute 19 sec latency 26041.929 ms 1 404 0.00 MB/sec execute 20 sec latency 27042.186 ms 1 404 0.00 MB/sec execute 21 sec latency 28042.443 ms 1 404 0.00 MB/sec execute 22 sec latency 29042.684 ms 1 404 0.00 MB/sec execute 23 sec latency 30043.142 ms 1 404 0.00 MB/sec execute 24 sec latency 31044.646 ms 1 404 0.00 MB/sec execute 25 sec latency 32046.456 ms oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all 1 404 0.00 MB/sec execute 26 sec latency 33046.891 ms 1 404 0.00 MB/sec execute 27 sec latency 34047.150 ms pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 1 426 0.02 MB/sec execute 28 sec latency 34378.569 ms Started lustre-MDT0000 12:27:45 (1788625665) targets are mounted 12:27:45 (1788625665) facet_failover done 1 461 0.06 MB/sec execute 29 sec latency 153.677 ms 1 494 0.10 MB/sec execute 30 sec latency 121.088 ms 1 525 0.12 MB/sec execute 31 sec latency 140.794 ms 1 608 0.21 MB/sec execute 32 sec latency 97.493 ms 1 654 0.20 MB/sec execute 33 sec latency 120.288 ms 1 689 0.20 MB/sec execute 34 sec latency 119.710 ms 1 712 0.20 MB/sec execute 35 sec latency 136.563 ms oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 740 0.19 MB/sec execute 36 sec latency 131.124 ms 1 769 0.19 MB/sec execute 37 sec latency 165.156 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 799 0.19 MB/sec execute 38 sec latency 123.112 ms 1 825 0.18 MB/sec execute 39 sec latency 142.044 ms 1 878 0.18 MB/sec execute 40 sec latency 105.572 ms 1 884 0.18 MB/sec execute 41 sec latency 891.643 ms 1 903 0.17 MB/sec execute 42 sec latency 1144.814 ms 1 948 0.17 MB/sec execute 43 sec latency 83.079 ms 1 977 0.17 MB/sec execute 44 sec latency 121.211 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2864 1268440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2624 1268680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37160 3567636 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 27184 3575020 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 64344 7142656 1% /mnt/lustre 1 1009 0.18 MB/sec execute 45 sec latency 117.456 ms 1 1074 0.19 MB/sec execute 46 sec latency 95.945 ms 1 1118 0.18 MB/sec execute 47 sec latency 157.002 ms 1 1154 0.18 MB/sec execute 48 sec latency 106.035 ms 1 1192 0.18 MB/sec execute 49 sec latency 106.147 ms 1 1231 0.17 MB/sec execute 50 sec latency 99.992 ms 1 1263 0.17 MB/sec execute 51 sec latency 150.533 ms 1 1392 0.22 MB/sec execute 52 sec latency 86.144 ms 1 1426 0.22 MB/sec execute 53 sec latency 91.701 ms 1 1467 0.22 MB/sec execute 54 sec latency 294.714 ms test_26 fail mds2 4 times 1 1504 0.21 MB/sec execute 55 sec latency 308.807 ms 1 1543 0.21 MB/sec execute 56 sec latency 114.496 ms Failing mds2 on oleg317-server 1 1585 0.21 MB/sec execute 57 sec latency 110.991 ms Stopping /mnt/lustre-mds2 (opts:) on oleg317-server 1 1624 0.20 MB/sec execute 58 sec latency 99.532 ms 1 1676 0.20 MB/sec execute 59 sec latency 115.727 ms 1 1715 0.20 MB/sec execute 60 sec latency 156.911 ms 1 1806 0.20 MB/sec execute 61 sec latency 55.418 ms 1 1858 0.19 MB/sec execute 62 sec latency 94.486 ms 1 1941 0.19 MB/sec execute 63 sec latency 67.575 ms 1 1996 0.19 MB/sec execute 64 sec latency 101.673 ms 1 2033 0.19 MB/sec execute 65 sec latency 117.261 ms 1 2095 0.19 MB/sec execute 66 sec latency 90.606 ms 1 2154 0.18 MB/sec execute 67 sec latency 78.209 ms 12:28:24 (1788625704) shut down facet: mds2 facet_host: oleg317-server facet_failover_host: oleg317-server 1 2196 0.18 MB/sec execute 68 sec latency 103.018 ms 1 2259 0.18 MB/sec execute 69 sec latency 411.506 ms 1 2259 0.18 MB/sec execute 70 sec latency 1411.817 ms 1 2259 0.17 MB/sec execute 71 sec latency 2412.069 ms 1 2259 0.17 MB/sec execute 72 sec latency 3412.328 ms 1 2259 0.17 MB/sec execute 73 sec latency 4413.639 ms 1 2259 0.17 MB/sec execute 74 sec latency 5414.864 ms 1 2259 0.17 MB/sec execute 75 sec latency 6417.091 ms 1 2259 0.16 MB/sec execute 76 sec latency 7420.232 ms 1 2259 0.16 MB/sec execute 77 sec latency 8422.596 ms Failover mds2 to oleg317-server mount facets: mds2 1 2259 0.16 MB/sec execute 78 sec latency 9422.876 ms 1 2259 0.16 MB/sec execute 79 sec latency 10423.082 ms 1 2259 0.16 MB/sec execute 80 sec latency 11423.280 ms 1 2259 0.15 MB/sec execute 81 sec latency 12425.531 ms 1 2259 0.15 MB/sec execute 82 sec latency 13425.889 ms 1 2259 0.15 MB/sec execute 83 sec latency 14426.832 ms 1 2259 0.15 MB/sec execute 84 sec latency 15426.968 ms 1 2259 0.15 MB/sec execute 85 sec latency 16427.133 ms 1 2259 0.14 MB/sec execute 86 sec latency 17427.724 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.14 MB/sec execute 87 sec latency 18428.026 ms 1 2259 0.14 MB/sec execute 88 sec latency 19428.242 ms 1 2259 0.14 MB/sec execute 89 sec latency 20431.488 ms 1 2259 0.14 MB/sec execute 90 sec latency 21433.833 ms 1 2259 0.14 MB/sec execute 91 sec latency 22434.932 ms 1 2259 0.14 MB/sec execute 92 sec latency 23436.496 ms oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all 1 2259 0.13 MB/sec execute 93 sec latency 24437.041 ms pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 1 2259 0.13 MB/sec execute 94 sec latency 25437.493 ms Started lustre-MDT0001 12:28:52 (1788625732) targets are mounted 12:28:52 (1788625732) facet_failover done 1 2259 0.13 MB/sec execute 95 sec latency 26437.865 ms 1 2298 0.14 MB/sec execute 96 sec latency 26795.144 ms 1 2345 0.14 MB/sec execute 97 sec latency 176.131 ms 1 2373 0.14 MB/sec execute 98 sec latency 208.458 ms 1 2401 0.14 MB/sec execute 99 sec latency 124.387 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 cleanup 104 sec 1 cleanup 105 sec 0 cleanup 106 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 348 135.953 26795.118 Close 304 126.483 34378.547 Rename 13 97.496 156.880 Unlink 34 26.753 73.545 Qpathinfo 318 16.682 99.516 Qfileinfo 88 2.433 19.027 Qfsinfo 39 4.947 22.804 Sfileinfo 44 72.000 142.022 Find 107 13.742 81.279 WriteX 311 14.393 111.296 ReadX 381 2.508 89.147 Flush 33 57.792 208.445 Throughput 0.137453 MB/sec 1 clients 1 procs max_latency=34378.569 ms stopping dbench on /mnt/lustre at Sat Sep 5 12:29:03 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Sep 5 12:29:13 EDT 2026 waiting for dbench pid 69962 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients tar: Unexpected EOF in archive tar: Unexpected EOF in archive 0 10 0.00 MB/sec warmup 1 sec latency 48.730 ms 1 89 2.06 MB/sec warmup 2 sec latency 114.603 ms 1 137 2.00 MB/sec warmup 3 sec latency 152.346 ms 1 166 1.70 MB/sec warmup 4 sec latency 217.055 ms 1 190 1.51 MB/sec warmup 5 sec latency 501.931 ms 1 226 1.43 MB/sec warmup 6 sec latency 164.692 ms 1 264 1.41 MB/sec warmup 7 sec latency 126.806 ms 1 305 1.40 MB/sec warmup 8 sec latency 125.665 ms 1 339 1.36 MB/sec warmup 9 sec latency 147.018 ms 1 382 1.35 MB/sec warmup 10 sec latency 94.390 ms 1 420 1.35 MB/sec warmup 11 sec latency 143.205 ms 1 456 1.32 MB/sec warmup 12 sec latency 145.215 ms 1 486 1.30 MB/sec warmup 13 sec latency 174.250 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sat Sep 5 12:29:26 EDT 2026 with return code 0 69962 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 69962 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished