== replay-dual test 26: dbench and tar with mds failover ========================================================== 00:57:38 (1781240258) Starting client oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre Started clients oleg250-client.virtnet: 192.168.202.150@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 64949 Started dbench loop with 64951 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1944 1269360 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1756 1269548 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13832 3584864 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 15396 7190320 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jun 12 00:57:45 EDT 2026 waiting for dbench pid 65028 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 10 0.00 MB/sec warmup 1 sec latency 68.341 ms 1 57 1.15 MB/sec warmup 2 sec latency 151.221 ms 1 100 1.53 MB/sec warmup 3 sec latency 142.050 ms 1 126 1.41 MB/sec warmup 4 sec latency 263.123 ms 1 155 1.31 MB/sec warmup 5 sec latency 156.196 ms 1 157 1.09 MB/sec warmup 6 sec latency 762.851 ms 1 173 1.01 MB/sec warmup 7 sec latency 1099.577 ms 1 184 0.92 MB/sec warmup 8 sec latency 541.673 ms 1 219 0.94 MB/sec warmup 9 sec latency 573.276 ms test_26 fail mds1 1 times 1 259 0.97 MB/sec warmup 10 sec latency 160.729 ms 1 289 0.97 MB/sec warmup 11 sec latency 184.602 ms 1 313 0.96 MB/sec warmup 12 sec latency 200.627 ms Failing mds1 on oleg250-server 1 339 0.94 MB/sec warmup 13 sec latency 214.874 ms Stopping /mnt/lustre-mds1 (opts:) on oleg250-server 1 365 0.93 MB/sec warmup 14 sec latency 186.654 ms 1 388 0.92 MB/sec warmup 15 sec latency 309.310 ms 1 388 0.86 MB/sec warmup 16 sec latency 1310.149 ms 1 388 0.81 MB/sec warmup 17 sec latency 2310.415 ms 00:58:02 (1781240282) shut down facet: mds1 facet_host: oleg250-server facet_failover_host: oleg250-server 1 388 0.77 MB/sec warmup 18 sec latency 3311.137 ms 1 388 0.73 MB/sec warmup 19 sec latency 4316.452 ms 1 388 0.00 MB/sec execute 1 sec latency 6317.136 ms 1 388 0.00 MB/sec execute 2 sec latency 7317.523 ms 1 388 0.00 MB/sec execute 3 sec latency 8317.816 ms 1 388 0.00 MB/sec execute 4 sec latency 9318.981 ms 1 388 0.00 MB/sec execute 5 sec latency 10319.175 ms 1 388 0.00 MB/sec execute 6 sec latency 11319.459 ms 1 388 0.00 MB/sec execute 7 sec latency 12319.960 ms Failover mds1 to oleg250-server mount facets: mds1 1 388 0.00 MB/sec execute 8 sec latency 13320.363 ms 1 388 0.00 MB/sec execute 9 sec latency 14320.890 ms 1 388 0.00 MB/sec execute 10 sec latency 15324.373 ms 1 388 0.00 MB/sec execute 11 sec latency 16325.331 ms 1 388 0.00 MB/sec execute 12 sec latency 17325.940 ms 1 388 0.00 MB/sec execute 13 sec latency 18326.263 ms 1 388 0.00 MB/sec execute 14 sec latency 19326.887 ms 1 388 0.00 MB/sec execute 15 sec latency 20328.769 ms 1 388 0.00 MB/sec execute 16 sec latency 21329.641 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 388 0.00 MB/sec execute 17 sec latency 22330.215 ms 1 388 0.00 MB/sec execute 18 sec latency 23330.694 ms 1 388 0.00 MB/sec execute 19 sec latency 24330.904 ms 1 388 0.00 MB/sec execute 20 sec latency 25331.105 ms 1 388 0.00 MB/sec execute 21 sec latency 26331.520 ms 1 388 0.00 MB/sec execute 22 sec latency 27333.422 ms 1 388 0.00 MB/sec execute 23 sec latency 28333.624 ms 1 388 0.00 MB/sec execute 24 sec latency 29337.250 ms 1 388 0.00 MB/sec execute 25 sec latency 30339.750 ms 1 388 0.00 MB/sec execute 26 sec latency 31340.011 ms 1 388 0.00 MB/sec execute 27 sec latency 32342.063 ms 1 388 0.00 MB/sec execute 28 sec latency 33348.211 ms 1 388 0.00 MB/sec execute 29 sec latency 34348.647 ms 1 388 0.00 MB/sec execute 30 sec latency 35348.852 ms oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all 1 388 0.00 MB/sec execute 31 sec latency 36349.076 ms pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 1 388 0.00 MB/sec execute 32 sec latency 37349.798 ms Started lustre-MDT0000 00:58:37 (1781240317) targets are mounted 00:58:37 (1781240317) facet_failover done 1 388 0.00 MB/sec execute 33 sec latency 38350.024 ms 1 397 0.01 MB/sec execute 34 sec latency 39000.943 ms 1 430 0.04 MB/sec execute 35 sec latency 150.612 ms 1 477 0.08 MB/sec execute 36 sec latency 104.849 ms oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 514 0.11 MB/sec execute 37 sec latency 144.209 ms 1 552 0.14 MB/sec execute 38 sec latency 514.516 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 636 0.19 MB/sec execute 39 sec latency 544.989 ms 1 673 0.18 MB/sec execute 40 sec latency 122.470 ms 1 717 0.18 MB/sec execute 41 sec latency 126.277 ms 1 747 0.18 MB/sec execute 42 sec latency 123.143 ms 1 803 0.18 MB/sec execute 43 sec latency 97.885 ms 1 854 0.18 MB/sec execute 44 sec latency 93.481 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2052 1269252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8456 3591332 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 35372 3566624 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43828 7157956 1% /mnt/lustre 1 888 0.17 MB/sec execute 45 sec latency 115.354 ms 1 919 0.17 MB/sec execute 46 sec latency 103.822 ms 1 967 0.17 MB/sec execute 47 sec latency 124.794 ms 1 993 0.17 MB/sec execute 48 sec latency 125.949 ms 1 1051 0.18 MB/sec execute 49 sec latency 82.307 ms 1 1105 0.18 MB/sec execute 50 sec latency 84.083 ms 1 1154 0.18 MB/sec execute 51 sec latency 89.349 ms 1 1172 0.18 MB/sec execute 52 sec latency 583.605 ms 1 1192 0.17 MB/sec execute 53 sec latency 1084.138 ms 1 1242 0.17 MB/sec execute 54 sec latency 67.590 ms test_26 fail mds2 2 times 1 1318 0.19 MB/sec execute 55 sec latency 187.625 ms Failing mds2 on oleg250-server 1 1418 0.22 MB/sec execute 56 sec latency 93.178 ms Stopping /mnt/lustre-mds2 (opts:) on oleg250-server 1 1468 0.21 MB/sec execute 57 sec latency 82.908 ms 1 1520 0.21 MB/sec execute 58 sec latency 105.499 ms 1 1613 0.21 MB/sec execute 59 sec latency 51.829 ms 1 1691 0.21 MB/sec execute 60 sec latency 66.721 ms 1 1777 0.20 MB/sec execute 61 sec latency 69.263 ms 1 1861 0.20 MB/sec execute 62 sec latency 58.120 ms 1 1971 0.20 MB/sec execute 63 sec latency 66.943 ms 1 2018 0.20 MB/sec execute 64 sec latency 103.182 ms 1 2088 0.20 MB/sec execute 65 sec latency 89.367 ms 00:59:10 (1781240350) shut down facet: mds2 facet_host: oleg250-server facet_failover_host: oleg250-server 1 2160 0.19 MB/sec execute 66 sec latency 88.019 ms 1 2225 0.19 MB/sec execute 67 sec latency 77.745 ms 1 2259 0.19 MB/sec execute 68 sec latency 731.729 ms 1 2259 0.19 MB/sec execute 69 sec latency 1734.707 ms 1 2259 0.18 MB/sec execute 70 sec latency 2734.897 ms 1 2259 0.18 MB/sec execute 71 sec latency 3736.405 ms 1 2259 0.18 MB/sec execute 72 sec latency 4736.616 ms 1 2259 0.18 MB/sec execute 73 sec latency 5736.950 ms 1 2259 0.17 MB/sec execute 74 sec latency 6737.185 ms 1 2259 0.17 MB/sec execute 75 sec latency 7738.760 ms Failover mds2 to oleg250-server mount facets: mds2 1 2259 0.17 MB/sec execute 76 sec latency 8744.697 ms 1 2259 0.17 MB/sec execute 77 sec latency 9744.874 ms 1 2259 0.17 MB/sec execute 78 sec latency 10745.641 ms 1 2259 0.16 MB/sec execute 79 sec latency 11749.643 ms 1 2259 0.16 MB/sec execute 80 sec latency 12754.210 ms 1 2259 0.16 MB/sec execute 81 sec latency 13754.585 ms 1 2259 0.16 MB/sec execute 82 sec latency 14766.312 ms 1 2259 0.16 MB/sec execute 83 sec latency 15766.510 ms 1 2259 0.15 MB/sec execute 84 sec latency 16771.335 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.15 MB/sec execute 85 sec latency 17771.829 ms 1 2259 0.15 MB/sec execute 86 sec latency 18772.474 ms 1 2259 0.15 MB/sec execute 87 sec latency 19777.425 ms 1 2259 0.15 MB/sec execute 88 sec latency 20778.163 ms 1 2259 0.15 MB/sec execute 89 sec latency 21778.362 ms 1 2259 0.14 MB/sec execute 90 sec latency 22778.562 ms oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all 1 2259 0.14 MB/sec execute 91 sec latency 23778.954 ms 1 2259 0.14 MB/sec execute 92 sec latency 24779.891 ms pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 1 2259 0.14 MB/sec execute 93 sec latency 25780.649 ms Started lustre-MDT0001 00:59:38 (1781240378) targets are mounted 00:59:38 (1781240378) facet_failover done 1 2259 0.14 MB/sec execute 94 sec latency 26780.942 ms 1 2327 0.15 MB/sec execute 95 sec latency 26789.693 ms 1 2357 0.15 MB/sec execute 96 sec latency 107.800 ms 1 2406 0.15 MB/sec execute 97 sec latency 133.304 ms oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 2434 0.15 MB/sec execute 98 sec latency 209.531 ms 1 2473 0.15 MB/sec execute 99 sec latency 79.320 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 365 119.331 26789.673 Close 316 133.426 39000.909 Rename 15 94.594 209.511 Unlink 38 21.434 48.914 Qpathinfo 339 13.992 83.293 Qfileinfo 94 2.381 19.476 Qfsinfo 43 3.854 43.181 Sfileinfo 48 61.968 123.127 Find 115 9.810 47.075 WriteX 338 12.631 67.900 ReadX 427 1.571 65.467 Flush 34 51.296 122.444 Throughput 0.149226 MB/sec 1 clients 1 procs max_latency=39000.943 ms stopping dbench on /mnt/lustre at Fri Jun 12 00:59:48 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2324 1268980 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2128 1269176 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19968 3585644 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37616 3567952 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 57584 7153596 1% /mnt/lustre striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jun 12 00:59:56 EDT 2026 waiting for dbench pid 67185 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 3 0.00 MB/sec warmup 1 sec latency 236.533 ms 1 54 1.05 MB/sec warmup 2 sec latency 98.166 ms 1 111 1.75 MB/sec warmup 3 sec latency 106.121 ms test_26 fail mds1 3 times 1 147 1.57 MB/sec warmup 4 sec latency 146.623 ms 1 177 1.42 MB/sec warmup 5 sec latency 146.730 ms Failing mds1 on oleg250-server 1 199 1.31 MB/sec warmup 6 sec latency 391.535 ms Stopping /mnt/lustre-mds1 (opts:) on oleg250-server 1 247 1.34 MB/sec warmup 7 sec latency 118.137 ms 1 289 1.34 MB/sec warmup 8 sec latency 112.487 ms 1 339 1.36 MB/sec warmup 9 sec latency 242.776 ms 1 339 1.23 MB/sec warmup 10 sec latency 1243.632 ms 1 339 1.11 MB/sec warmup 11 sec latency 2244.613 ms 1 339 1.02 MB/sec warmup 12 sec latency 3244.854 ms 1 339 0.94 MB/sec warmup 13 sec latency 4245.987 ms 1 339 0.88 MB/sec warmup 14 sec latency 5246.262 ms 1 339 0.82 MB/sec warmup 15 sec latency 6246.440 ms 01:00:11 (1781240411) shut down facet: mds1 facet_host: oleg250-server facet_failover_host: oleg250-server 1 339 0.77 MB/sec warmup 16 sec latency 7246.692 ms 1 339 0.72 MB/sec warmup 17 sec latency 8246.916 ms 1 339 0.68 MB/sec warmup 18 sec latency 9247.603 ms 1 339 0.65 MB/sec warmup 19 sec latency 10247.770 ms 1 339 0.00 MB/sec execute 1 sec latency 12248.862 ms 1 339 0.00 MB/sec execute 2 sec latency 13249.105 ms 1 339 0.00 MB/sec execute 3 sec latency 14249.752 ms 1 339 0.00 MB/sec execute 4 sec latency 15249.963 ms 1 339 0.00 MB/sec execute 5 sec latency 16250.296 ms Failover mds1 to oleg250-server mount facets: mds1 1 339 0.00 MB/sec execute 6 sec latency 17250.812 ms 1 339 0.00 MB/sec execute 7 sec latency 18251.074 ms 1 339 0.00 MB/sec execute 8 sec latency 19252.239 ms 1 339 0.00 MB/sec execute 9 sec latency 20252.409 ms 1 339 0.00 MB/sec execute 10 sec latency 21262.608 ms 1 339 0.00 MB/sec execute 11 sec latency 22262.965 ms 1 339 0.00 MB/sec execute 12 sec latency 23263.420 ms 1 339 0.00 MB/sec execute 13 sec latency 24265.073 ms 1 339 0.00 MB/sec execute 14 sec latency 25265.665 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 339 0.00 MB/sec execute 15 sec latency 26266.012 ms 1 339 0.00 MB/sec execute 16 sec latency 27283.330 ms 1 339 0.00 MB/sec execute 17 sec latency 28284.433 ms 1 339 0.00 MB/sec execute 18 sec latency 29284.674 ms 1 339 0.00 MB/sec execute 19 sec latency 30284.936 ms 1 339 0.00 MB/sec execute 20 sec latency 31285.226 ms 1 339 0.00 MB/sec execute 21 sec latency 32285.432 ms 1 339 0.00 MB/sec execute 22 sec latency 33286.880 ms 1 339 0.00 MB/sec execute 23 sec latency 34288.723 ms 1 339 0.00 MB/sec execute 24 sec latency 35288.909 ms 1 339 0.00 MB/sec execute 25 sec latency 36289.101 ms 1 339 0.00 MB/sec execute 26 sec latency 37289.430 ms 1 339 0.00 MB/sec execute 27 sec latency 38289.759 ms oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all 1 339 0.00 MB/sec execute 28 sec latency 39289.973 ms pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 1 339 0.00 MB/sec execute 29 sec latency 40290.740 ms Started lustre-MDT0000 01:00:46 (1781240446) targets are mounted 01:00:46 (1781240446) facet_failover done 1 339 0.00 MB/sec execute 30 sec latency 41291.382 ms 1 357 0.02 MB/sec execute 31 sec latency 41923.171 ms 1 393 0.05 MB/sec execute 32 sec latency 152.306 ms 1 432 0.09 MB/sec execute 33 sec latency 135.385 ms 1 465 0.12 MB/sec execute 34 sec latency 171.532 ms oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 499 0.15 MB/sec execute 35 sec latency 142.842 ms 1 521 0.16 MB/sec execute 36 sec latency 202.626 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 595 0.23 MB/sec execute 37 sec latency 131.632 ms 1 639 0.23 MB/sec execute 38 sec latency 205.394 ms 1 670 0.23 MB/sec execute 39 sec latency 152.541 ms 1 711 0.22 MB/sec execute 40 sec latency 135.382 ms 1 744 0.22 MB/sec execute 41 sec latency 109.735 ms 1 783 0.22 MB/sec execute 42 sec latency 127.154 ms 1 848 0.22 MB/sec execute 43 sec latency 122.371 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2640 1268664 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2396 1268908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 29480 3573820 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 32496 3568364 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 61976 7142184 1% /mnt/lustre 1 876 0.21 MB/sec execute 44 sec latency 176.187 ms 1 884 0.21 MB/sec execute 45 sec latency 637.097 ms 1 885 0.20 MB/sec execute 46 sec latency 1493.580 ms 1 917 0.20 MB/sec execute 47 sec latency 153.365 ms 1 949 0.20 MB/sec execute 48 sec latency 125.819 ms 1 981 0.20 MB/sec execute 49 sec latency 139.016 ms 1 995 0.19 MB/sec execute 50 sec latency 555.775 ms 1 1062 0.21 MB/sec execute 51 sec latency 648.388 ms 1 1128 0.21 MB/sec execute 52 sec latency 72.020 ms 1 1180 0.20 MB/sec execute 53 sec latency 99.897 ms test_26 fail mds2 4 times 1 1226 0.20 MB/sec execute 54 sec latency 98.149 ms Failing mds2 on oleg250-server 1 1277 0.20 MB/sec execute 55 sec latency 118.407 ms Stopping /mnt/lustre-mds2 (opts:) on oleg250-server 1 1426 0.25 MB/sec execute 56 sec latency 74.858 ms 1 1433 0.24 MB/sec execute 57 sec latency 949.835 ms 1 1433 0.24 MB/sec execute 58 sec latency 1951.269 ms 1 1433 0.23 MB/sec execute 59 sec latency 2951.655 ms 1 1433 0.23 MB/sec execute 60 sec latency 3951.922 ms 1 1433 0.23 MB/sec execute 61 sec latency 4952.152 ms 1 1433 0.22 MB/sec execute 62 sec latency 5953.238 ms 1 1433 0.22 MB/sec execute 63 sec latency 6961.308 ms 01:01:20 (1781240480) shut down 1 1433 0.21 MB/sec execute 64 sec latency 7961.608 ms facet: mds2 facet_host: oleg250-server facet_failover_host: oleg250-server 1 1433 0.21 MB/sec execute 65 sec latency 8966.320 ms 1 1433 0.21 MB/sec execute 66 sec latency 9967.312 ms 1 1433 0.20 MB/sec execute 67 sec latency 10969.347 ms 1 1433 0.20 MB/sec execute 68 sec latency 11969.534 ms 1 1433 0.20 MB/sec execute 69 sec latency 12970.935 ms 1 1433 0.20 MB/sec execute 70 sec latency 13972.126 ms 1 1433 0.19 MB/sec execute 71 sec latency 14972.303 ms 1 1433 0.19 MB/sec execute 72 sec latency 15972.525 ms 1 1433 0.19 MB/sec execute 73 sec latency 16973.548 ms 1 1433 0.19 MB/sec execute 74 sec latency 17975.965 ms Failover mds2 to oleg250-server mount facets: mds2 1 1433 0.18 MB/sec execute 75 sec latency 18976.194 ms 1 1433 0.18 MB/sec execute 76 sec latency 19976.401 ms 1 1433 0.18 MB/sec execute 77 sec latency 20977.118 ms 1 1433 0.18 MB/sec execute 78 sec latency 21977.313 ms 1 1433 0.17 MB/sec execute 79 sec latency 22977.510 ms 1 1433 0.17 MB/sec execute 80 sec latency 23978.903 ms 1 1433 0.17 MB/sec execute 81 sec latency 24979.711 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1433 0.17 MB/sec execute 82 sec latency 25980.514 ms 1 1433 0.17 MB/sec execute 83 sec latency 26980.755 ms 1 1433 0.16 MB/sec execute 84 sec latency 27982.103 ms 1 1433 0.16 MB/sec execute 85 sec latency 28982.705 ms 1 1433 0.16 MB/sec execute 86 sec latency 29983.042 ms oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all 1 1433 0.16 MB/sec execute 87 sec latency 30985.335 ms pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 1 1433 0.16 MB/sec execute 88 sec latency 31985.606 ms Started lustre-MDT0001 01:01:45 (1781240505) targets are mounted 01:01:45 (1781240505) facet_failover done 1 1433 0.15 MB/sec execute 89 sec latency 32985.811 ms 1 1433 0.15 MB/sec execute 90 sec latency 33987.235 ms 1 1467 0.15 MB/sec execute 91 sec latency 34088.664 ms 1 1493 0.15 MB/sec execute 92 sec latency 454.040 ms 1 1534 0.15 MB/sec execute 93 sec latency 99.396 ms oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1591 0.15 MB/sec execute 94 sec latency 87.299 ms 1 1643 0.15 MB/sec execute 95 sec latency 105.891 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1686 0.15 MB/sec execute 96 sec latency 90.920 ms 1 1738 0.14 MB/sec execute 97 sec latency 125.654 ms 1 1814 0.14 MB/sec execute 98 sec latency 80.892 ms 1 1861 0.14 MB/sec execute 99 sec latency 123.679 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 277 211.825 41922.940 Close 241 152.841 34088.643 Rename 11 86.367 138.983 Unlink 25 25.352 49.831 Qpathinfo 217 14.945 125.789 Qfileinfo 78 2.254 9.828 Qfsinfo 33 3.321 17.100 Sfileinfo 34 50.058 104.326 Find 75 12.369 98.559 WriteX 298 14.294 115.125 ReadX 297 1.968 95.709 Flush 30 43.301 87.262 Throughput 0.142153 MB/sec 1 clients 1 procs max_latency=41923.171 ms stopping dbench on /mnt/lustre at Fri Jun 12 01:01:59 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench tar: Unexpected EOF in archive tar: Unexpected EOF in archive looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jun 12 01:02:05 EDT 2026 waiting for dbench pid 69243 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 19 0.33 MB/sec warmup 1 sec latency 22.024 ms 1 92 2.10 MB/sec warmup 2 sec latency 76.894 ms 1 147 2.09 MB/sec warmup 3 sec latency 112.038 ms 1 184 1.83 MB/sec warmup 4 sec latency 197.257 ms 1 215 1.67 MB/sec warmup 5 sec latency 399.412 ms 1 263 1.65 MB/sec warmup 6 sec latency 98.892 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Jun 12 01:02:11 EDT 2026 with return code 0 69243 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 69243 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished