== replay-dual test 26: dbench and tar with mds failover ========================================================== 20:25:25 (1786753525) Starting client oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre Started clients oleg625-client.virtnet: 192.168.206.125@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65627 Started dbench loop with 65628 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1956 1269348 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1760 1269544 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3584780 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3108 7190256 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Aug 14 20:25:31 EDT 2026 waiting for dbench pid 65706 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 14 0.00 MB/sec warmup 1 sec latency 4.285 ms 1 68 1.51 MB/sec warmup 2 sec latency 178.017 ms 1 111 1.75 MB/sec warmup 3 sec latency 170.634 ms 1 140 1.51 MB/sec warmup 4 sec latency 193.552 ms 1 173 1.41 MB/sec warmup 5 sec latency 150.044 ms 1 184 1.23 MB/sec warmup 6 sec latency 530.449 ms 1 215 1.19 MB/sec warmup 7 sec latency 658.035 ms 1 252 1.19 MB/sec warmup 8 sec latency 137.703 ms test_26 fail mds1 1 times 1 297 1.22 MB/sec warmup 9 sec latency 89.118 ms 1 331 1.20 MB/sec warmup 10 sec latency 130.784 ms Failing mds1 on oleg625-server 1 361 1.17 MB/sec warmup 11 sec latency 162.466 ms 1 397 1.17 MB/sec warmup 12 sec latency 134.904 ms Stopping /mnt/lustre-mds1 (opts:) on oleg625-server 1 437 1.18 MB/sec warmup 13 sec latency 124.648 ms 1 444 1.12 MB/sec warmup 14 sec latency 842.607 ms 1 444 1.04 MB/sec warmup 15 sec latency 1844.620 ms 20:25:47 (1786753547) shut down facet: mds1 facet_host: oleg625-server facet_failover_host: oleg625-server 1 444 0.98 MB/sec warmup 16 sec latency 2845.596 ms 1 444 0.92 MB/sec warmup 17 sec latency 3846.062 ms 1 444 0.87 MB/sec warmup 18 sec latency 4847.899 ms 1 444 0.82 MB/sec warmup 19 sec latency 5848.259 ms 1 444 0.00 MB/sec execute 1 sec latency 7848.711 ms 1 444 0.00 MB/sec execute 2 sec latency 8849.904 ms 1 444 0.00 MB/sec execute 3 sec latency 9850.345 ms 1 444 0.00 MB/sec execute 4 sec latency 10850.974 ms 1 444 0.00 MB/sec execute 5 sec latency 11851.256 ms Failover mds1 to oleg625-server mount facets: mds1 1 444 0.00 MB/sec execute 6 sec latency 12852.861 ms 1 444 0.00 MB/sec execute 7 sec latency 13853.179 ms 1 444 0.00 MB/sec execute 8 sec latency 14853.351 ms 1 444 0.00 MB/sec execute 9 sec latency 15854.477 ms 1 444 0.00 MB/sec execute 10 sec latency 16858.893 ms 1 444 0.00 MB/sec execute 11 sec latency 17860.423 ms 1 444 0.00 MB/sec execute 12 sec latency 18860.953 ms 1 444 0.00 MB/sec execute 13 sec latency 19861.148 ms 1 444 0.00 MB/sec execute 14 sec latency 20864.579 ms 1 444 0.00 MB/sec execute 15 sec latency 21864.789 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 444 0.00 MB/sec execute 16 sec latency 22864.978 ms 1 444 0.00 MB/sec execute 17 sec latency 23865.256 ms 1 444 0.00 MB/sec execute 18 sec latency 24866.635 ms 1 444 0.00 MB/sec execute 19 sec latency 25867.042 ms 1 444 0.00 MB/sec execute 20 sec latency 26868.009 ms 1 444 0.00 MB/sec execute 21 sec latency 27868.226 ms 1 444 0.00 MB/sec execute 22 sec latency 28868.425 ms 1 444 0.00 MB/sec execute 23 sec latency 29871.682 ms 1 444 0.00 MB/sec execute 24 sec latency 30871.863 ms 1 444 0.00 MB/sec execute 25 sec latency 31872.940 ms 1 444 0.00 MB/sec execute 26 sec latency 32874.341 ms 1 444 0.00 MB/sec execute 27 sec latency 33874.855 ms 1 444 0.00 MB/sec execute 28 sec latency 34875.103 ms oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all 1 444 0.00 MB/sec execute 29 sec latency 35875.313 ms pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 1 444 0.00 MB/sec execute 30 sec latency 36875.593 ms Started lustre-MDT0000 20:26:22 (1786753582) targets are mounted 20:26:22 (1786753582) facet_failover done 1 444 0.00 MB/sec execute 31 sec latency 37878.102 ms 1 480 0.03 MB/sec execute 32 sec latency 38134.426 ms 1 512 0.06 MB/sec execute 33 sec latency 144.776 ms 1 552 0.10 MB/sec execute 34 sec latency 167.291 ms 1 598 0.15 MB/sec execute 35 sec latency 620.073 ms 1 642 0.15 MB/sec execute 36 sec latency 115.666 ms 1 675 0.15 MB/sec execute 37 sec latency 142.170 ms 1 709 0.15 MB/sec execute 38 sec latency 116.165 ms oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 738 0.15 MB/sec execute 39 sec latency 110.431 ms 1 767 0.14 MB/sec execute 40 sec latency 150.205 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 799 0.14 MB/sec execute 41 sec latency 163.297 ms 1 825 0.14 MB/sec execute 42 sec latency 125.519 ms 1 873 0.14 MB/sec execute 43 sec latency 195.515 ms 1 900 0.14 MB/sec execute 44 sec latency 131.729 ms 1 919 0.14 MB/sec execute 45 sec latency 173.181 ms 1 952 0.13 MB/sec execute 46 sec latency 136.307 ms 1 980 0.13 MB/sec execute 47 sec latency 181.264 ms 1 1031 0.15 MB/sec execute 48 sec latency 129.896 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2144 1269160 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1944 1269360 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 44184 3560892 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9304 3593272 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 53488 7154164 1% /mnt/lustre 1 1064 0.15 MB/sec execute 49 sec latency 105.420 ms 1 1118 0.15 MB/sec execute 50 sec latency 102.870 ms 1 1162 0.14 MB/sec execute 51 sec latency 102.751 ms 1 1172 0.14 MB/sec execute 52 sec latency 804.696 ms 1 1196 0.14 MB/sec execute 53 sec latency 1108.078 ms 1 1235 0.14 MB/sec execute 54 sec latency 95.895 ms 1 1274 0.14 MB/sec execute 55 sec latency 124.877 ms 1 1399 0.18 MB/sec execute 56 sec latency 94.857 ms 1 1446 0.18 MB/sec execute 57 sec latency 80.291 ms test_26 fail mds2 2 times 1 1496 0.18 MB/sec execute 58 sec latency 116.164 ms 1 1538 0.18 MB/sec execute 59 sec latency 105.058 ms Failing mds2 on oleg625-server 1 1570 0.17 MB/sec execute 60 sec latency 108.133 ms Stopping /mnt/lustre-mds2 (opts:) on oleg625-server 1 1616 0.17 MB/sec execute 61 sec latency 92.102 ms 1 1656 0.17 MB/sec execute 62 sec latency 90.730 ms 1 1719 0.17 MB/sec execute 63 sec latency 131.377 ms 20:26:55 (1786753615) shut down 1 1787 0.17 MB/sec execute 64 sec latency 90.701 ms facet: mds2 facet_host: oleg625-server facet_failover_host: oleg625-server 1 1861 0.16 MB/sec execute 65 sec latency 68.384 ms 1 1945 0.16 MB/sec execute 66 sec latency 134.440 ms 1 2014 0.16 MB/sec execute 67 sec latency 98.985 ms 1 2094 0.16 MB/sec execute 68 sec latency 93.301 ms 1 2164 0.16 MB/sec execute 69 sec latency 68.240 ms 1 2246 0.16 MB/sec execute 70 sec latency 65.441 ms 1 2259 0.16 MB/sec execute 71 sec latency 840.954 ms 1 2259 0.15 MB/sec execute 72 sec latency 1841.178 ms 1 2259 0.15 MB/sec execute 73 sec latency 2841.373 ms 1 2259 0.15 MB/sec execute 74 sec latency 3841.718 ms Failover mds2 to oleg625-server mount facets: mds2 1 2259 0.15 MB/sec execute 75 sec latency 4841.956 ms 1 2259 0.15 MB/sec execute 76 sec latency 5842.137 ms 1 2259 0.14 MB/sec execute 77 sec latency 6842.373 ms 1 2259 0.14 MB/sec execute 78 sec latency 7843.075 ms 1 2259 0.14 MB/sec execute 79 sec latency 8843.320 ms 1 2259 0.14 MB/sec execute 80 sec latency 9843.695 ms 1 2259 0.14 MB/sec execute 81 sec latency 10844.266 ms 1 2259 0.14 MB/sec execute 82 sec latency 11846.689 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.13 MB/sec execute 83 sec latency 12846.963 ms 1 2259 0.13 MB/sec execute 84 sec latency 13849.141 ms 1 2259 0.13 MB/sec execute 85 sec latency 14850.729 ms 1 2259 0.13 MB/sec execute 86 sec latency 15851.160 ms 1 2259 0.13 MB/sec execute 87 sec latency 16851.961 ms 1 2259 0.13 MB/sec execute 88 sec latency 17852.509 ms oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all 1 2259 0.13 MB/sec execute 89 sec latency 18853.935 ms pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 1 2259 0.12 MB/sec execute 90 sec latency 19854.260 ms Started lustre-MDT0001 20:27:22 (1786753642) targets are mounted 20:27:22 (1786753642) facet_failover done 1 2259 0.12 MB/sec execute 91 sec latency 20857.686 ms 1 2298 0.13 MB/sec execute 92 sec latency 21423.669 ms 1 2348 0.13 MB/sec execute 93 sec latency 128.563 ms 1 2378 0.13 MB/sec execute 94 sec latency 120.157 ms 1 2408 0.13 MB/sec execute 95 sec latency 151.231 ms 1 2441 0.13 MB/sec execute 96 sec latency 143.060 ms 1 2487 0.13 MB/sec execute 97 sec latency 106.355 ms oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 2567 0.14 MB/sec execute 98 sec latency 148.818 ms 1 2602 0.13 MB/sec execute 99 sec latency 152.618 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 376 109.718 21423.592 Close 323 129.698 38134.393 Rename 17 107.429 181.242 Unlink 41 30.105 57.847 Qpathinfo 355 17.108 136.289 Qfileinfo 93 2.438 25.910 Qfsinfo 52 3.988 22.300 Sfileinfo 50 68.190 195.502 Find 126 13.008 70.533 WriteX 310 13.637 58.712 ReadX 540 2.399 115.651 Flush 34 55.863 129.880 Throughput 0.134819 MB/sec 1 clients 1 procs max_latency=38134.426 ms stopping dbench on /mnt/lustre at Fri Aug 14 20:27:35 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2324 1268980 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1269200 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21372 3585296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2536 3602848 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 23908 7188144 1% /mnt/lustre looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Aug 14 20:27:43 EDT 2026 waiting for dbench pid 67904 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 2 0.00 MB/sec warmup 1 sec latency 955.728 ms 0 5 0.00 MB/sec warmup 2 sec latency 62.562 ms 1 71 1.05 MB/sec warmup 3 sec latency 71.487 ms 1 125 1.39 MB/sec warmup 4 sec latency 177.369 ms test_26 fail mds1 3 times 1 157 1.30 MB/sec warmup 5 sec latency 152.902 ms 1 184 1.23 MB/sec warmup 6 sec latency 245.756 ms Failing mds1 on oleg625-server 1 215 1.19 MB/sec warmup 7 sec latency 411.213 ms Stopping /mnt/lustre-mds1 (opts:) on oleg625-server 1 255 1.21 MB/sec warmup 8 sec latency 121.263 ms 1 289 1.19 MB/sec warmup 9 sec latency 156.557 ms 1 331 1.20 MB/sec warmup 10 sec latency 129.956 ms 20:27:54 (1786753674) shut down 1 339 1.11 MB/sec warmup 11 sec latency 851.119 ms facet: mds1 facet_host: oleg625-server facet_failover_host: oleg625-server 1 339 1.02 MB/sec warmup 12 sec latency 1852.117 ms 1 339 0.94 MB/sec warmup 13 sec latency 2852.953 ms 1 339 0.87 MB/sec warmup 14 sec latency 3853.145 ms 1 339 0.82 MB/sec warmup 15 sec latency 4853.349 ms 1 339 0.76 MB/sec warmup 16 sec latency 5853.513 ms 1 339 0.72 MB/sec warmup 17 sec latency 6853.717 ms 1 339 0.68 MB/sec warmup 18 sec latency 7854.002 ms 1 339 0.64 MB/sec warmup 19 sec latency 8854.178 ms 1 339 0.00 MB/sec execute 1 sec latency 10855.259 ms Failover mds1 to oleg625-server mount facets: mds1 1 339 0.00 MB/sec execute 2 sec latency 11856.894 ms 1 339 0.00 MB/sec execute 3 sec latency 12857.634 ms 1 339 0.00 MB/sec execute 4 sec latency 13858.661 ms 1 339 0.00 MB/sec execute 5 sec latency 14859.687 ms 1 339 0.00 MB/sec execute 6 sec latency 15861.107 ms 1 339 0.00 MB/sec execute 7 sec latency 16861.487 ms 1 339 0.00 MB/sec execute 8 sec latency 17861.771 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 339 0.00 MB/sec execute 9 sec latency 18861.998 ms 1 339 0.00 MB/sec execute 10 sec latency 19862.196 ms 1 339 0.00 MB/sec execute 11 sec latency 20862.392 ms 1 339 0.00 MB/sec execute 12 sec latency 21862.576 ms 1 339 0.00 MB/sec execute 13 sec latency 22862.906 ms 1 339 0.00 MB/sec execute 14 sec latency 23869.188 ms 1 339 0.00 MB/sec execute 15 sec latency 24870.406 ms 1 339 0.00 MB/sec execute 16 sec latency 25872.237 ms 1 339 0.00 MB/sec execute 17 sec latency 26872.452 ms 1 339 0.00 MB/sec execute 18 sec latency 27872.681 ms 1 339 0.00 MB/sec execute 19 sec latency 28873.035 ms 1 339 0.00 MB/sec execute 20 sec latency 29873.545 ms 1 339 0.00 MB/sec execute 21 sec latency 30873.853 ms 1 339 0.00 MB/sec execute 22 sec latency 31874.453 ms oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all 1 339 0.00 MB/sec execute 23 sec latency 32874.624 ms pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 1 339 0.00 MB/sec execute 24 sec latency 33874.849 ms Started lustre-MDT0000 20:28:28 (1786753708) targets are mounted 20:28:28 (1786753708) facet_failover done 1 339 0.00 MB/sec execute 25 sec latency 34875.702 ms 1 339 0.00 MB/sec execute 26 sec latency 35878.950 ms 1 365 0.03 MB/sec execute 27 sec latency 36298.685 ms 1 397 0.06 MB/sec execute 28 sec latency 249.084 ms 1 452 0.13 MB/sec execute 29 sec latency 92.672 ms 1 501 0.17 MB/sec execute 30 sec latency 93.913 ms oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 595 0.27 MB/sec execute 31 sec latency 101.720 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 658 0.27 MB/sec execute 32 sec latency 93.135 ms 1 710 0.27 MB/sec execute 33 sec latency 72.909 ms 1 762 0.27 MB/sec execute 34 sec latency 97.477 ms 1 799 0.26 MB/sec execute 35 sec latency 99.792 ms 1 857 0.26 MB/sec execute 36 sec latency 104.269 ms 1 901 0.25 MB/sec execute 37 sec latency 91.713 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2852 1268452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2580 1268724 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 52556 3550516 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9416 3591948 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 61972 7142464 1% /mnt/lustre 1 926 0.25 MB/sec execute 38 sec latency 111.930 ms 1 971 0.24 MB/sec execute 39 sec latency 99.040 ms 1 1028 0.27 MB/sec execute 40 sec latency 85.976 ms 1 1035 0.26 MB/sec execute 41 sec latency 935.969 ms 1 1082 0.25 MB/sec execute 42 sec latency 995.971 ms 1 1142 0.25 MB/sec execute 43 sec latency 76.912 ms 1 1190 0.24 MB/sec execute 44 sec latency 120.220 ms 1 1242 0.24 MB/sec execute 45 sec latency 56.955 ms test_26 fail mds2 4 times 1 1384 0.30 MB/sec execute 46 sec latency 84.928 ms 1 1421 0.29 MB/sec execute 47 sec latency 145.358 ms Failing mds2 on oleg625-server 1 1481 0.29 MB/sec execute 48 sec latency 83.888 ms Stopping /mnt/lustre-mds2 (opts:) on oleg625-server 1 1519 0.28 MB/sec execute 49 sec latency 101.456 ms 1 1597 0.28 MB/sec execute 50 sec latency 71.523 ms 1 1653 0.27 MB/sec execute 51 sec latency 112.118 ms 1 1715 0.27 MB/sec execute 52 sec latency 86.811 ms 1 1799 0.27 MB/sec execute 53 sec latency 90.130 ms 1 1865 0.26 MB/sec execute 54 sec latency 89.980 ms 1 1954 0.26 MB/sec execute 55 sec latency 70.775 ms 1 2008 0.25 MB/sec execute 56 sec latency 66.562 ms 1 2088 0.25 MB/sec execute 57 sec latency 64.700 ms 20:29:01 (1786753741) shut down facet: mds2 facet_host: oleg625-server facet_failover_host: oleg625-server 1 2158 0.25 MB/sec execute 58 sec latency 75.622 ms 1 2230 0.25 MB/sec execute 59 sec latency 56.840 ms 1 2259 0.24 MB/sec execute 60 sec latency 805.903 ms 1 2259 0.24 MB/sec execute 61 sec latency 1806.103 ms 1 2259 0.23 MB/sec execute 62 sec latency 2806.396 ms 1 2259 0.23 MB/sec execute 63 sec latency 3807.582 ms 1 2259 0.23 MB/sec execute 64 sec latency 4809.556 ms 1 2259 0.22 MB/sec execute 65 sec latency 5809.943 ms 1 2259 0.22 MB/sec execute 66 sec latency 6810.211 ms 1 2259 0.22 MB/sec execute 67 sec latency 7810.602 ms Failover mds2 to oleg625-server mount facets: mds2 1 2259 0.21 MB/sec execute 68 sec latency 8810.861 ms 1 2259 0.21 MB/sec execute 69 sec latency 9811.465 ms 1 2259 0.21 MB/sec execute 70 sec latency 10812.070 ms 1 2259 0.20 MB/sec execute 71 sec latency 11813.053 ms 1 2259 0.20 MB/sec execute 72 sec latency 12814.063 ms 1 2259 0.20 MB/sec execute 73 sec latency 13814.288 ms 1 2259 0.20 MB/sec execute 74 sec latency 14815.875 ms 1 2259 0.19 MB/sec execute 75 sec latency 15816.402 ms 1 2259 0.19 MB/sec execute 76 sec latency 16816.695 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.19 MB/sec execute 77 sec latency 17817.458 ms 1 2259 0.19 MB/sec execute 78 sec latency 18817.758 ms 1 2259 0.18 MB/sec execute 79 sec latency 19818.071 ms 1 2259 0.18 MB/sec execute 80 sec latency 20818.846 ms 1 2259 0.18 MB/sec execute 81 sec latency 21821.238 ms 1 2259 0.18 MB/sec execute 82 sec latency 22821.729 ms oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all 1 2259 0.17 MB/sec execute 83 sec latency 23822.030 ms 1 2259 0.17 MB/sec execute 84 sec latency 24824.903 ms pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 1 2259 0.17 MB/sec execute 85 sec latency 25825.403 ms Started lustre-MDT0001 20:29:29 (1786753769) targets are mounted 20:29:29 (1786753769) facet_failover done 1 2291 0.18 MB/sec execute 86 sec latency 26387.264 ms 1 2350 0.18 MB/sec execute 87 sec latency 82.585 ms 1 2386 0.18 MB/sec execute 88 sec latency 113.045 ms 1 2417 0.18 MB/sec execute 89 sec latency 145.149 ms 1 2454 0.18 MB/sec execute 90 sec latency 97.148 ms 1 2497 0.18 MB/sec execute 91 sec latency 91.780 ms 1 2581 0.18 MB/sec execute 92 sec latency 97.536 ms oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 2617 0.18 MB/sec execute 93 sec latency 126.048 ms 1 2765 0.21 MB/sec execute 94 sec latency 94.372 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2891 0.21 MB/sec execute 95 sec latency 95.896 ms 1 2931 0.22 MB/sec execute 96 sec latency 74.639 ms 1 2989 0.22 MB/sec execute 97 sec latency 103.049 ms 1 3169 0.22 MB/sec execute 98 sec latency 96.591 ms 1 3209 0.22 MB/sec execute 99 sec latency 104.942 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 440 188.701 36298.671 Close 367 9.734 47.999 Rename 22 85.139 144.823 Unlink 49 25.941 63.246 Qpathinfo 410 15.374 103.019 Qfileinfo 125 2.087 13.213 Qfsinfo 64 3.611 25.817 Sfileinfo 50 54.791 113.029 Find 137 11.033 56.752 WriteX 428 12.328 71.146 ReadX 805 0.738 105.174 LockX 2 9.663 12.086 UnlockX 2 17.834 26.463 Flush 39 47.059 145.126 Throughput 0.222151 MB/sec 1 clients 1 procs max_latency=36298.685 ms stopping dbench on /mnt/lustre at Fri Aug 14 20:29:47 EDT 2026 with return code 0 tar: Unexpected EOF in archive tar: Unexpected EOF in archive clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Aug 14 20:29:56 EDT 2026 waiting for dbench pid 70005 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 10 0.00 MB/sec warmup 1 sec latency 116.055 ms 1 66 1.44 MB/sec warmup 2 sec latency 96.369 ms 1 113 1.75 MB/sec warmup 3 sec latency 136.205 ms 1 147 1.57 MB/sec warmup 4 sec latency 143.576 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Aug 14 20:30:01 EDT 2026 with return code 0 1 181 1.46 MB/sec warmup 5 sec latency 169.286 ms 70005 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 70005 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished