== replay-dual test 26: dbench and tar with mds failover ========================================================== 16:04:24 (1777579464) Starting client oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre Started clients oleg416-client.virtnet: 192.168.204.116@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 64952 Started dbench loop with 64953 striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1756 1269548 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5640 3588964 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 7204 7194420 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Apr 30 16:04:30 EDT 2026 waiting for dbench pid 65030 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 8 0.00 MB/sec warmup 1 sec latency 117.641 ms 1 60 1.24 MB/sec warmup 2 sec latency 99.766 ms 1 111 1.74 MB/sec warmup 3 sec latency 137.595 ms 1 138 1.50 MB/sec warmup 4 sec latency 209.161 ms 1 168 1.36 MB/sec warmup 5 sec latency 171.588 ms 1 190 1.26 MB/sec warmup 6 sec latency 401.270 ms 1 221 1.22 MB/sec warmup 7 sec latency 172.885 ms 1 263 1.24 MB/sec warmup 8 sec latency 112.886 ms 1 297 1.22 MB/sec warmup 9 sec latency 147.669 ms test_26 fail mds1 1 times 1 324 1.17 MB/sec warmup 10 sec latency 164.738 ms Failing mds1 on oleg416-server 1 357 1.16 MB/sec warmup 11 sec latency 206.994 ms 1 392 1.15 MB/sec warmup 12 sec latency 130.401 ms Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 427 1.16 MB/sec warmup 13 sec latency 131.695 ms 1 429 1.08 MB/sec warmup 14 sec latency 974.948 ms 1 429 1.01 MB/sec warmup 15 sec latency 1975.790 ms 16:04:46 (1777579486) shut down facet: mds1 facet_host: oleg416-server facet_failover_host: oleg416-server 1 429 0.94 MB/sec warmup 16 sec latency 2976.047 ms 1 429 0.89 MB/sec warmup 17 sec latency 3977.988 ms 1 429 0.84 MB/sec warmup 18 sec latency 4978.251 ms 1 429 0.80 MB/sec warmup 19 sec latency 5978.486 ms 1 429 0.00 MB/sec execute 1 sec latency 7979.359 ms 1 429 0.00 MB/sec execute 2 sec latency 8979.712 ms 1 429 0.00 MB/sec execute 3 sec latency 9980.032 ms 1 429 0.00 MB/sec execute 4 sec latency 10980.208 ms 1 429 0.00 MB/sec execute 5 sec latency 11980.482 ms 1 429 0.00 MB/sec execute 6 sec latency 12980.696 ms Failover mds1 to oleg416-server mount facets: mds1 1 429 0.00 MB/sec execute 7 sec latency 13980.872 ms 1 429 0.00 MB/sec execute 8 sec latency 14982.126 ms 1 429 0.00 MB/sec execute 9 sec latency 15998.923 ms 1 429 0.00 MB/sec execute 10 sec latency 16999.730 ms 1 429 0.00 MB/sec execute 11 sec latency 18000.506 ms 1 429 0.00 MB/sec execute 12 sec latency 19001.880 ms 1 429 0.00 MB/sec execute 13 sec latency 20002.140 ms 1 429 0.00 MB/sec execute 14 sec latency 21005.544 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 429 0.00 MB/sec execute 15 sec latency 22010.353 ms 1 429 0.00 MB/sec execute 16 sec latency 23011.063 ms 1 429 0.00 MB/sec execute 17 sec latency 24015.128 ms 1 429 0.00 MB/sec execute 18 sec latency 25015.404 ms 1 429 0.00 MB/sec execute 19 sec latency 26016.729 ms 1 429 0.00 MB/sec execute 20 sec latency 27017.352 ms 1 429 0.00 MB/sec execute 21 sec latency 28017.709 ms 1 429 0.00 MB/sec execute 22 sec latency 29020.301 ms 1 429 0.00 MB/sec execute 23 sec latency 30020.754 ms 1 429 0.00 MB/sec execute 24 sec latency 31021.874 ms 1 429 0.00 MB/sec execute 25 sec latency 32022.359 ms 1 429 0.00 MB/sec execute 26 sec latency 33022.958 ms 1 429 0.00 MB/sec execute 27 sec latency 34023.295 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 1 429 0.00 MB/sec execute 28 sec latency 35024.112 ms 1 429 0.00 MB/sec execute 29 sec latency 36024.380 ms pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 1 429 0.00 MB/sec execute 30 sec latency 37024.734 ms Started lustre-MDT0000 16:05:21 (1777579521) targets are mounted 16:05:21 (1777579521) facet_failover done 1 429 0.00 MB/sec execute 31 sec latency 38028.486 ms 1 453 0.02 MB/sec execute 32 sec latency 38310.427 ms 1 485 0.06 MB/sec execute 33 sec latency 162.446 ms 1 514 0.08 MB/sec execute 34 sec latency 153.418 ms 1 552 0.11 MB/sec execute 35 sec latency 204.320 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 592 0.15 MB/sec execute 36 sec latency 696.281 ms 1 640 0.16 MB/sec execute 37 sec latency 107.127 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 674 0.16 MB/sec execute 38 sec latency 123.893 ms 1 704 0.16 MB/sec execute 39 sec latency 149.770 ms 1 733 0.15 MB/sec execute 40 sec latency 142.221 ms 1 768 0.15 MB/sec execute 41 sec latency 129.024 ms 1 802 0.15 MB/sec execute 42 sec latency 100.217 ms 1 851 0.15 MB/sec execute 43 sec latency 97.336 ms 1 887 0.15 MB/sec execute 44 sec latency 132.209 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1269216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8532 3592076 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36344 3565508 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 44876 7157584 1% /mnt/lustre 1 909 0.15 MB/sec execute 45 sec latency 177.520 ms 1 936 0.14 MB/sec execute 46 sec latency 118.340 ms 1 967 0.14 MB/sec execute 47 sec latency 159.638 ms 1 988 0.14 MB/sec execute 48 sec latency 199.781 ms 1 1045 0.16 MB/sec execute 49 sec latency 138.554 ms 1 1056 0.16 MB/sec execute 50 sec latency 759.116 ms 1 1073 0.15 MB/sec execute 51 sec latency 1370.727 ms 1 1126 0.15 MB/sec execute 52 sec latency 77.005 ms 1 1157 0.15 MB/sec execute 53 sec latency 106.463 ms test_26 fail mds2 2 times 1 1172 0.15 MB/sec execute 54 sec latency 723.696 ms 1 1207 0.14 MB/sec execute 55 sec latency 870.054 ms Failing mds2 on oleg416-server 1 1245 0.14 MB/sec execute 56 sec latency 93.579 ms Stopping /mnt/lustre-mds2 (opts:) on oleg416-server 1 1379 0.19 MB/sec execute 57 sec latency 90.062 ms 1 1419 0.19 MB/sec execute 58 sec latency 104.765 ms 1 1433 0.18 MB/sec execute 59 sec latency 722.764 ms 1 1433 0.18 MB/sec execute 60 sec latency 1723.157 ms 1 1433 0.18 MB/sec execute 61 sec latency 2723.453 ms 1 1433 0.18 MB/sec execute 62 sec latency 3724.632 ms 1 1433 0.17 MB/sec execute 63 sec latency 4724.897 ms 1 1433 0.17 MB/sec execute 64 sec latency 5730.958 ms 1 1433 0.17 MB/sec execute 65 sec latency 6731.541 ms 16:05:56 (1777579556) shut down 1 1433 0.16 MB/sec execute 66 sec latency 7732.048 ms facet: mds2 facet_host: oleg416-server facet_failover_host: oleg416-server 1 1433 0.16 MB/sec execute 67 sec latency 8732.340 ms 1 1433 0.16 MB/sec execute 68 sec latency 9732.651 ms 1 1433 0.16 MB/sec execute 69 sec latency 10732.990 ms 1 1433 0.16 MB/sec execute 70 sec latency 11736.278 ms 1 1433 0.15 MB/sec execute 71 sec latency 12736.912 ms 1 1433 0.15 MB/sec execute 72 sec latency 13737.137 ms 1 1433 0.15 MB/sec execute 73 sec latency 14737.362 ms 1 1433 0.15 MB/sec execute 74 sec latency 15737.640 ms 1 1433 0.15 MB/sec execute 75 sec latency 16739.056 ms 1 1433 0.14 MB/sec execute 76 sec latency 17739.293 ms Failover mds2 to oleg416-server mount facets: mds2 1 1433 0.14 MB/sec execute 77 sec latency 18739.585 ms 1 1433 0.14 MB/sec execute 78 sec latency 19739.932 ms 1 1433 0.14 MB/sec execute 79 sec latency 20741.236 ms 1 1433 0.14 MB/sec execute 80 sec latency 21741.887 ms 1 1433 0.13 MB/sec execute 81 sec latency 22743.302 ms 1 1433 0.13 MB/sec execute 82 sec latency 23743.619 ms 1 1433 0.13 MB/sec execute 83 sec latency 24744.195 ms 1 1433 0.13 MB/sec execute 84 sec latency 25746.234 ms 1 1433 0.13 MB/sec execute 85 sec latency 26746.421 ms 1 1433 0.13 MB/sec execute 86 sec latency 27747.052 ms 1 1433 0.13 MB/sec execute 87 sec latency 28747.198 ms 1 1433 0.12 MB/sec execute 88 sec latency 29749.624 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1433 0.12 MB/sec execute 89 sec latency 30750.918 ms 1 1433 0.12 MB/sec execute 90 sec latency 31752.728 ms 1 1433 0.12 MB/sec execute 91 sec latency 32753.633 ms 1 1433 0.12 MB/sec execute 92 sec latency 33753.924 ms 1 1433 0.12 MB/sec execute 93 sec latency 34754.360 ms 1 1433 0.12 MB/sec execute 94 sec latency 35755.064 ms 1 1433 0.11 MB/sec execute 95 sec latency 36756.875 ms 1 1433 0.11 MB/sec execute 96 sec latency 37759.442 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 1 1433 0.11 MB/sec execute 97 sec latency 38760.324 ms 1 1433 0.11 MB/sec execute 98 sec latency 39760.570 ms pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 1 1433 0.11 MB/sec execute 99 sec latency 40761.156 ms 1 cleanup 100 sec Started lustre-MDT0001 16:06:31 (1777579591) targets are mounted 16:06:31 (1777579591) facet_failover done 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 139 349.322 38310.405 Close 131 342.202 41631.631 Rename 10 102.969 199.757 Unlink 19 28.992 55.851 Qpathinfo 105 24.467 105.860 Qfileinfo 46 2.973 13.663 Qfsinfo 33 3.958 20.261 Sfileinfo 34 64.011 121.072 Find 48 13.140 66.538 WriteX 203 15.067 58.846 ReadX 225 2.808 59.418 Flush 24 53.959 132.194 Throughput 0.109878 MB/sec 1 clients 1 procs max_latency=40761.156 ms stopping dbench on /mnt/lustre at Thu Apr 30 16:06:34 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Apr 30 16:06:44 EDT 2026 waiting for dbench pid 67123 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 8 0.00 MB/sec warmup 1 sec latency 121.638 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1268920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1269184 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 16096 3581480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26372 3579172 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42468 7160652 1% /mnt/lustre 1 51 1.04 MB/sec warmup 2 sec latency 141.302 ms 1 98 1.48 MB/sec warmup 3 sec latency 134.277 ms 1 137 1.50 MB/sec warmup 4 sec latency 225.418 ms 1 165 1.36 MB/sec warmup 5 sec latency 157.319 ms 1 197 1.30 MB/sec warmup 6 sec latency 263.567 ms 1 224 1.23 MB/sec warmup 7 sec latency 203.157 ms 1 275 1.28 MB/sec warmup 8 sec latency 138.427 ms 1 312 1.27 MB/sec warmup 9 sec latency 122.707 ms 1 346 1.24 MB/sec warmup 10 sec latency 157.358 ms test_26 fail mds1 3 times 1 392 1.25 MB/sec warmup 11 sec latency 107.011 ms 1 422 1.24 MB/sec warmup 12 sec latency 195.195 ms 1 452 1.22 MB/sec warmup 13 sec latency 163.524 ms Failing mds1 on oleg416-server 1 485 1.21 MB/sec warmup 14 sec latency 150.087 ms Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 513 1.18 MB/sec warmup 15 sec latency 170.385 ms 1 581 1.23 MB/sec warmup 16 sec latency 170.600 ms 1 606 1.23 MB/sec warmup 17 sec latency 624.693 ms 1 606 1.16 MB/sec warmup 18 sec latency 1624.906 ms 1 606 1.10 MB/sec warmup 19 sec latency 2625.332 ms 1 606 0.00 MB/sec execute 1 sec latency 4627.081 ms 1 606 0.00 MB/sec execute 2 sec latency 5630.599 ms 1 606 0.00 MB/sec execute 3 sec latency 6630.782 ms 16:07:08 (1777579628) shut down 1 606 0.00 MB/sec execute 4 sec latency 7638.751 ms facet: mds1 facet_host: oleg416-server facet_failover_host: oleg416-server 1 606 0.00 MB/sec execute 5 sec latency 8638.976 ms 1 606 0.00 MB/sec execute 6 sec latency 9639.312 ms 1 606 0.00 MB/sec execute 7 sec latency 10639.971 ms 1 606 0.00 MB/sec execute 8 sec latency 11640.204 ms 1 606 0.00 MB/sec execute 9 sec latency 12644.514 ms 1 606 0.00 MB/sec execute 10 sec latency 13650.807 ms 1 606 0.00 MB/sec execute 11 sec latency 14651.888 ms 1 606 0.00 MB/sec execute 12 sec latency 15652.213 ms 1 606 0.00 MB/sec execute 13 sec latency 16654.451 ms 1 606 0.00 MB/sec execute 14 sec latency 17661.692 ms Failover mds1 to oleg416-server mount facets: mds1 1 606 0.00 MB/sec execute 15 sec latency 18661.976 ms 1 606 0.00 MB/sec execute 16 sec latency 19671.752 ms 1 606 0.00 MB/sec execute 17 sec latency 20672.160 ms 1 606 0.00 MB/sec execute 18 sec latency 21676.539 ms 1 606 0.00 MB/sec execute 19 sec latency 22676.923 ms 1 606 0.00 MB/sec execute 20 sec latency 23678.013 ms 1 606 0.00 MB/sec execute 21 sec latency 24678.433 ms 1 606 0.00 MB/sec execute 22 sec latency 25678.867 ms 1 606 0.00 MB/sec execute 23 sec latency 26679.369 ms 1 606 0.00 MB/sec execute 24 sec latency 27679.661 ms 1 606 0.00 MB/sec execute 25 sec latency 28694.792 ms 1 606 0.00 MB/sec execute 26 sec latency 29697.380 ms 1 606 0.00 MB/sec execute 27 sec latency 30698.214 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 606 0.00 MB/sec execute 28 sec latency 31698.479 ms 1 606 0.00 MB/sec execute 29 sec latency 32699.686 ms 1 606 0.00 MB/sec execute 30 sec latency 33699.868 ms 1 606 0.00 MB/sec execute 31 sec latency 34700.021 ms 1 606 0.00 MB/sec execute 32 sec latency 35700.211 ms 1 606 0.00 MB/sec execute 33 sec latency 36700.439 ms 1 606 0.00 MB/sec execute 34 sec latency 37700.711 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 1 606 0.00 MB/sec execute 35 sec latency 38701.524 ms pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 1 606 0.00 MB/sec execute 36 sec latency 39701.883 ms Started lustre-MDT0000 16:07:41 (1777579661) targets are mounted 16:07:41 (1777579661) facet_failover done 1 606 0.00 MB/sec execute 37 sec latency 40706.796 ms 1 606 0.00 MB/sec execute 38 sec latency 41709.635 ms 1 612 0.00 MB/sec execute 39 sec latency 42531.411 ms 1 664 0.00 MB/sec execute 40 sec latency 108.968 ms 1 697 0.01 MB/sec execute 41 sec latency 142.180 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 731 0.01 MB/sec execute 42 sec latency 132.460 ms 1 762 0.01 MB/sec execute 43 sec latency 153.134 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 786 0.01 MB/sec execute 44 sec latency 132.799 ms 1 808 0.01 MB/sec execute 45 sec latency 131.831 ms 1 857 0.01 MB/sec execute 46 sec latency 109.812 ms 1 884 0.01 MB/sec execute 47 sec latency 124.878 ms 1 888 0.01 MB/sec execute 48 sec latency 732.439 ms 1 921 0.02 MB/sec execute 49 sec latency 145.833 ms 1 962 0.02 MB/sec execute 50 sec latency 99.332 ms 1 986 0.02 MB/sec execute 51 sec latency 246.970 ms 1 1058 0.04 MB/sec execute 52 sec latency 87.802 ms 1 1114 0.04 MB/sec execute 53 sec latency 92.912 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1145 0.04 MB/sec execute 54 sec latency 120.462 ms 1 1190 0.04 MB/sec execute 55 sec latency 94.142 ms 1 1225 0.04 MB/sec execute 56 sec latency 127.141 ms 1 1259 0.04 MB/sec execute 57 sec latency 98.336 ms 1 1278 0.04 MB/sec execute 58 sec latency 165.300 ms 1 1396 0.09 MB/sec execute 59 sec latency 196.705 ms 1 1426 0.08 MB/sec execute 60 sec latency 346.909 ms 1 1455 0.08 MB/sec execute 61 sec latency 151.434 ms 1 1467 0.08 MB/sec execute 62 sec latency 567.914 ms 1 1492 0.08 MB/sec execute 63 sec latency 713.233 ms 1 1512 0.08 MB/sec execute 64 sec latency 174.965 ms 1 1554 0.08 MB/sec execute 65 sec latency 141.800 ms 1 1582 0.08 MB/sec execute 66 sec latency 84.859 ms 1 1619 0.08 MB/sec execute 67 sec latency 154.546 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Apr 30 16:08:12 EDT 2026 with return code 0 67123 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 67123 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished