== replay-dual test 26: dbench and tar with mds failover ========================================================== 09:33:10 (1790775190) Starting client oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre Started clients oleg226-client.virtnet: 192.168.202.126@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65651 Started dbench loop with 65652 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1968 1269336 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1756 1269548 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17928 3584888 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19492 7190344 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Sep 30 09:33:17 EDT 2026 waiting for dbench pid 65731 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 0 8 0.00 MB/sec warmup 1 sec latency 120.684 ms 1 31 0.52 MB/sec warmup 2 sec latency 270.503 ms 1 83 1.24 MB/sec warmup 3 sec latency 271.283 ms 1 120 1.37 MB/sec warmup 4 sec latency 153.494 ms 1 147 1.25 MB/sec warmup 5 sec latency 179.764 ms 1 163 1.12 MB/sec warmup 6 sec latency 326.004 ms 1 184 1.05 MB/sec warmup 7 sec latency 273.698 ms 1 215 1.04 MB/sec warmup 8 sec latency 401.444 ms 1 263 1.10 MB/sec warmup 9 sec latency 97.686 ms 1 300 1.09 MB/sec warmup 10 sec latency 131.719 ms test_26 fail mds1 1 times 1 330 1.09 MB/sec warmup 11 sec latency 171.815 ms 1 357 1.06 MB/sec warmup 12 sec latency 225.875 ms Failing mds1 on oleg226-server 1 394 1.07 MB/sec warmup 13 sec latency 163.661 ms Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 1 429 1.08 MB/sec warmup 14 sec latency 130.784 ms 1 470 1.09 MB/sec warmup 15 sec latency 134.397 ms 1 501 1.09 MB/sec warmup 16 sec latency 480.767 ms 09:33:34 (1790775214) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server 1 501 1.02 MB/sec warmup 17 sec latency 1482.029 ms 1 501 0.97 MB/sec warmup 18 sec latency 2482.664 ms 1 501 0.92 MB/sec warmup 19 sec latency 3482.914 ms 1 501 0.00 MB/sec execute 1 sec latency 5483.424 ms 1 501 0.00 MB/sec execute 2 sec latency 6483.668 ms 1 501 0.00 MB/sec execute 3 sec latency 7484.370 ms 1 501 0.00 MB/sec execute 4 sec latency 8484.797 ms 1 501 0.00 MB/sec execute 5 sec latency 9485.043 ms 1 501 0.00 MB/sec execute 6 sec latency 10485.251 ms 1 501 0.00 MB/sec execute 7 sec latency 11493.836 ms Failover mds1 to oleg226-server mount facets: mds1 1 501 0.00 MB/sec execute 8 sec latency 12494.596 ms 1 501 0.00 MB/sec execute 9 sec latency 13494.984 ms 1 501 0.00 MB/sec execute 10 sec latency 14496.359 ms 1 501 0.00 MB/sec execute 11 sec latency 15496.615 ms 1 501 0.00 MB/sec execute 12 sec latency 16502.281 ms 1 501 0.00 MB/sec execute 13 sec latency 17504.025 ms 1 501 0.00 MB/sec execute 14 sec latency 18504.308 ms 1 501 0.00 MB/sec execute 15 sec latency 19504.506 ms 1 501 0.00 MB/sec execute 16 sec latency 20505.060 ms 1 501 0.00 MB/sec execute 17 sec latency 21505.341 ms 1 501 0.00 MB/sec execute 18 sec latency 22505.645 ms 1 501 0.00 MB/sec execute 19 sec latency 23506.393 ms 1 501 0.00 MB/sec execute 20 sec latency 24510.506 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 501 0.00 MB/sec execute 21 sec latency 25510.880 ms 1 501 0.00 MB/sec execute 22 sec latency 26511.324 ms 1 501 0.00 MB/sec execute 23 sec latency 27511.643 ms 1 501 0.00 MB/sec execute 24 sec latency 28514.258 ms 1 501 0.00 MB/sec execute 25 sec latency 29515.587 ms 1 501 0.00 MB/sec execute 26 sec latency 30520.515 ms 1 501 0.00 MB/sec execute 27 sec latency 31520.918 ms 1 501 0.00 MB/sec execute 28 sec latency 32521.283 ms 1 501 0.00 MB/sec execute 29 sec latency 33521.630 ms 1 501 0.00 MB/sec execute 30 sec latency 34522.471 ms 1 501 0.00 MB/sec execute 31 sec latency 35522.765 ms oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all 1 501 0.00 MB/sec execute 32 sec latency 36523.556 ms pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 1 503 0.00 MB/sec execute 33 sec latency 37404.795 ms Started lustre-MDT0000 09:34:11 (1790775251) targets are mounted 09:34:11 (1790775251) facet_failover done 1 525 0.02 MB/sec execute 34 sec latency 212.872 ms 1 552 0.05 MB/sec execute 35 sec latency 694.563 ms 1 616 0.10 MB/sec execute 36 sec latency 803.990 ms 1 664 0.10 MB/sec execute 37 sec latency 136.805 ms 1 695 0.10 MB/sec execute 38 sec latency 136.314 ms 1 718 0.10 MB/sec execute 39 sec latency 141.360 ms 1 747 0.10 MB/sec execute 40 sec latency 153.126 ms 1 780 0.09 MB/sec execute 41 sec latency 149.565 ms oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 808 0.10 MB/sec execute 42 sec latency 141.855 ms 1 857 0.10 MB/sec execute 43 sec latency 132.436 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 882 0.10 MB/sec execute 44 sec latency 212.242 ms 1 910 0.10 MB/sec execute 45 sec latency 133.854 ms 1 933 0.09 MB/sec execute 46 sec latency 113.002 ms 1 981 0.09 MB/sec execute 47 sec latency 86.926 ms 1 1033 0.11 MB/sec execute 48 sec latency 111.747 ms 1 1082 0.11 MB/sec execute 49 sec latency 115.602 ms 1 1136 0.11 MB/sec execute 50 sec latency 114.338 ms 1 1172 0.11 MB/sec execute 51 sec latency 199.548 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2188 1269116 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1932 1269372 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 10912 3593664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 44604 3560748 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 55516 7154412 1% /mnt/lustre 1 1173 0.11 MB/sec execute 52 sec latency 1181.270 ms 1 1211 0.11 MB/sec execute 53 sec latency 106.775 ms 1 1244 0.10 MB/sec execute 54 sec latency 100.247 ms 1 1378 0.15 MB/sec execute 55 sec latency 114.806 ms 1 1405 0.15 MB/sec execute 56 sec latency 315.131 ms 1 1405 0.15 MB/sec execute 57 sec latency 1315.253 ms 1 1445 0.15 MB/sec execute 58 sec latency 1501.577 ms 1 1493 0.15 MB/sec execute 59 sec latency 100.676 ms 1 1554 0.14 MB/sec execute 60 sec latency 70.207 ms test_26 fail mds2 2 times 1 1625 0.14 MB/sec execute 61 sec latency 64.604 ms Failing mds2 on oleg226-server 1 1662 0.14 MB/sec execute 62 sec latency 110.700 ms Stopping /mnt/lustre-mds2 (opts:) on oleg226-server 1 1698 0.14 MB/sec execute 63 sec latency 130.144 ms 1 1751 0.14 MB/sec execute 64 sec latency 343.849 ms 1 1751 0.14 MB/sec execute 65 sec latency 1344.061 ms 1 1751 0.13 MB/sec execute 66 sec latency 2348.853 ms 1 1751 0.13 MB/sec execute 67 sec latency 3349.092 ms 1 1751 0.13 MB/sec execute 68 sec latency 4350.127 ms 1 1751 0.13 MB/sec execute 69 sec latency 5350.803 ms 1 1751 0.13 MB/sec execute 70 sec latency 6351.222 ms 1 1751 0.12 MB/sec execute 71 sec latency 7351.827 ms 09:34:49 (1790775289) shut down facet: mds2 facet_host: oleg226-server facet_failover_host: oleg226-server 1 1751 0.12 MB/sec execute 72 sec latency 8352.056 ms 1 1751 0.12 MB/sec execute 73 sec latency 9358.503 ms 1 1751 0.12 MB/sec execute 74 sec latency 10358.825 ms 1 1751 0.12 MB/sec execute 75 sec latency 11360.303 ms 1 1751 0.12 MB/sec execute 76 sec latency 12360.917 ms 1 1751 0.11 MB/sec execute 77 sec latency 13362.004 ms 1 1751 0.11 MB/sec execute 78 sec latency 14362.548 ms 1 1751 0.11 MB/sec execute 79 sec latency 15362.875 ms 1 1751 0.11 MB/sec execute 80 sec latency 16363.097 ms 1 1751 0.11 MB/sec execute 81 sec latency 17363.399 ms Failover mds2 to oleg226-server 1 1751 0.11 MB/sec execute 82 sec latency 18363.538 ms mount facets: mds2 1 1751 0.11 MB/sec execute 83 sec latency 19363.747 ms 1 1751 0.10 MB/sec execute 84 sec latency 20364.136 ms 1 1751 0.10 MB/sec execute 85 sec latency 21364.675 ms 1 1751 0.10 MB/sec execute 86 sec latency 22366.999 ms 1 1751 0.10 MB/sec execute 87 sec latency 23367.234 ms 1 1751 0.10 MB/sec execute 88 sec latency 24368.306 ms 1 1751 0.10 MB/sec execute 89 sec latency 25372.215 ms 1 1751 0.10 MB/sec execute 90 sec latency 26372.580 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1751 0.10 MB/sec execute 91 sec latency 27394.401 ms 1 1751 0.10 MB/sec execute 92 sec latency 28394.639 ms 1 1751 0.09 MB/sec execute 93 sec latency 29394.837 ms 1 1751 0.09 MB/sec execute 94 sec latency 30395.367 ms 1 1751 0.09 MB/sec execute 95 sec latency 31397.676 ms 1 1751 0.09 MB/sec execute 96 sec latency 32397.901 ms 1 1751 0.09 MB/sec execute 97 sec latency 33398.080 ms oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all 1 1751 0.09 MB/sec execute 98 sec latency 34398.490 ms pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 1 1751 0.09 MB/sec execute 99 sec latency 35400.694 ms Started lustre-MDT0001 09:35:17 (1790775317) targets are mounted 09:35:17 (1790775317) facet_failover done 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 211 245.115 37404.771 Close 179 220.390 36860.533 Rename 11 95.637 136.151 Unlink 22 25.613 41.253 Qpathinfo 165 17.163 94.344 Qfileinfo 50 2.031 16.021 Qfsinfo 33 4.810 22.997 Sfileinfo 34 70.542 212.223 Find 63 13.672 52.715 WriteX 185 13.814 73.433 ReadX 271 0.128 1.917 Flush 27 53.378 114.105 Throughput 0.0888709 MB/sec 1 clients 1 procs max_latency=37404.795 ms stopping dbench on /mnt/lustre at Wed Sep 30 09:35:21 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Sep 30 09:35:28 EDT 2026 waiting for dbench pid 67808 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 25 0.72 MB/sec warmup 1 sec latency 8.666 ms 1 113 2.62 MB/sec warmup 2 sec latency 114.287 ms 1 155 2.18 MB/sec warmup 3 sec latency 134.111 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2372 1268932 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26488 3579056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 20220 3585996 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 46708 7165052 1% /mnt/lustre 1 184 1.85 MB/sec warmup 4 sec latency 157.325 ms 1 220 1.70 MB/sec warmup 5 sec latency 260.647 ms 1 269 1.69 MB/sec warmup 6 sec latency 97.942 ms 1 312 1.64 MB/sec warmup 7 sec latency 116.281 ms 1 341 1.53 MB/sec warmup 8 sec latency 187.319 ms 1 390 1.54 MB/sec warmup 9 sec latency 164.175 ms 1 437 1.54 MB/sec warmup 10 sec latency 131.123 ms 1 477 1.52 MB/sec warmup 11 sec latency 111.331 ms 1 514 1.48 MB/sec warmup 12 sec latency 112.146 ms test_26 fail mds1 3 times 1 603 1.61 MB/sec warmup 13 sec latency 100.250 ms 1 654 1.50 MB/sec warmup 14 sec latency 119.616 ms Failing mds1 on oleg226-server 1 700 1.41 MB/sec warmup 15 sec latency 124.019 ms Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 1 759 1.33 MB/sec warmup 16 sec latency 100.021 ms 1 799 1.26 MB/sec warmup 17 sec latency 119.026 ms 1 848 1.20 MB/sec warmup 18 sec latency 128.743 ms 09:35:47 (1790775347) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server 1 884 1.14 MB/sec warmup 19 sec latency 99.292 ms 1 940 0.11 MB/sec execute 1 sec latency 1142.069 ms 1 985 0.12 MB/sec execute 2 sec latency 220.576 ms 1 995 0.08 MB/sec execute 3 sec latency 841.851 ms 1 995 0.06 MB/sec execute 4 sec latency 1842.485 ms 1 995 0.05 MB/sec execute 5 sec latency 2843.865 ms 1 995 0.04 MB/sec execute 6 sec latency 3844.812 ms 1 995 0.03 MB/sec execute 7 sec latency 4851.375 ms 1 995 0.03 MB/sec execute 8 sec latency 5851.792 ms 1 995 0.03 MB/sec execute 9 sec latency 6851.942 ms Failover mds1 to oleg226-server mount facets: mds1 1 995 0.02 MB/sec execute 10 sec latency 7852.487 ms 1 995 0.02 MB/sec execute 11 sec latency 8861.323 ms 1 995 0.02 MB/sec execute 12 sec latency 9861.636 ms 1 995 0.02 MB/sec execute 13 sec latency 10861.883 ms 1 995 0.02 MB/sec execute 14 sec latency 11862.178 ms 1 995 0.02 MB/sec execute 15 sec latency 12863.414 ms 1 995 0.01 MB/sec execute 16 sec latency 13863.658 ms 1 995 0.01 MB/sec execute 17 sec latency 14865.275 ms 1 995 0.01 MB/sec execute 18 sec latency 15865.503 ms 1 995 0.01 MB/sec execute 19 sec latency 16865.736 ms 1 995 0.01 MB/sec execute 20 sec latency 17872.844 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.01 MB/sec execute 21 sec latency 18873.063 ms 1 995 0.01 MB/sec execute 22 sec latency 19873.951 ms 1 995 0.01 MB/sec execute 23 sec latency 20874.157 ms 1 995 0.01 MB/sec execute 24 sec latency 21875.774 ms 1 995 0.01 MB/sec execute 25 sec latency 22875.913 ms 1 995 0.01 MB/sec execute 26 sec latency 23878.521 ms 1 995 0.01 MB/sec execute 27 sec latency 24878.713 ms 1 995 0.01 MB/sec execute 28 sec latency 25879.669 ms 1 995 0.01 MB/sec execute 29 sec latency 26880.078 ms 1 995 0.01 MB/sec execute 30 sec latency 27880.723 ms oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all 1 995 0.01 MB/sec execute 31 sec latency 28880.880 ms pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 1 997 0.01 MB/sec execute 32 sec latency 29807.978 ms 1 1012 0.02 MB/sec execute 33 sec latency 238.707 ms Started lustre-MDT0000 09:36:22 (1790775382) targets are mounted 09:36:22 (1790775382) facet_failover done 1 1076 0.04 MB/sec execute 34 sec latency 77.874 ms 1 1127 0.04 MB/sec execute 35 sec latency 106.218 ms 1 1147 0.04 MB/sec execute 36 sec latency 131.443 ms 1 1192 0.04 MB/sec execute 37 sec latency 207.258 ms 1 1242 0.04 MB/sec execute 38 sec latency 133.614 ms 1 1316 0.06 MB/sec execute 39 sec latency 165.514 ms oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1417 0.11 MB/sec execute 40 sec latency 99.249 ms 1 1465 0.11 MB/sec execute 41 sec latency 72.176 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1499 0.10 MB/sec execute 42 sec latency 333.254 ms 1 1552 0.10 MB/sec execute 43 sec latency 138.980 ms 1 1609 0.10 MB/sec execute 44 sec latency 102.010 ms 1 1650 0.10 MB/sec execute 45 sec latency 155.012 ms 1 1689 0.10 MB/sec execute 46 sec latency 118.332 ms 1 1746 0.10 MB/sec execute 47 sec latency 109.163 ms 1 1805 0.10 MB/sec execute 48 sec latency 87.246 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1855 0.10 MB/sec execute 49 sec latency 88.192 ms 1 1915 0.10 MB/sec execute 50 sec latency 111.058 ms 1 1980 0.09 MB/sec execute 51 sec latency 81.608 ms 1 2016 0.09 MB/sec execute 52 sec latency 102.301 ms 1 2088 0.09 MB/sec execute 53 sec latency 84.361 ms 1 2141 0.09 MB/sec execute 54 sec latency 103.146 ms 1 2184 0.09 MB/sec execute 55 sec latency 155.939 ms 1 2289 0.11 MB/sec execute 56 sec latency 80.737 ms 1 2353 0.11 MB/sec execute 57 sec latency 130.629 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Sep 30 09:36:46 EDT 2026 with return code 0 67808 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 67808 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished