== replay-dual test 26: dbench and tar with mds failover ========================================================== 20:41:32 (1788309692) Starting client oleg607-client.virtnet: -o user_xattr,flock 192.168.206.107@tcp:/lustre /mnt/lustre Started clients oleg607-client.virtnet: 192.168.206.107@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65625 Started dbench loop with 65626 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2004 1269300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1840 1269464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3580652 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3108 7186128 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Sep 1 20:41:37 EDT 2026 waiting for dbench pid 65705 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 34 1.05 MB/sec warmup 1 sec latency 54.543 ms 1 111 2.62 MB/sec warmup 2 sec latency 90.369 ms 1 154 2.18 MB/sec warmup 3 sec latency 140.109 ms 1 184 1.85 MB/sec warmup 4 sec latency 247.595 ms 1 219 1.68 MB/sec warmup 5 sec latency 321.179 ms 1 271 1.70 MB/sec warmup 6 sec latency 107.934 ms 1 323 1.68 MB/sec warmup 7 sec latency 134.505 ms test_26 fail mds1 1 times 1 378 1.68 MB/sec warmup 8 sec latency 82.400 ms 1 420 1.65 MB/sec warmup 9 sec latency 151.228 ms Failing mds1 on oleg607-server 1 453 1.59 MB/sec warmup 10 sec latency 165.542 ms Stopping /mnt/lustre-mds1 (opts:) on oleg607-server 1 492 1.56 MB/sec warmup 11 sec latency 118.501 ms 1 522 1.50 MB/sec warmup 12 sec latency 178.255 ms 1 606 1.61 MB/sec warmup 13 sec latency 295.386 ms 20:41:50 (1788309710) shut down facet: mds1 facet_host: oleg607-server facet_failover_host: oleg607-server 1 606 1.50 MB/sec warmup 14 sec latency 1296.113 ms 1 606 1.40 MB/sec warmup 15 sec latency 2298.531 ms 1 606 1.31 MB/sec warmup 16 sec latency 3298.992 ms 1 606 1.23 MB/sec warmup 17 sec latency 4299.269 ms 1 606 1.16 MB/sec warmup 18 sec latency 5299.515 ms 1 606 1.10 MB/sec warmup 19 sec latency 6301.473 ms 1 606 0.00 MB/sec execute 1 sec latency 8302.289 ms 1 606 0.00 MB/sec execute 2 sec latency 9302.661 ms 1 606 0.00 MB/sec execute 3 sec latency 10303.471 ms Failover mds1 to oleg607-server 1 606 0.00 MB/sec execute 4 sec latency 11306.773 ms mount facets: mds1 1 606 0.00 MB/sec execute 5 sec latency 12307.677 ms 1 606 0.00 MB/sec execute 6 sec latency 13308.028 ms 1 606 0.00 MB/sec execute 7 sec latency 14308.250 ms 1 606 0.00 MB/sec execute 8 sec latency 15308.564 ms 1 606 0.00 MB/sec execute 9 sec latency 16308.834 ms 1 606 0.00 MB/sec execute 10 sec latency 17309.677 ms 1 606 0.00 MB/sec execute 11 sec latency 18309.901 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 606 0.00 MB/sec execute 12 sec latency 19311.112 ms 1 606 0.00 MB/sec execute 13 sec latency 20311.392 ms 1 606 0.00 MB/sec execute 14 sec latency 21311.574 ms 1 606 0.00 MB/sec execute 15 sec latency 22313.476 ms 1 606 0.00 MB/sec execute 16 sec latency 23313.732 ms 1 606 0.00 MB/sec execute 17 sec latency 24314.813 ms 1 606 0.00 MB/sec execute 18 sec latency 25315.021 ms 1 606 0.00 MB/sec execute 19 sec latency 26316.009 ms 1 606 0.00 MB/sec execute 20 sec latency 27317.075 ms 1 606 0.00 MB/sec execute 21 sec latency 28317.245 ms 1 606 0.00 MB/sec execute 22 sec latency 29318.395 ms 1 606 0.00 MB/sec execute 23 sec latency 30318.625 ms 1 606 0.00 MB/sec execute 24 sec latency 31321.546 ms oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all 1 606 0.00 MB/sec execute 25 sec latency 32321.899 ms pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 1 606 0.00 MB/sec execute 26 sec latency 33322.107 ms Started lustre-MDT0000 20:42:24 (1788309744) targets are mounted 20:42:24 (1788309744) facet_failover done 1 606 0.00 MB/sec execute 27 sec latency 34322.288 ms 1 606 0.00 MB/sec execute 28 sec latency 35322.879 ms 1 638 0.00 MB/sec execute 29 sec latency 35980.239 ms 1 692 0.01 MB/sec execute 30 sec latency 78.665 ms 1 720 0.01 MB/sec execute 31 sec latency 363.668 ms 1 740 0.01 MB/sec execute 32 sec latency 684.894 ms oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 783 0.01 MB/sec execute 33 sec latency 118.851 ms 1 818 0.02 MB/sec execute 34 sec latency 104.501 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 873 0.02 MB/sec execute 35 sec latency 104.695 ms 1 910 0.02 MB/sec execute 36 sec latency 118.209 ms 1 970 0.02 MB/sec execute 37 sec latency 76.840 ms 1 1031 0.05 MB/sec execute 38 sec latency 125.869 ms 1 1095 0.05 MB/sec execute 39 sec latency 70.143 ms 1 1155 0.05 MB/sec execute 40 sec latency 118.042 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1960 1269344 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 44604 3557600 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14260 3592004 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 58864 7149604 1% /mnt/lustre 1 1198 0.05 MB/sec execute 41 sec latency 74.457 ms 1 1242 0.05 MB/sec execute 42 sec latency 90.211 ms 1 1319 0.07 MB/sec execute 43 sec latency 139.546 ms 1 1398 0.11 MB/sec execute 44 sec latency 468.960 ms 1 1436 0.11 MB/sec execute 45 sec latency 102.406 ms 1 1481 0.11 MB/sec execute 46 sec latency 96.705 ms 1 1524 0.11 MB/sec execute 47 sec latency 103.383 ms 1 1580 0.11 MB/sec execute 48 sec latency 81.371 ms 1 1632 0.11 MB/sec execute 49 sec latency 114.478 ms test_26 fail mds2 2 times 1 1670 0.10 MB/sec execute 50 sec latency 98.438 ms 1 1721 0.10 MB/sec execute 51 sec latency 93.435 ms Failing mds2 on oleg607-server Stopping /mnt/lustre-mds2 (opts:) on oleg607-server 1 1793 0.10 MB/sec execute 52 sec latency 65.248 ms 1 1845 0.10 MB/sec execute 53 sec latency 89.246 ms 1 1911 0.10 MB/sec execute 54 sec latency 87.497 ms 20:42:52 (1788309772) shut down facet: mds2 facet_host: oleg607-server facet_failover_host: oleg607-server 1 1986 0.10 MB/sec execute 55 sec latency 102.647 ms 1 2053 0.10 MB/sec execute 56 sec latency 70.291 ms 1 2150 0.10 MB/sec execute 57 sec latency 72.743 ms 1 2193 0.10 MB/sec execute 58 sec latency 81.900 ms 1 2259 0.10 MB/sec execute 59 sec latency 317.791 ms 1 2259 0.10 MB/sec execute 60 sec latency 1318.036 ms 1 2259 0.10 MB/sec execute 61 sec latency 2319.231 ms 1 2259 0.09 MB/sec execute 62 sec latency 3319.537 ms 1 2259 0.09 MB/sec execute 63 sec latency 4319.782 ms 1 2259 0.09 MB/sec execute 64 sec latency 5320.059 ms 1 2259 0.09 MB/sec execute 65 sec latency 6320.968 ms Failover mds2 to oleg607-server mount facets: mds2 1 2259 0.09 MB/sec execute 66 sec latency 7321.840 ms 1 2259 0.09 MB/sec execute 67 sec latency 8322.215 ms 1 2259 0.09 MB/sec execute 68 sec latency 9322.435 ms 1 2259 0.08 MB/sec execute 69 sec latency 10322.903 ms 1 2259 0.08 MB/sec execute 70 sec latency 11323.100 ms 1 2259 0.08 MB/sec execute 71 sec latency 12325.004 ms 1 2259 0.08 MB/sec execute 72 sec latency 13325.298 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.08 MB/sec execute 73 sec latency 14325.497 ms 1 2259 0.08 MB/sec execute 74 sec latency 15325.869 ms 1 2259 0.08 MB/sec execute 75 sec latency 16326.919 ms 1 2259 0.08 MB/sec execute 76 sec latency 17327.169 ms 1 2259 0.08 MB/sec execute 77 sec latency 18327.373 ms oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all 1 2259 0.07 MB/sec execute 78 sec latency 19327.640 ms pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-MDT0001 20:43:16 (1788309796) targets are mounted 20:43:16 (1788309796) facet_failover done 1 2259 0.07 MB/sec execute 79 sec latency 20328.927 ms 1 2264 0.07 MB/sec execute 80 sec latency 21210.758 ms 1 2347 0.09 MB/sec execute 81 sec latency 105.793 ms 1 2391 0.09 MB/sec execute 82 sec latency 127.698 ms 1 2431 0.09 MB/sec execute 83 sec latency 103.118 ms 1 2478 0.09 MB/sec execute 84 sec latency 84.473 ms oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 2565 0.09 MB/sec execute 85 sec latency 75.857 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2605 0.09 MB/sec execute 86 sec latency 98.592 ms 1 2768 0.13 MB/sec execute 87 sec latency 76.741 ms 1 2916 0.14 MB/sec execute 88 sec latency 80.803 ms 1 2979 0.14 MB/sec execute 89 sec latency 122.042 ms 1 3082 0.14 MB/sec execute 90 sec latency 83.294 ms 1 3246 0.15 MB/sec execute 91 sec latency 91.397 ms 1 3341 0.16 MB/sec execute 92 sec latency 78.521 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2188 1269116 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 54292 3552172 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15464 3589316 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 69756 7141488 1% /mnt/lustre 1 3375 0.16 MB/sec execute 93 sec latency 116.158 ms 1 3504 0.18 MB/sec execute 94 sec latency 131.199 ms 1 3564 0.18 MB/sec execute 95 sec latency 67.975 ms 1 3644 0.18 MB/sec execute 96 sec latency 70.617 ms 1 3697 0.18 MB/sec execute 97 sec latency 65.272 ms 1 3779 0.19 MB/sec execute 98 sec latency 87.211 ms 1 3817 0.19 MB/sec execute 99 sec latency 116.571 ms 1 cleanup 100 sec test_26 fail mds1 3 times 1 cleanup 101 sec 1 cleanup 102 sec Failing mds1 on oleg607-server 1 cleanup 103 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 518 84.424 21210.745 Close 414 10.492 43.887 Rename 26 73.976 139.523 Unlink 74 21.313 46.239 Qpathinfo 495 87.953 35980.224 Qfileinfo 96 1.840 12.915 Qfsinfo 90 2.515 18.685 Sfileinfo 50 50.179 104.487 Find 192 9.487 31.033 WriteX 297 11.063 65.810 ReadX 953 0.972 73.693 LockX 2 3.641 3.654 UnlockX 2 6.574 7.353 Flush 43 48.175 174.297 Throughput 0.189658 MB/sec 1 clients 1 procs max_latency=35980.239 ms stopping dbench on /mnt/lustre at Tue Sep 1 20:43:40 EDT 2026 with return code 0 Stopping /mnt/lustre-mds1 (opts:) on oleg607-server clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre 20:43:49 (1788309829) shut down facet: mds1 facet_host: oleg607-server facet_failover_host: oleg607-server Failover mds1 to oleg607-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-MDT0000 20:44:24 (1788309864) targets are mounted 20:44:24 (1788309864) facet_failover done removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Sep 1 20:44:31 EDT 2026 waiting for dbench pid 68684 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 31.958 ms 1 90 2.10 MB/sec warmup 2 sec latency 134.731 ms 1 137 2.01 MB/sec warmup 3 sec latency 118.400 ms 1 181 1.83 MB/sec warmup 4 sec latency 158.033 ms 1 203 1.58 MB/sec warmup 5 sec latency 388.349 ms 1 247 1.56 MB/sec warmup 6 sec latency 131.114 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2884 1268420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2644 1268660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 48916 3555284 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 4184 3598780 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 53100 7154064 1% /mnt/lustre 1 287 1.53 MB/sec warmup 7 sec latency 121.047 ms 1 323 1.47 MB/sec warmup 8 sec latency 132.043 ms 1 361 1.43 MB/sec warmup 9 sec latency 149.769 ms 1 405 1.43 MB/sec warmup 10 sec latency 107.624 ms 1 445 1.42 MB/sec warmup 11 sec latency 108.168 ms 1 485 1.41 MB/sec warmup 12 sec latency 146.766 ms 1 532 1.42 MB/sec warmup 13 sec latency 124.915 ms 1 654 1.50 MB/sec warmup 14 sec latency 58.526 ms 1 708 1.41 MB/sec warmup 15 sec latency 102.645 ms test_26 fail mds2 4 times 1 744 1.33 MB/sec warmup 16 sec latency 86.802 ms Failing mds2 on oleg607-server 1 789 1.26 MB/sec warmup 17 sec latency 120.644 ms Stopping /mnt/lustre-mds2 (opts:) on oleg607-server 1 825 1.19 MB/sec warmup 18 sec latency 110.618 ms 1 884 1.14 MB/sec warmup 19 sec latency 111.968 ms 1 921 0.10 MB/sec execute 1 sec latency 969.901 ms 1 985 0.12 MB/sec execute 2 sec latency 64.853 ms 1 995 0.08 MB/sec execute 3 sec latency 795.854 ms 1 995 0.06 MB/sec execute 4 sec latency 1797.968 ms 1 995 0.05 MB/sec execute 5 sec latency 2798.741 ms 1 995 0.04 MB/sec execute 6 sec latency 3801.359 ms 20:44:58 (1788309898) shut down 1 995 0.03 MB/sec execute 7 sec latency 4801.474 ms facet: mds2 facet_host: oleg607-server facet_failover_host: oleg607-server 1 995 0.03 MB/sec execute 8 sec latency 5801.986 ms 1 995 0.03 MB/sec execute 9 sec latency 6802.154 ms 1 995 0.02 MB/sec execute 10 sec latency 7802.444 ms 1 995 0.02 MB/sec execute 11 sec latency 8803.157 ms 1 995 0.02 MB/sec execute 12 sec latency 9803.623 ms 1 995 0.02 MB/sec execute 13 sec latency 10803.897 ms 1 995 0.02 MB/sec execute 14 sec latency 11804.269 ms 1 995 0.02 MB/sec execute 15 sec latency 12804.508 ms 1 995 0.01 MB/sec execute 16 sec latency 13805.027 ms 1 995 0.01 MB/sec execute 17 sec latency 14808.694 ms Failover mds2 to oleg607-server mount facets: mds2 1 995 0.01 MB/sec execute 18 sec latency 15809.134 ms 1 995 0.01 MB/sec execute 19 sec latency 16809.632 ms 1 995 0.01 MB/sec execute 20 sec latency 17810.497 ms 1 995 0.01 MB/sec execute 21 sec latency 18810.633 ms 1 995 0.01 MB/sec execute 22 sec latency 19817.936 ms 1 995 0.01 MB/sec execute 23 sec latency 20820.440 ms 1 995 0.01 MB/sec execute 24 sec latency 21821.065 ms 1 995 0.01 MB/sec execute 25 sec latency 22821.259 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 995 0.01 MB/sec execute 26 sec latency 23821.742 ms 1 995 0.01 MB/sec execute 27 sec latency 24823.451 ms 1 995 0.01 MB/sec execute 28 sec latency 25823.784 ms 1 995 0.01 MB/sec execute 29 sec latency 26824.020 ms 1 995 0.01 MB/sec execute 30 sec latency 27824.860 ms 1 995 0.01 MB/sec execute 31 sec latency 28827.559 ms oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all 1 995 0.01 MB/sec execute 32 sec latency 29827.874 ms pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 1 995 0.01 MB/sec execute 33 sec latency 30828.113 ms Started lustre-MDT0001 20:45:25 (1788309925) targets are mounted 20:45:25 (1788309925) facet_failover done 1 995 0.01 MB/sec execute 34 sec latency 31828.529 ms 1 1018 0.02 MB/sec execute 35 sec latency 32047.038 ms 1 1080 0.03 MB/sec execute 36 sec latency 86.451 ms 1 1139 0.04 MB/sec execute 37 sec latency 86.630 ms 1 1193 0.04 MB/sec execute 38 sec latency 78.407 ms 1 1242 0.04 MB/sec execute 39 sec latency 87.857 ms oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 1 1380 0.11 MB/sec execute 40 sec latency 94.252 ms 1 1426 0.11 MB/sec execute 41 sec latency 94.062 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1470 0.10 MB/sec execute 42 sec latency 305.993 ms 1 1522 0.10 MB/sec execute 43 sec latency 77.431 ms 1 1597 0.10 MB/sec execute 44 sec latency 60.766 ms 1 1644 0.10 MB/sec execute 45 sec latency 113.088 ms 1 1689 0.10 MB/sec execute 46 sec latency 88.416 ms 1 1770 0.10 MB/sec execute 47 sec latency 78.860 ms 1 1845 0.10 MB/sec execute 48 sec latency 62.405 ms 1 1908 0.10 MB/sec execute 49 sec latency 97.039 ms 1 1988 0.10 MB/sec execute 50 sec latency 67.770 ms 1 2060 0.10 MB/sec execute 51 sec latency 76.166 ms 1 2134 0.10 MB/sec execute 52 sec latency 103.159 ms 1 2162 0.10 MB/sec execute 53 sec latency 158.599 ms 1 2225 0.09 MB/sec execute 54 sec latency 74.269 ms 1 2337 0.11 MB/sec execute 55 sec latency 57.803 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Sep 1 20:45:47 EDT 2026 with return code 0 1 2377 0.11 MB/sec execute 56 sec latency 102.987 ms 68684 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 68684 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished