== replay-dual test 26: dbench and tar with mds failover ========================================================== 18:19:12 (1786832352) Starting client oleg618-client.virtnet: -o user_xattr,flock 192.168.206.118@tcp:/lustre /mnt/lustre Started clients oleg618-client.virtnet: 192.168.206.118@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45166 Started dbench loop with 45167 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3760128 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3730432 1% /mnt/lustre[OST:1] running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Aug 15 18:19:16 EDT 2026 filesystem_summary: 7542784 20480 7490560 1% /mnt/lustre waiting for dbench pid 45206 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 49 2.03 MB/sec warmup 1 sec latency 95.030 ms 1 113 2.62 MB/sec warmup 2 sec latency 129.066 ms 1 150 2.09 MB/sec warmup 3 sec latency 144.989 ms 1 189 1.89 MB/sec warmup 4 sec latency 137.453 ms 1 239 1.82 MB/sec warmup 5 sec latency 95.790 ms test_26 fail mds1 1 times 1 284 1.76 MB/sec warmup 6 sec latency 136.236 ms Failing mds1 on oleg618-server 1 323 1.67 MB/sec warmup 7 sec latency 121.555 ms Stopping /mnt/lustre-mds1 (opts:) on oleg618-server 1 365 1.63 MB/sec warmup 8 sec latency 100.687 ms 1 381 1.50 MB/sec warmup 9 sec latency 720.066 ms 1 381 1.35 MB/sec warmup 10 sec latency 1720.269 ms 18:19:27 (1786832367) shut down facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server 1 381 1.23 MB/sec warmup 11 sec latency 2720.491 ms 1 381 1.13 MB/sec warmup 12 sec latency 3720.659 ms 1 381 1.04 MB/sec warmup 13 sec latency 4720.890 ms 1 381 0.97 MB/sec warmup 14 sec latency 5721.751 ms 1 381 0.90 MB/sec warmup 15 sec latency 6722.107 ms 1 381 0.85 MB/sec warmup 16 sec latency 7722.880 ms 1 381 0.80 MB/sec warmup 17 sec latency 8723.517 ms 1 381 0.75 MB/sec warmup 18 sec latency 9723.694 ms 1 381 0.71 MB/sec warmup 19 sec latency 10725.363 ms Failover mds1 to oleg618-server mount facets: mds1 1 381 0.00 MB/sec execute 1 sec latency 12725.679 ms 1 381 0.00 MB/sec execute 2 sec latency 13725.889 ms 1 381 0.00 MB/sec execute 3 sec latency 14726.274 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 381 0.00 MB/sec execute 4 sec latency 15730.710 ms 1 381 0.00 MB/sec execute 5 sec latency 16730.976 ms 1 381 0.00 MB/sec execute 6 sec latency 17733.276 ms 1 381 0.00 MB/sec execute 7 sec latency 18733.458 ms 1 381 0.00 MB/sec execute 8 sec latency 19733.862 ms 1 405 0.09 MB/sec execute 9 sec latency 20221.321 ms oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all 1 460 0.26 MB/sec execute 10 sec latency 96.965 ms pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 Started lustre-MDT0000 18:19:47 (1786832387) targets are mounted 18:19:47 (1786832387) facet_failover done 1 511 0.38 MB/sec execute 11 sec latency 107.880 ms 1 615 0.62 MB/sec execute 12 sec latency 112.299 ms 1 669 0.58 MB/sec execute 13 sec latency 222.716 ms 1 715 0.54 MB/sec execute 14 sec latency 84.135 ms 1 759 0.52 MB/sec execute 15 sec latency 143.065 ms oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 794 0.49 MB/sec execute 16 sec latency 134.423 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 851 0.47 MB/sec execute 17 sec latency 85.970 ms 1 897 0.45 MB/sec execute 18 sec latency 126.814 ms 1 943 0.43 MB/sec execute 19 sec latency 150.014 ms 1 986 0.42 MB/sec execute 20 sec latency 196.792 ms 1 1076 0.45 MB/sec execute 21 sec latency 73.462 ms 1 1139 0.43 MB/sec execute 22 sec latency 59.812 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3701760 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 36864 3677184 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7378944 1% /mnt/lustre 1 1192 0.41 MB/sec execute 23 sec latency 79.088 ms 1 1242 0.40 MB/sec execute 24 sec latency 68.727 ms 1 1379 0.50 MB/sec execute 25 sec latency 100.555 ms 1 1419 0.48 MB/sec execute 26 sec latency 80.987 ms 1 1481 0.46 MB/sec execute 27 sec latency 71.334 ms test_26 fail mds1 2 times 1 1532 0.45 MB/sec execute 28 sec latency 149.799 ms Failing mds1 on oleg618-server 1 1603 0.43 MB/sec execute 29 sec latency 80.171 ms Stopping /mnt/lustre-mds1 (opts:) on oleg618-server 1 1651 0.42 MB/sec execute 30 sec latency 170.230 ms 1 1684 0.41 MB/sec execute 31 sec latency 395.001 ms 1 1684 0.39 MB/sec execute 32 sec latency 1395.900 ms 1 1684 0.38 MB/sec execute 33 sec latency 2396.036 ms 18:20:09 (1786832409) shut down facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server 1 1684 0.37 MB/sec execute 34 sec latency 3396.245 ms 1 1684 0.36 MB/sec execute 35 sec latency 4397.071 ms 1 1684 0.35 MB/sec execute 36 sec latency 5398.090 ms 1 1684 0.34 MB/sec execute 37 sec latency 6399.152 ms 1 1684 0.33 MB/sec execute 38 sec latency 7399.351 ms 1 1684 0.32 MB/sec execute 39 sec latency 8399.607 ms 1 1684 0.32 MB/sec execute 40 sec latency 9399.971 ms 1 1684 0.31 MB/sec execute 41 sec latency 10400.155 ms 1 1684 0.30 MB/sec execute 42 sec latency 11401.862 ms 1 1684 0.29 MB/sec execute 43 sec latency 12403.947 ms Failover mds1 to oleg618-server mount facets: mds1 1 1684 0.29 MB/sec execute 44 sec latency 13406.678 ms 1 1684 0.28 MB/sec execute 45 sec latency 14408.805 ms 1 1684 0.27 MB/sec execute 46 sec latency 15409.033 ms 1 1684 0.27 MB/sec execute 47 sec latency 16409.240 ms 1 1684 0.26 MB/sec execute 48 sec latency 17410.948 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1684 0.26 MB/sec execute 49 sec latency 18411.184 ms 1 1684 0.25 MB/sec execute 50 sec latency 19411.634 ms 1 1684 0.25 MB/sec execute 51 sec latency 20411.873 ms 1 1684 0.24 MB/sec execute 52 sec latency 21412.287 ms 1 1684 0.24 MB/sec execute 53 sec latency 22412.458 ms 1 1684 0.23 MB/sec execute 54 sec latency 23414.284 ms 1 1684 0.23 MB/sec execute 55 sec latency 24414.599 ms 1 1696 0.23 MB/sec execute 56 sec latency 24573.783 ms oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all 1 1725 0.22 MB/sec execute 57 sec latency 138.254 ms pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 1 1773 0.22 MB/sec execute 58 sec latency 101.715 ms 1 1821 0.22 MB/sec execute 59 sec latency 103.000 ms Started lustre-MDT0000 18:20:35 (1786832435) targets are mounted 18:20:35 (1786832435) facet_failover done 1 1861 0.21 MB/sec execute 60 sec latency 131.327 ms 1 1888 0.21 MB/sec execute 61 sec latency 310.067 ms 1 1956 0.21 MB/sec execute 62 sec latency 331.635 ms 1 2002 0.21 MB/sec execute 63 sec latency 122.923 ms 1 2062 0.20 MB/sec execute 64 sec latency 90.971 ms oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 2139 0.20 MB/sec execute 65 sec latency 66.135 ms 1 2170 0.20 MB/sec execute 66 sec latency 114.980 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2226 0.20 MB/sec execute 67 sec latency 146.450 ms 1 2337 0.21 MB/sec execute 68 sec latency 50.763 ms 1 2376 0.21 MB/sec execute 69 sec latency 187.788 ms 1 2418 0.21 MB/sec execute 70 sec latency 194.790 ms 1 2475 0.21 MB/sec execute 71 sec latency 87.809 ms 1 2559 0.21 MB/sec execute 72 sec latency 260.186 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3712000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 51200 3679232 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 74752 7391232 2% /mnt/lustre 1 2566 0.21 MB/sec execute 73 sec latency 875.173 ms 1 2594 0.21 MB/sec execute 74 sec latency 1197.759 ms 1 2719 0.24 MB/sec execute 75 sec latency 116.392 ms 1 2780 0.24 MB/sec execute 76 sec latency 217.369 ms 1 2918 0.26 MB/sec execute 77 sec latency 69.487 ms test_26 fail mds1 3 times 1 2984 0.26 MB/sec execute 78 sec latency 137.897 ms 1 3024 0.25 MB/sec execute 79 sec latency 175.919 ms Failing mds1 on oleg618-server 1 3212 0.26 MB/sec execute 80 sec latency 158.399 ms Stopping /mnt/lustre-mds1 (opts:) on oleg618-server 1 3297 0.27 MB/sec execute 81 sec latency 174.752 ms 1 3297 0.27 MB/sec execute 82 sec latency 1175.598 ms 1 3297 0.26 MB/sec execute 83 sec latency 2175.968 ms 18:20:59 (1786832459) shut down facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server 1 3297 0.26 MB/sec execute 84 sec latency 3176.492 ms 1 3297 0.26 MB/sec execute 85 sec latency 4177.650 ms 1 3297 0.25 MB/sec execute 86 sec latency 5178.095 ms 1 3297 0.25 MB/sec execute 87 sec latency 6182.506 ms 1 3297 0.25 MB/sec execute 88 sec latency 7182.700 ms 1 3297 0.24 MB/sec execute 89 sec latency 8182.927 ms 1 3297 0.24 MB/sec execute 90 sec latency 9183.155 ms 1 3297 0.24 MB/sec execute 91 sec latency 10183.820 ms 1 3297 0.24 MB/sec execute 92 sec latency 11184.206 ms 1 3297 0.23 MB/sec execute 93 sec latency 12184.403 ms Failover mds1 to oleg618-server mount facets: mds1 1 3297 0.23 MB/sec execute 94 sec latency 13184.915 ms 1 3297 0.23 MB/sec execute 95 sec latency 14185.946 ms 1 3297 0.23 MB/sec execute 96 sec latency 15187.105 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3297 0.22 MB/sec execute 97 sec latency 16187.274 ms 1 3297 0.22 MB/sec execute 98 sec latency 17188.073 ms 1 3297 0.22 MB/sec execute 99 sec latency 18188.267 ms 1 cleanup 100 sec 1 cleanup 101 sec oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all 1 cleanup 102 sec pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 1 cleanup 103 sec Started lustre-MDT0000 18:21:20 (1786832480) targets are mounted 18:21:20 (1786832480) facet_failover done 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 0 cleanup 106 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 434 151.082 24573.763 Close 361 10.027 45.138 Rename 22 87.669 333.691 Unlink 49 23.833 45.916 Qpathinfo 410 15.415 109.221 Qfileinfo 121 1.900 13.484 Qfsinfo 67 2.099 16.296 Sfileinfo 50 43.868 81.577 Find 150 4.787 33.002 WriteX 404 9.848 98.940 ReadX 805 1.186 83.474 LockX 2 6.138 7.435 UnlockX 2 9.554 10.156 Flush 40 654.692 21572.378 Throughput 0.219839 MB/sec 1 clients 1 procs max_latency=24573.783 ms stopping dbench on /mnt/lustre at Sat Aug 15 18:21:23 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Aug 15 18:21:29 EDT 2026 waiting for dbench pid 48077 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.04 MB/sec warmup 1 sec latency 57.810 ms 1 98 2.22 MB/sec warmup 2 sec latency 129.824 ms 1 139 2.00 MB/sec warmup 3 sec latency 134.929 ms 1 189 1.89 MB/sec warmup 4 sec latency 104.629 ms 1 231 1.77 MB/sec warmup 5 sec latency 109.818 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3714048 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 43008 3681280 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 59392 7395328 1% /mnt/lustre 1 279 1.74 MB/sec warmup 6 sec latency 119.222 ms 1 314 1.64 MB/sec warmup 7 sec latency 173.164 ms 1 365 1.63 MB/sec warmup 8 sec latency 117.245 ms 1 421 1.65 MB/sec warmup 9 sec latency 80.221 ms 1 481 1.67 MB/sec warmup 10 sec latency 85.244 ms test_26 fail mds1 4 times 1 522 1.64 MB/sec warmup 11 sec latency 122.004 ms 1 616 1.75 MB/sec warmup 12 sec latency 105.826 ms Failing mds1 on oleg618-server 1 666 1.63 MB/sec warmup 13 sec latency 271.202 ms Stopping /mnt/lustre-mds1 (opts:) on oleg618-server 1 702 1.51 MB/sec warmup 14 sec latency 116.846 ms 1 704 1.41 MB/sec warmup 15 sec latency 979.476 ms 1 704 1.32 MB/sec warmup 16 sec latency 1979.662 ms 18:21:45 (1786832505) shut down facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server 1 704 1.25 MB/sec warmup 17 sec latency 2980.922 ms 1 704 1.18 MB/sec warmup 18 sec latency 3981.129 ms 1 704 1.11 MB/sec warmup 19 sec latency 4988.751 ms 1 704 0.00 MB/sec execute 1 sec latency 6991.154 ms 1 704 0.00 MB/sec execute 2 sec latency 7995.331 ms 1 704 0.00 MB/sec execute 3 sec latency 8996.666 ms 1 704 0.00 MB/sec execute 4 sec latency 10001.646 ms 1 704 0.00 MB/sec execute 5 sec latency 11003.156 ms 1 704 0.00 MB/sec execute 6 sec latency 12003.359 ms Failover mds1 to oleg618-server mount facets: mds1 1 704 0.00 MB/sec execute 7 sec latency 13003.496 ms 1 704 0.00 MB/sec execute 8 sec latency 14005.005 ms 1 704 0.00 MB/sec execute 9 sec latency 15007.108 ms 1 704 0.00 MB/sec execute 10 sec latency 16007.250 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 704 0.00 MB/sec execute 11 sec latency 17007.974 ms 1 704 0.00 MB/sec execute 12 sec latency 18008.406 ms 1 704 0.00 MB/sec execute 13 sec latency 19008.614 ms 1 704 0.00 MB/sec execute 14 sec latency 20008.941 ms 1 704 0.00 MB/sec execute 15 sec latency 21009.215 ms 1 704 0.00 MB/sec execute 16 sec latency 22009.647 ms 1 704 0.00 MB/sec execute 17 sec latency 23009.970 ms 1 704 0.00 MB/sec execute 18 sec latency 24012.182 ms 1 704 0.00 MB/sec execute 19 sec latency 25013.847 ms 1 704 0.00 MB/sec execute 20 sec latency 26016.294 ms 1 704 0.00 MB/sec execute 21 sec latency 27016.490 ms 1 704 0.00 MB/sec execute 22 sec latency 28017.350 ms 1 704 0.00 MB/sec execute 23 sec latency 29017.562 ms 1 704 0.00 MB/sec execute 24 sec latency 30017.825 ms oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all 1 704 0.00 MB/sec execute 25 sec latency 31017.990 ms pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1 1 704 0.00 MB/sec execute 26 sec latency 32018.414 ms Started lustre-MDT0000 18:22:15 (1786832535) targets are mounted 18:22:15 (1786832535) facet_failover done 1 704 0.00 MB/sec execute 27 sec latency 33019.033 ms 1 704 0.00 MB/sec execute 28 sec latency 34021.441 ms 1 704 0.00 MB/sec execute 29 sec latency 35023.571 ms 1 704 0.00 MB/sec execute 30 sec latency 36023.778 ms 1 704 0.00 MB/sec execute 31 sec latency 37024.347 ms 1 704 0.00 MB/sec execute 32 sec latency 38024.700 ms 1 704 0.00 MB/sec execute 33 sec latency 39025.766 ms 1 704 0.00 MB/sec execute 34 sec latency 40026.604 ms 1 705 0.00 MB/sec execute 35 sec latency 41023.378 ms 1 728 0.00 MB/sec execute 36 sec latency 206.859 ms 1 770 0.00 MB/sec execute 37 sec latency 89.292 ms 1 791 0.01 MB/sec execute 38 sec latency 310.942 ms 1 817 0.01 MB/sec execute 39 sec latency 111.000 ms oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 873 0.01 MB/sec execute 40 sec latency 98.873 ms 1 900 0.01 MB/sec execute 41 sec latency 227.572 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 918 0.01 MB/sec execute 42 sec latency 202.103 ms 1 966 0.01 MB/sec execute 43 sec latency 114.426 ms 1 995 0.02 MB/sec execute 44 sec latency 141.973 ms 1 1050 0.04 MB/sec execute 45 sec latency 151.795 ms 1 1117 0.04 MB/sec execute 46 sec latency 97.778 ms 1 1155 0.04 MB/sec execute 47 sec latency 115.822 ms 1 1202 0.04 MB/sec execute 48 sec latency 75.874 ms 1 1257 0.04 MB/sec execute 49 sec latency 73.068 ms 1 1391 0.10 MB/sec execute 50 sec latency 94.497 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1447 0.09 MB/sec execute 51 sec latency 82.907 ms 1 1502 0.09 MB/sec execute 52 sec latency 157.078 ms 1 1564 0.09 MB/sec execute 53 sec latency 62.252 ms 1 1609 0.09 MB/sec execute 54 sec latency 78.817 ms 1 1654 0.09 MB/sec execute 55 sec latency 246.237 ms 1 1707 0.09 MB/sec execute 56 sec latency 97.379 ms 1 1799 0.09 MB/sec execute 57 sec latency 65.343 ms 1 1860 0.09 MB/sec execute 58 sec latency 88.778 ms 1 1902 0.09 MB/sec execute 59 sec latency 321.774 ms 1 1986 0.09 MB/sec execute 60 sec latency 65.146 ms 1 2060 0.09 MB/sec execute 61 sec latency 78.641 ms 1 2133 0.09 MB/sec execute 62 sec latency 65.251 ms tar: Error is not recoverable: exiting now 1 2178 0.09 MB/sec execute 63 sec latency 83.329 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Sat Aug 15 18:22:52 EDT 2026 with return code 0 48077 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48077 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished