== replay-dual test 26: dbench and tar with mds failover ========================================================== 16:57:28 (1780088248) Starting client oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre Started clients oleg414-client.virtnet: 192.168.204.114@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44442 Started dbench loop with 44443 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri May 29 16:57:31 EDT 2026 waiting for dbench pid 44475 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3732480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3757056 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 28672 7489536 1% /mnt/lustre 1 111 5.24 MB/sec warmup 1 sec latency 49.354 ms 1 181 3.66 MB/sec warmup 2 sec latency 89.924 ms 1 272 3.39 MB/sec warmup 3 sec latency 50.856 ms 1 373 3.33 MB/sec warmup 4 sec latency 49.378 ms test_26 fail mds1 1 times 1 455 3.18 MB/sec warmup 5 sec latency 71.651 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 584 3.32 MB/sec warmup 6 sec latency 53.022 ms 1 600 2.99 MB/sec warmup 7 sec latency 917.033 ms 16:57:38 (1780088258) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server 1 600 2.62 MB/sec warmup 8 sec latency 1917.309 ms 1 600 2.33 MB/sec warmup 9 sec latency 2918.152 ms 1 600 2.10 MB/sec warmup 10 sec latency 3918.417 ms 1 600 1.91 MB/sec warmup 11 sec latency 4918.772 ms 1 600 1.75 MB/sec warmup 12 sec latency 5918.991 ms 1 600 1.61 MB/sec warmup 13 sec latency 6919.263 ms 1 600 1.50 MB/sec warmup 14 sec latency 7919.516 ms 1 600 1.40 MB/sec warmup 15 sec latency 8919.776 ms 1 600 1.31 MB/sec warmup 16 sec latency 9919.996 ms 1 600 1.23 MB/sec warmup 17 sec latency 10920.177 ms Failover mds1 to oleg414-server mount facets: mds1 1 600 1.16 MB/sec warmup 18 sec latency 11920.356 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 600 1.10 MB/sec warmup 19 sec latency 12920.771 ms 1 600 0.00 MB/sec execute 1 sec latency 14922.370 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 1 600 0.00 MB/sec execute 2 sec latency 15922.551 ms Started lustre-MDT0000 16:57:53 (1780088273) targets are mounted 16:57:53 (1780088273) facet_failover done 1 600 0.00 MB/sec execute 3 sec latency 16922.728 ms 1 600 0.00 MB/sec execute 4 sec latency 17923.050 ms 1 600 0.00 MB/sec execute 5 sec latency 18923.298 ms 1 600 0.00 MB/sec execute 6 sec latency 19923.499 ms 1 616 0.00 MB/sec execute 7 sec latency 20700.447 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 713 0.03 MB/sec execute 8 sec latency 95.000 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 785 0.05 MB/sec execute 9 sec latency 100.179 ms 1 863 0.06 MB/sec execute 10 sec latency 109.372 ms 1 919 0.07 MB/sec execute 11 sec latency 77.863 ms 1 995 0.08 MB/sec execute 12 sec latency 87.303 ms 1 1082 0.15 MB/sec execute 13 sec latency 91.415 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 46080 3684352 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7419904 1% /mnt/lustre 1 1173 0.15 MB/sec execute 14 sec latency 75.371 ms 1 1246 0.14 MB/sec execute 15 sec latency 53.356 ms 1 1422 0.32 MB/sec execute 16 sec latency 46.947 ms 1 1523 0.30 MB/sec execute 17 sec latency 78.396 ms test_26 fail mds1 2 times 1 1582 0.29 MB/sec execute 18 sec latency 72.867 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 1650 0.28 MB/sec execute 19 sec latency 96.427 ms 1 1650 0.26 MB/sec execute 20 sec latency 1059.288 ms 16:58:11 (1780088291) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server 1 1650 0.25 MB/sec execute 21 sec latency 2059.620 ms 1 1650 0.24 MB/sec execute 22 sec latency 3059.819 ms 1 1650 0.23 MB/sec execute 23 sec latency 4060.082 ms 1 1650 0.22 MB/sec execute 24 sec latency 5060.313 ms 1 1650 0.21 MB/sec execute 25 sec latency 6060.548 ms 1 1650 0.20 MB/sec execute 26 sec latency 7060.883 ms 1 1650 0.19 MB/sec execute 27 sec latency 8061.090 ms 1 1650 0.19 MB/sec execute 28 sec latency 9061.288 ms 1 1650 0.18 MB/sec execute 29 sec latency 10061.495 ms 1 1650 0.17 MB/sec execute 30 sec latency 11061.739 ms Failover mds1 to oleg414-server mount facets: mds1 1 1650 0.17 MB/sec execute 31 sec latency 12062.206 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1650 0.16 MB/sec execute 32 sec latency 13063.190 ms 1 1650 0.16 MB/sec execute 33 sec latency 14063.499 ms 1 1650 0.15 MB/sec execute 34 sec latency 15065.208 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 1 1650 0.15 MB/sec execute 35 sec latency 16065.553 ms pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 16:58:26 (1780088306) targets are mounted 16:58:26 (1780088306) facet_failover done 1 1650 0.15 MB/sec execute 36 sec latency 17065.747 ms 1 1700 0.14 MB/sec execute 37 sec latency 17430.926 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1823 0.14 MB/sec execute 38 sec latency 44.629 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1907 0.14 MB/sec execute 39 sec latency 91.589 ms 1 2020 0.14 MB/sec execute 40 sec latency 42.576 ms 1 2160 0.14 MB/sec execute 41 sec latency 35.607 ms 1 2342 0.16 MB/sec execute 42 sec latency 36.287 ms 1 2429 0.17 MB/sec execute 43 sec latency 100.504 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 51200 3596288 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3628032 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7224320 2% /mnt/lustre 1 2566 0.18 MB/sec execute 44 sec latency 115.942 ms 1 2609 0.18 MB/sec execute 45 sec latency 684.526 ms 1 2931 0.27 MB/sec execute 46 sec latency 56.523 ms test_26 fail mds1 3 times 1 3197 0.28 MB/sec execute 47 sec latency 46.466 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 3377 0.30 MB/sec execute 48 sec latency 58.031 ms 1 3390 0.30 MB/sec execute 49 sec latency 912.119 ms 16:58:40 (1780088320) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server 1 3390 0.29 MB/sec execute 50 sec latency 1912.327 ms 1 3390 0.28 MB/sec execute 51 sec latency 2912.553 ms 1 3390 0.28 MB/sec execute 52 sec latency 3913.538 ms 1 3390 0.27 MB/sec execute 53 sec latency 4913.759 ms 1 3390 0.27 MB/sec execute 54 sec latency 5913.937 ms 1 3390 0.26 MB/sec execute 55 sec latency 6914.168 ms 1 3390 0.26 MB/sec execute 56 sec latency 7914.406 ms 1 3390 0.25 MB/sec execute 57 sec latency 8914.608 ms 1 3390 0.25 MB/sec execute 58 sec latency 9914.873 ms 1 3390 0.25 MB/sec execute 59 sec latency 10915.036 ms Failover mds1 to oleg414-server mount facets: mds1 1 3390 0.24 MB/sec execute 60 sec latency 11915.356 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3390 0.24 MB/sec execute 61 sec latency 12915.535 ms 1 3390 0.23 MB/sec execute 62 sec latency 13917.734 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 1 3390 0.23 MB/sec execute 63 sec latency 14917.901 ms pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 16:58:54 (1780088334) targets are mounted 16:58:54 (1780088334) facet_failover done 1 3390 0.23 MB/sec execute 64 sec latency 15918.084 ms 1 3390 0.22 MB/sec execute 65 sec latency 16918.286 ms 1 3390 0.22 MB/sec execute 66 sec latency 17918.466 ms 1 3390 0.22 MB/sec execute 67 sec latency 18918.922 ms 1 3535 0.26 MB/sec execute 68 sec latency 19302.770 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3669 0.26 MB/sec execute 69 sec latency 30.404 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3785 0.27 MB/sec execute 70 sec latency 44.977 ms 1 3886 0.26 MB/sec execute 71 sec latency 43.390 ms 1 3998 0.27 MB/sec execute 72 sec latency 34.879 ms 1 4094 0.26 MB/sec execute 73 sec latency 74.127 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 55296 3701760 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3723264 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 80896 7425024 2% /mnt/lustre 1 4226 0.26 MB/sec execute 74 sec latency 34.489 ms 1 4302 0.26 MB/sec execute 75 sec latency 144.402 ms 1 4387 0.26 MB/sec execute 76 sec latency 77.243 ms test_26 fail mds1 4 times 1 4489 0.26 MB/sec execute 77 sec latency 83.868 ms Failing mds1 on oleg414-server 1 4589 0.26 MB/sec execute 78 sec latency 67.058 ms Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 4651 0.27 MB/sec execute 79 sec latency 614.608 ms 16:59:10 (1780088350) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server 1 4651 0.26 MB/sec execute 80 sec latency 1615.401 ms 1 4651 0.26 MB/sec execute 81 sec latency 2615.661 ms 1 4651 0.26 MB/sec execute 82 sec latency 3615.915 ms 1 4651 0.25 MB/sec execute 83 sec latency 4616.118 ms 1 4651 0.25 MB/sec execute 84 sec latency 5616.298 ms 1 4651 0.25 MB/sec execute 85 sec latency 6616.551 ms 1 4651 0.25 MB/sec execute 86 sec latency 7616.763 ms 1 4651 0.24 MB/sec execute 87 sec latency 8616.944 ms 1 4651 0.24 MB/sec execute 88 sec latency 9617.162 ms 1 4651 0.24 MB/sec execute 89 sec latency 10617.346 ms Failover mds1 to oleg414-server mount facets: mds1 1 4651 0.23 MB/sec execute 90 sec latency 11617.538 ms 1 4651 0.23 MB/sec execute 91 sec latency 12617.781 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4651 0.23 MB/sec execute 92 sec latency 13617.913 ms 1 4651 0.23 MB/sec execute 93 sec latency 14618.118 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 1 4651 0.22 MB/sec execute 94 sec latency 15618.291 ms Started lustre-MDT0000 16:59:25 (1780088365) targets are mounted 16:59:25 (1780088365) facet_failover done 1 4651 0.22 MB/sec execute 95 sec latency 16618.475 ms 1 4651 0.22 MB/sec execute 96 sec latency 17618.649 ms 1 4651 0.22 MB/sec execute 97 sec latency 18618.866 ms 1 4652 0.22 MB/sec execute 98 sec latency 19576.540 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4768 0.21 MB/sec execute 99 sec latency 103.911 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 cleanup 100 sec 1 cleanup 101 sec 0 cleanup 102 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 723 22.660 86.241 Close 551 109.315 20700.427 Rename 36 37.090 103.896 Unlink 141 148.533 19302.739 Qpathinfo 666 7.330 46.353 Qfileinfo 113 1.344 6.849 Qfsinfo 135 1.278 56.964 Sfileinfo 84 24.632 55.184 Find 257 3.396 50.596 WriteX 384 4.860 38.286 ReadX 1136 0.886 91.397 LockX 2 3.107 3.396 UnlockX 2 4.119 6.292 Flush 67 48.248 144.374 Throughput 0.214037 MB/sec 1 clients 1 procs max_latency=20700.447 ms stopping dbench on /mnt/lustre at Fri May 29 16:59:32 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri May 29 16:59:35 EDT 2026 waiting for dbench pid 48349 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 55296 3569664 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3628032 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 79872 7197696 2% /mnt/lustre 1 of 1 processes prepared for launch 0 sec releasing clients 1 117 5.31 MB/sec warmup 1 sec latency 59.725 ms 1 213 4.15 MB/sec warmup 2 sec latency 64.505 ms 1 329 3.98 MB/sec warmup 3 sec latency 44.286 ms test_26 fail mds1 5 times Failing mds1 on oleg414-server 1 445 3.91 MB/sec warmup 4 sec latency 42.603 ms Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 513 3.54 MB/sec warmup 5 sec latency 394.397 ms 16:59:41 (1780088381) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server 1 513 2.95 MB/sec warmup 6 sec latency 1394.561 ms 1 513 2.53 MB/sec warmup 7 sec latency 2395.560 ms 1 513 2.22 MB/sec warmup 8 sec latency 3395.819 ms 1 513 1.97 MB/sec warmup 9 sec latency 4396.012 ms 1 513 1.77 MB/sec warmup 10 sec latency 5396.177 ms 1 513 1.61 MB/sec warmup 11 sec latency 6396.376 ms 1 513 1.48 MB/sec warmup 12 sec latency 7396.566 ms 1 513 1.36 MB/sec warmup 13 sec latency 8396.815 ms 1 513 1.27 MB/sec warmup 14 sec latency 9397.010 ms 1 513 1.18 MB/sec warmup 15 sec latency 10397.205 ms Failover mds1 to oleg414-server mount facets: mds1 1 513 1.11 MB/sec warmup 16 sec latency 11397.329 ms 1 513 1.04 MB/sec warmup 17 sec latency 12397.516 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 513 0.98 MB/sec warmup 18 sec latency 13397.709 ms 1 513 0.93 MB/sec warmup 19 sec latency 14397.897 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 16:59:55 (1780088395) targets are mounted 16:59:55 (1780088395) facet_failover done 1 513 0.00 MB/sec execute 1 sec latency 16398.233 ms 1 513 0.00 MB/sec execute 2 sec latency 17398.525 ms 1 513 0.00 MB/sec execute 3 sec latency 18398.734 ms 1 513 0.00 MB/sec execute 4 sec latency 19398.933 ms 1 520 0.05 MB/sec execute 5 sec latency 20363.723 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 664 0.57 MB/sec execute 6 sec latency 136.340 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 730 0.51 MB/sec execute 7 sec latency 178.728 ms 1 801 0.47 MB/sec execute 8 sec latency 77.103 ms 1 902 0.44 MB/sec execute 9 sec latency 99.527 ms 1 985 0.42 MB/sec execute 10 sec latency 74.921 ms 1 1105 0.47 MB/sec execute 11 sec latency 38.371 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4608 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 48128 3649536 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3684352 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 7333888 1% /mnt/lustre 1 1214 0.44 MB/sec execute 12 sec latency 43.534 ms 1 1274 0.42 MB/sec execute 13 sec latency 96.851 ms 1 1461 0.59 MB/sec execute 14 sec latency 58.243 ms test_26 fail mds1 6 times 1 1552 0.56 MB/sec execute 15 sec latency 124.723 ms Failing mds1 on oleg414-server 1 1616 0.53 MB/sec execute 16 sec latency 51.400 ms Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 1655 0.50 MB/sec execute 17 sec latency 409.496 ms 17:00:13 (1780088413) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server 1 1655 0.47 MB/sec execute 18 sec latency 1409.618 ms 1 1655 0.45 MB/sec execute 19 sec latency 2409.831 ms 1 1655 0.42 MB/sec execute 20 sec latency 3410.138 ms 1 1655 0.40 MB/sec execute 21 sec latency 4410.369 ms 1 1655 0.39 MB/sec execute 22 sec latency 5410.829 ms 1 1655 0.37 MB/sec execute 23 sec latency 6411.718 ms 1 1655 0.35 MB/sec execute 24 sec latency 7411.940 ms 1 1655 0.34 MB/sec execute 25 sec latency 8412.142 ms 1 1655 0.33 MB/sec execute 26 sec latency 9412.513 ms 1 1655 0.31 MB/sec execute 27 sec latency 10413.156 ms 1 1655 0.30 MB/sec execute 28 sec latency 11415.777 ms Failover mds1 to oleg414-server mount facets: mds1 1 1655 0.29 MB/sec execute 29 sec latency 12416.051 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1655 0.28 MB/sec execute 30 sec latency 13416.392 ms 1 1655 0.27 MB/sec execute 31 sec latency 14416.616 ms 1 1655 0.26 MB/sec execute 32 sec latency 15417.010 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 1 1655 0.26 MB/sec execute 33 sec latency 16417.141 ms Started lustre-MDT0000 17:00:28 (1780088428) targets are mounted 17:00:28 (1780088428) facet_failover done 1 1655 0.25 MB/sec execute 34 sec latency 17417.485 ms 1 1689 0.24 MB/sec execute 35 sec latency 17937.947 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1812 0.24 MB/sec execute 36 sec latency 47.023 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1891 0.23 MB/sec execute 37 sec latency 205.607 ms 1 2012 0.23 MB/sec execute 38 sec latency 41.384 ms 1 2152 0.23 MB/sec execute 39 sec latency 40.136 ms 1 2305 0.25 MB/sec execute 40 sec latency 40.688 ms 1 2377 0.25 MB/sec execute 41 sec latency 332.456 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3456 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 52224 3695616 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 12288 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 64512 7440384 1% /mnt/lustre 1 2447 0.25 MB/sec execute 42 sec latency 142.205 ms 1 2566 0.26 MB/sec execute 43 sec latency 152.366 ms 1 2597 0.26 MB/sec execute 44 sec latency 645.348 ms 1 2896 0.34 MB/sec execute 45 sec latency 145.331 ms test_26 fail mds1 7 times Failing mds1 on oleg414-server 1 2989 0.34 MB/sec execute 46 sec latency 158.713 ms Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 3198 0.35 MB/sec execute 47 sec latency 129.150 ms 1 3198 0.34 MB/sec execute 48 sec latency 1129.577 ms 17:00:43 (1780088443) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server 1 3198 0.34 MB/sec execute 49 sec latency 2130.755 ms 1 3198 0.33 MB/sec execute 50 sec latency 3130.923 ms 1 3198 0.32 MB/sec execute 51 sec latency 4131.111 ms 1 3198 0.32 MB/sec execute 52 sec latency 5131.389 ms 1 3198 0.31 MB/sec execute 53 sec latency 6131.599 ms 1 3198 0.31 MB/sec execute 54 sec latency 7132.059 ms 1 3198 0.30 MB/sec execute 55 sec latency 8132.251 ms 1 3198 0.30 MB/sec execute 56 sec latency 9132.600 ms 1 3198 0.29 MB/sec execute 57 sec latency 10132.797 ms 1 3198 0.29 MB/sec execute 58 sec latency 11132.991 ms Failover mds1 to oleg414-server mount facets: mds1 1 3198 0.28 MB/sec execute 59 sec latency 12133.181 ms 1 3198 0.28 MB/sec execute 60 sec latency 13133.666 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3198 0.27 MB/sec execute 61 sec latency 14133.886 ms 1 3198 0.27 MB/sec execute 62 sec latency 15134.156 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 1 3198 0.26 MB/sec execute 63 sec latency 16134.339 ms pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 17:00:59 (1780088459) targets are mounted 17:00:59 (1780088459) facet_failover done 1 3198 0.26 MB/sec execute 64 sec latency 17134.584 ms 1 3198 0.25 MB/sec execute 65 sec latency 18134.905 ms 1 3341 0.27 MB/sec execute 66 sec latency 18210.224 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3548 0.31 MB/sec execute 67 sec latency 36.657 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3688 0.31 MB/sec execute 68 sec latency 32.226 ms 1 3805 0.32 MB/sec execute 69 sec latency 58.256 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri May 29 17:01:05 EDT 2026 with return code 0 1 3910 0.32 MB/sec execute 70 sec latency 35.444 ms 48349 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48349 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished