== replay-dual test 26: dbench and tar with mds failover ========================================================== 05:33:07 (1786354387) Starting client oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre Started clients oleg123-client.virtnet: 192.168.201.123@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45133 Started dbench loop with 45134 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Aug 10 05:33:09 EDT 2026 waiting for dbench pid 45164 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3743744 1% /mnt/lustre[OST:0] 1 129 5.76 MB/sec warmup 1 sec latency 60.284 ms lustre-OST0001_UUID 3771392 28672 3730432 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 30720 7474176 1% /mnt/lustre 1 206 4.05 MB/sec warmup 2 sec latency 85.361 ms 1 308 3.74 MB/sec warmup 3 sec latency 50.248 ms 1 425 3.73 MB/sec warmup 4 sec latency 48.272 ms test_26 fail mds1 1 times 1 535 3.74 MB/sec warmup 5 sec latency 51.926 ms Failing mds1 on oleg123-server Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 1 678 3.53 MB/sec warmup 6 sec latency 107.504 ms 1 678 3.03 MB/sec warmup 7 sec latency 1076.269 ms 05:33:16 (1786354396) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server 1 678 2.65 MB/sec warmup 8 sec latency 2076.417 ms 1 678 2.35 MB/sec warmup 9 sec latency 3076.604 ms 1 678 2.12 MB/sec warmup 10 sec latency 4078.436 ms 1 678 1.92 MB/sec warmup 11 sec latency 5078.589 ms 1 678 1.76 MB/sec warmup 12 sec latency 6078.721 ms 1 678 1.63 MB/sec warmup 13 sec latency 7078.853 ms 1 678 1.51 MB/sec warmup 14 sec latency 8079.101 ms 1 678 1.41 MB/sec warmup 15 sec latency 9079.248 ms 1 678 1.32 MB/sec warmup 16 sec latency 10079.370 ms 1 678 1.25 MB/sec warmup 17 sec latency 11079.500 ms Failover mds1 to oleg123-server mount facets: mds1 1 678 1.18 MB/sec warmup 18 sec latency 12079.612 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 678 1.11 MB/sec warmup 19 sec latency 13081.167 ms 1 678 0.00 MB/sec execute 1 sec latency 15081.355 ms oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 1 678 0.00 MB/sec execute 2 sec latency 16081.831 ms Started lustre-MDT0000 05:33:31 (1786354411) targets are mounted 05:33:31 (1786354411) facet_failover done 1 678 0.00 MB/sec execute 3 sec latency 17081.956 ms 1 678 0.00 MB/sec execute 4 sec latency 18082.119 ms 1 678 0.00 MB/sec execute 5 sec latency 19082.217 ms 1 678 0.00 MB/sec execute 6 sec latency 20082.382 ms 1 740 0.02 MB/sec execute 7 sec latency 20350.967 ms oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 825 0.04 MB/sec execute 8 sec latency 90.401 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 923 0.07 MB/sec execute 9 sec latency 68.649 ms 1 1029 0.17 MB/sec execute 10 sec latency 75.918 ms 1 1161 0.17 MB/sec execute 11 sec latency 46.631 ms 1 1273 0.16 MB/sec execute 12 sec latency 41.116 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3671040 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 45056 3650560 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7321600 1% /mnt/lustre 1 1474 0.37 MB/sec execute 13 sec latency 39.397 ms 1 1574 0.35 MB/sec execute 14 sec latency 98.234 ms 1 1659 0.34 MB/sec execute 15 sec latency 52.717 ms test_26 fail mds1 2 times 1 1791 0.32 MB/sec execute 16 sec latency 52.332 ms Failing mds1 on oleg123-server 1 1917 0.31 MB/sec execute 17 sec latency 78.048 ms Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 1 1986 0.29 MB/sec execute 18 sec latency 444.481 ms 05:33:48 (1786354428) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server 1 1986 0.28 MB/sec execute 19 sec latency 1445.014 ms 1 1986 0.26 MB/sec execute 20 sec latency 2445.162 ms 1 1986 0.25 MB/sec execute 21 sec latency 3445.312 ms 1 1986 0.24 MB/sec execute 22 sec latency 4445.498 ms 1 1986 0.23 MB/sec execute 23 sec latency 5445.780 ms 1 1986 0.22 MB/sec execute 24 sec latency 6446.134 ms 1 1986 0.21 MB/sec execute 25 sec latency 7446.549 ms 1 1986 0.20 MB/sec execute 26 sec latency 8446.763 ms 1 1986 0.20 MB/sec execute 27 sec latency 9446.942 ms 1 1986 0.19 MB/sec execute 28 sec latency 10447.147 ms Failover mds1 to oleg123-server mount facets: mds1 1 1986 0.18 MB/sec execute 29 sec latency 11447.321 ms 1 1986 0.18 MB/sec execute 30 sec latency 12447.509 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1986 0.17 MB/sec execute 31 sec latency 13447.763 ms 1 1986 0.17 MB/sec execute 32 sec latency 14447.936 ms oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all 1 1986 0.16 MB/sec execute 33 sec latency 15448.141 ms pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 05:34:03 (1786354443) targets are mounted 05:34:03 (1786354443) facet_failover done 1 1986 0.16 MB/sec execute 34 sec latency 16448.353 ms 1 1986 0.15 MB/sec execute 35 sec latency 17448.463 ms 1 2006 0.15 MB/sec execute 36 sec latency 18233.176 ms 1 2110 0.15 MB/sec execute 37 sec latency 53.601 ms oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 2204 0.15 MB/sec execute 38 sec latency 56.320 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2352 0.17 MB/sec execute 39 sec latency 125.089 ms 1 2448 0.17 MB/sec execute 40 sec latency 62.929 ms 1 2566 0.19 MB/sec execute 41 sec latency 331.108 ms 1 2759 0.26 MB/sec execute 42 sec latency 614.010 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3641344 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 57344 3588096 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7229440 2% /mnt/lustre 1 2929 0.29 MB/sec execute 43 sec latency 101.631 ms 1 3025 0.28 MB/sec execute 44 sec latency 79.474 ms 1 3297 0.31 MB/sec execute 45 sec latency 118.308 ms 1 3507 0.37 MB/sec execute 46 sec latency 70.699 ms test_26 fail mds1 3 times 1 3625 0.37 MB/sec execute 47 sec latency 42.659 ms Failing mds1 on oleg123-server Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 1 3721 0.37 MB/sec execute 48 sec latency 39.020 ms 1 3729 0.36 MB/sec execute 49 sec latency 932.649 ms 05:34:18 (1786354458) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server 1 3729 0.35 MB/sec execute 50 sec latency 1932.833 ms 1 3729 0.35 MB/sec execute 51 sec latency 2933.018 ms 1 3729 0.34 MB/sec execute 52 sec latency 3933.218 ms 1 3729 0.33 MB/sec execute 53 sec latency 4934.903 ms 1 3729 0.33 MB/sec execute 54 sec latency 5935.208 ms 1 3729 0.32 MB/sec execute 55 sec latency 6935.388 ms 1 3729 0.32 MB/sec execute 56 sec latency 7935.648 ms 1 3729 0.31 MB/sec execute 57 sec latency 8935.815 ms 1 3729 0.31 MB/sec execute 58 sec latency 9936.185 ms 1 3729 0.30 MB/sec execute 59 sec latency 10936.385 ms Failover mds1 to oleg123-server mount facets: mds1 1 3729 0.30 MB/sec execute 60 sec latency 11936.601 ms 1 3729 0.29 MB/sec execute 61 sec latency 12936.776 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3729 0.29 MB/sec execute 62 sec latency 13937.205 ms 1 3729 0.28 MB/sec execute 63 sec latency 14937.445 ms oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 1 3729 0.28 MB/sec execute 64 sec latency 15938.517 ms Started lustre-MDT0000 05:34:33 (1786354473) targets are mounted 05:34:33 (1786354473) facet_failover done 1 3729 0.27 MB/sec execute 65 sec latency 16938.655 ms 1 3780 0.28 MB/sec execute 66 sec latency 17567.126 ms 1 3885 0.28 MB/sec execute 67 sec latency 43.238 ms oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3979 0.28 MB/sec execute 68 sec latency 57.842 ms 1 4079 0.27 MB/sec execute 69 sec latency 43.452 ms 1 4212 0.27 MB/sec execute 70 sec latency 36.764 ms 1 4304 0.27 MB/sec execute 71 sec latency 184.092 ms 1 4421 0.27 MB/sec execute 72 sec latency 38.813 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3578880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 58368 3586048 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 75776 7164928 2% /mnt/lustre 1 4489 0.27 MB/sec execute 73 sec latency 81.155 ms 1 4606 0.28 MB/sec execute 74 sec latency 92.941 ms 1 4742 0.28 MB/sec execute 75 sec latency 40.095 ms test_26 fail mds1 4 times 1 4896 0.30 MB/sec execute 76 sec latency 106.497 ms Failing mds1 on oleg123-server Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 1 5016 0.31 MB/sec execute 77 sec latency 296.678 ms 1 5052 0.31 MB/sec execute 78 sec latency 554.985 ms 05:34:48 (1786354488) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server 1 5052 0.30 MB/sec execute 79 sec latency 1555.147 ms 1 5052 0.30 MB/sec execute 80 sec latency 2555.324 ms 1 5052 0.30 MB/sec execute 81 sec latency 3555.570 ms 1 5052 0.29 MB/sec execute 82 sec latency 4555.766 ms 1 5052 0.29 MB/sec execute 83 sec latency 5555.956 ms 1 5052 0.29 MB/sec execute 84 sec latency 6556.134 ms 1 5052 0.28 MB/sec execute 85 sec latency 7556.323 ms 1 5052 0.28 MB/sec execute 86 sec latency 8556.509 ms 1 5052 0.28 MB/sec execute 87 sec latency 9556.695 ms 1 5052 0.27 MB/sec execute 88 sec latency 10556.877 ms Failover mds1 to oleg123-server mount facets: mds1 1 5052 0.27 MB/sec execute 89 sec latency 11557.028 ms 1 5052 0.27 MB/sec execute 90 sec latency 12557.187 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5052 0.26 MB/sec execute 91 sec latency 13557.415 ms 1 5052 0.26 MB/sec execute 92 sec latency 14557.672 ms oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all 1 5052 0.26 MB/sec execute 93 sec latency 15558.939 ms pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:03 (1786354503) targets are mounted 05:35:03 (1786354503) facet_failover done 1 5052 0.26 MB/sec execute 94 sec latency 16559.190 ms 1 5052 0.25 MB/sec execute 95 sec latency 17559.647 ms 1 5052 0.25 MB/sec execute 96 sec latency 18559.805 ms 1 5098 0.25 MB/sec execute 97 sec latency 19257.165 ms 1 5225 0.25 MB/sec execute 98 sec latency 77.249 ms oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5340 0.24 MB/sec execute 99 sec latency 44.989 ms 1 cleanup 100 sec 1 cleanup 101 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 850 110.170 20350.956 Close 634 4.481 15.770 Rename 37 33.343 61.940 Unlink 147 11.757 25.254 Qpathinfo 764 6.557 45.424 Qfileinfo 135 1.145 8.178 Qfsinfo 129 0.746 14.358 Sfileinfo 81 21.004 42.561 Find 280 2.684 29.737 WriteX 415 4.674 27.019 ReadX 1232 0.720 75.904 LockX 2 3.891 5.476 UnlockX 2 2.469 2.925 Flush 66 48.018 184.077 Throughput 0.244593 MB/sec 1 clients 1 procs max_latency=20350.967 ms stopping dbench on /mnt/lustre at Mon Aug 10 05:35:10 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4352 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 19456 3588096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 60416 3602432 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 79872 7190528 2% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Aug 10 05:35:13 EDT 2026 waiting for dbench pid 49177 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 95 4.24 MB/sec warmup 1 sec latency 177.857 ms 1 188 3.78 MB/sec warmup 2 sec latency 69.817 ms 1 297 3.65 MB/sec warmup 3 sec latency 45.460 ms test_26 fail mds1 5 times Failing mds1 on oleg123-server 1 384 3.39 MB/sec warmup 4 sec latency 61.709 ms Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 1 460 3.23 MB/sec warmup 5 sec latency 297.747 ms 05:35:19 (1786354519) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server 1 460 2.69 MB/sec warmup 6 sec latency 1297.949 ms 1 460 2.31 MB/sec warmup 7 sec latency 2298.095 ms 1 460 2.02 MB/sec warmup 8 sec latency 3298.316 ms 1 460 1.79 MB/sec warmup 9 sec latency 4298.493 ms 1 460 1.62 MB/sec warmup 10 sec latency 5298.678 ms 1 460 1.47 MB/sec warmup 11 sec latency 6298.841 ms 1 460 1.35 MB/sec warmup 12 sec latency 7299.081 ms 1 460 1.24 MB/sec warmup 13 sec latency 8299.337 ms 1 460 1.15 MB/sec warmup 14 sec latency 9299.533 ms 1 460 1.08 MB/sec warmup 15 sec latency 10299.701 ms 1 460 1.01 MB/sec warmup 16 sec latency 11299.892 ms Failover mds1 to oleg123-server mount facets: mds1 1 460 0.95 MB/sec warmup 17 sec latency 12300.078 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 460 0.90 MB/sec warmup 18 sec latency 13300.237 ms 1 460 0.85 MB/sec warmup 19 sec latency 14300.428 ms oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:34 (1786354534) targets are mounted 05:35:34 (1786354534) facet_failover done 1 460 0.00 MB/sec execute 1 sec latency 16301.894 ms 1 460 0.00 MB/sec execute 2 sec latency 17302.093 ms 1 460 0.00 MB/sec execute 3 sec latency 18302.250 ms 1 532 0.59 MB/sec execute 4 sec latency 18626.854 ms oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 673 1.00 MB/sec execute 5 sec latency 102.018 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 773 0.87 MB/sec execute 6 sec latency 67.719 ms 1 885 0.79 MB/sec execute 7 sec latency 96.625 ms 1 981 0.72 MB/sec execute 8 sec latency 74.560 ms 1 1139 0.76 MB/sec execute 9 sec latency 31.156 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 37888 3679232 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3692544 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 7371776 1% /mnt/lustre 1 1253 0.69 MB/sec execute 10 sec latency 40.571 ms 1 1440 0.90 MB/sec execute 11 sec latency 106.847 ms 1 1564 0.83 MB/sec execute 12 sec latency 113.968 ms 1 1673 0.77 MB/sec execute 13 sec latency 104.524 ms test_26 fail mds1 6 times 1 1837 0.73 MB/sec execute 14 sec latency 33.656 ms Failing mds1 on oleg123-server Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 1 1973 0.69 MB/sec execute 15 sec latency 143.625 ms 05:35:49 (1786354549) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server 1 1973 0.64 MB/sec execute 16 sec latency 1136.239 ms 1 1973 0.60 MB/sec execute 17 sec latency 2136.416 ms 1 1973 0.57 MB/sec execute 18 sec latency 3136.863 ms 1 1973 0.54 MB/sec execute 19 sec latency 4137.063 ms 1 1973 0.51 MB/sec execute 20 sec latency 5137.239 ms 1 1973 0.49 MB/sec execute 21 sec latency 6137.685 ms 1 1973 0.47 MB/sec execute 22 sec latency 7137.860 ms 1 1973 0.45 MB/sec execute 23 sec latency 8138.067 ms 1 1973 0.43 MB/sec execute 24 sec latency 9138.240 ms 1 1973 0.41 MB/sec execute 25 sec latency 10138.455 ms 1 1973 0.40 MB/sec execute 26 sec latency 11138.698 ms Failover mds1 to oleg123-server mount facets: mds1 1 1973 0.38 MB/sec execute 27 sec latency 12138.871 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1973 0.37 MB/sec execute 28 sec latency 13139.096 ms 1 1973 0.35 MB/sec execute 29 sec latency 14139.290 ms 1 1973 0.34 MB/sec execute 30 sec latency 15139.720 ms oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 05:36:04 (1786354564) targets are mounted 05:36:04 (1786354564) facet_failover done 1 1973 0.33 MB/sec execute 31 sec latency 16140.284 ms 1 1973 0.32 MB/sec execute 32 sec latency 17140.432 ms 1 1973 0.31 MB/sec execute 33 sec latency 18140.564 ms 1 2020 0.31 MB/sec execute 34 sec latency 18390.197 ms oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 2162 0.30 MB/sec execute 35 sec latency 46.850 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2301 0.32 MB/sec execute 36 sec latency 42.813 ms 1 2376 0.32 MB/sec execute 37 sec latency 250.451 ms 1 2434 0.31 MB/sec execute 38 sec latency 138.817 ms 1 2566 0.33 MB/sec execute 39 sec latency 104.581 ms 1 2589 0.32 MB/sec execute 40 sec latency 741.035 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3719168 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7455744 1% /mnt/lustre 1 2770 0.39 MB/sec execute 41 sec latency 158.397 ms 1 2934 0.41 MB/sec execute 42 sec latency 89.675 ms 1 3190 0.42 MB/sec execute 43 sec latency 75.193 ms 1 3358 0.44 MB/sec execute 44 sec latency 72.544 ms test_26 fail mds1 7 times 1 3549 0.49 MB/sec execute 45 sec latency 39.047 ms Failing mds1 on oleg123-server Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 1 3669 0.49 MB/sec execute 46 sec latency 96.444 ms 1 3669 0.48 MB/sec execute 47 sec latency 1096.811 ms 05:36:20 (1786354580) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server 1 3669 0.47 MB/sec execute 48 sec latency 2096.944 ms 1 3669 0.46 MB/sec execute 49 sec latency 3097.919 ms 1 3669 0.45 MB/sec execute 50 sec latency 4098.068 ms 1 3669 0.44 MB/sec execute 51 sec latency 5098.237 ms 1 3669 0.43 MB/sec execute 52 sec latency 6098.400 ms 1 3669 0.43 MB/sec execute 53 sec latency 7098.539 ms 1 3669 0.42 MB/sec execute 54 sec latency 8098.754 ms 1 3669 0.41 MB/sec execute 55 sec latency 9098.936 ms 1 3669 0.40 MB/sec execute 56 sec latency 10101.922 ms 1 3669 0.40 MB/sec execute 57 sec latency 11102.153 ms Failover mds1 to oleg123-server mount facets: mds1 1 3669 0.39 MB/sec execute 58 sec latency 12102.298 ms 1 3669 0.38 MB/sec execute 59 sec latency 13102.536 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3669 0.38 MB/sec execute 60 sec latency 14102.779 ms 1 3669 0.37 MB/sec execute 61 sec latency 15103.359 ms oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all 1 3669 0.36 MB/sec execute 62 sec latency 16108.219 ms pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 05:36:36 (1786354596) targets are mounted 05:36:36 (1786354596) facet_failover done 1 3669 0.36 MB/sec execute 63 sec latency 17110.291 ms 1 3669 0.35 MB/sec execute 64 sec latency 18110.441 ms 1 3691 0.35 MB/sec execute 65 sec latency 18762.635 ms 1 3805 0.36 MB/sec execute 66 sec latency 48.935 ms oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 3918 0.36 MB/sec execute 67 sec latency 36.946 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4013 0.35 MB/sec execute 68 sec latency 42.509 ms 1 4134 0.35 MB/sec execute 69 sec latency 31.843 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Aug 10 05:36:43 EDT 2026 with return code 0 1 4245 0.34 MB/sec execute 70 sec latency 168.026 ms 49177 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 49177 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished