== replay-dual test 26: dbench and tar with mds failover ========================================================== 03:42:27 (1782632547) Starting client oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre Started clients oleg219-client.virtnet: 192.168.202.119@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44824 Started dbench loop with 44825 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Jun 28 03:42:28 EDT 2026 waiting for dbench pid 44858 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 129 5.76 MB/sec warmup 1 sec latency 51.186 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3719168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 29696 7456768 1% /mnt/lustre 1 242 4.57 MB/sec warmup 2 sec latency 73.161 ms 1 373 4.43 MB/sec warmup 3 sec latency 28.383 ms 1 507 4.38 MB/sec warmup 4 sec latency 27.570 ms test_26 fail mds1 1 times Failing mds1 on oleg219-server 1 666 4.23 MB/sec warmup 5 sec latency 119.534 ms Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 1 700 3.53 MB/sec warmup 6 sec latency 708.314 ms 03:42:35 (1782632555) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server 1 700 3.03 MB/sec warmup 7 sec latency 1708.469 ms 1 700 2.65 MB/sec warmup 8 sec latency 2708.700 ms 1 700 2.36 MB/sec warmup 9 sec latency 3708.972 ms 1 700 2.12 MB/sec warmup 10 sec latency 4709.110 ms 1 700 1.93 MB/sec warmup 11 sec latency 5709.316 ms 1 700 1.77 MB/sec warmup 12 sec latency 6709.478 ms 1 700 1.63 MB/sec warmup 13 sec latency 7709.626 ms 1 700 1.51 MB/sec warmup 14 sec latency 8709.757 ms 1 700 1.41 MB/sec warmup 15 sec latency 9709.890 ms 1 700 1.33 MB/sec warmup 16 sec latency 10710.137 ms Failover mds1 to oleg219-server mount facets: mds1 1 700 1.25 MB/sec warmup 17 sec latency 11710.287 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 700 1.18 MB/sec warmup 18 sec latency 12710.462 ms 1 700 1.12 MB/sec warmup 19 sec latency 13710.613 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 03:42:49 (1782632569) targets are mounted 03:42:49 (1782632569) facet_failover done 1 700 0.00 MB/sec execute 1 sec latency 15710.977 ms 1 700 0.00 MB/sec execute 2 sec latency 16711.171 ms 1 700 0.00 MB/sec execute 3 sec latency 17711.347 ms 1 730 0.03 MB/sec execute 4 sec latency 18272.417 ms 1 864 0.07 MB/sec execute 5 sec latency 71.041 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 970 0.10 MB/sec execute 6 sec latency 74.510 ms 1 1077 0.25 MB/sec execute 7 sec latency 70.140 ms 1 1232 0.23 MB/sec execute 8 sec latency 28.544 ms 1 1426 0.53 MB/sec execute 9 sec latency 39.602 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 46080 3599360 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 9216 3647488 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7246848 1% /mnt/lustre 1 1558 0.49 MB/sec execute 10 sec latency 121.603 ms 1 1677 0.46 MB/sec execute 11 sec latency 35.607 ms 1 1843 0.43 MB/sec execute 12 sec latency 35.633 ms test_26 fail mds1 2 times 1 2006 0.41 MB/sec execute 13 sec latency 74.155 ms Failing mds1 on oleg219-server Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 1 2160 0.39 MB/sec execute 14 sec latency 40.706 ms 1 2168 0.37 MB/sec execute 15 sec latency 920.197 ms 03:43:04 (1782632584) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server 1 2168 0.35 MB/sec execute 16 sec latency 1920.418 ms 1 2168 0.33 MB/sec execute 17 sec latency 2920.585 ms 1 2168 0.31 MB/sec execute 18 sec latency 3920.743 ms 1 2168 0.29 MB/sec execute 19 sec latency 4920.916 ms 1 2168 0.28 MB/sec execute 20 sec latency 5921.127 ms 1 2168 0.26 MB/sec execute 21 sec latency 6921.301 ms 1 2168 0.25 MB/sec execute 22 sec latency 7921.467 ms 1 2168 0.24 MB/sec execute 23 sec latency 8921.614 ms 1 2168 0.23 MB/sec execute 24 sec latency 9921.791 ms 1 2168 0.22 MB/sec execute 25 sec latency 10922.041 ms Failover mds1 to oleg219-server mount facets: mds1 1 2168 0.21 MB/sec execute 26 sec latency 11922.205 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2168 0.20 MB/sec execute 27 sec latency 12922.472 ms 1 2168 0.20 MB/sec execute 28 sec latency 13922.684 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 1 2168 0.19 MB/sec execute 29 sec latency 14922.834 ms pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 03:43:18 (1782632598) targets are mounted 03:43:18 (1782632598) facet_failover done 1 2168 0.18 MB/sec execute 30 sec latency 15923.076 ms 1 2168 0.18 MB/sec execute 31 sec latency 16923.243 ms 1 2168 0.17 MB/sec execute 32 sec latency 17923.382 ms 1 2168 0.17 MB/sec execute 33 sec latency 18923.500 ms 1 2251 0.16 MB/sec execute 34 sec latency 19529.751 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2416 0.20 MB/sec execute 35 sec latency 103.642 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2566 0.21 MB/sec execute 36 sec latency 216.250 ms 1 2717 0.29 MB/sec execute 37 sec latency 702.792 ms 1 2937 0.32 MB/sec execute 38 sec latency 76.057 ms 1 3261 0.36 MB/sec execute 39 sec latency 50.838 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 50176 3542016 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3612672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 74752 7154688 2% /mnt/lustre 1 3537 0.43 MB/sec execute 40 sec latency 28.263 ms 1 3693 0.43 MB/sec execute 41 sec latency 45.454 ms 1 3812 0.44 MB/sec execute 42 sec latency 34.707 ms test_26 fail mds1 3 times 1 3946 0.44 MB/sec execute 43 sec latency 29.433 ms Failing mds1 on oleg219-server Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 1 4079 0.43 MB/sec execute 44 sec latency 54.019 ms 1 4093 0.42 MB/sec execute 45 sec latency 931.919 ms 03:43:34 (1782632614) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server 1 4093 0.41 MB/sec execute 46 sec latency 1932.114 ms 1 4093 0.40 MB/sec execute 47 sec latency 2932.348 ms 1 4093 0.39 MB/sec execute 48 sec latency 3932.590 ms 1 4093 0.39 MB/sec execute 49 sec latency 4932.783 ms 1 4093 0.38 MB/sec execute 50 sec latency 5932.938 ms 1 4093 0.37 MB/sec execute 51 sec latency 6933.120 ms 1 4093 0.36 MB/sec execute 52 sec latency 7933.345 ms 1 4093 0.36 MB/sec execute 53 sec latency 8933.547 ms 1 4093 0.35 MB/sec execute 54 sec latency 9933.811 ms 1 4093 0.34 MB/sec execute 55 sec latency 10934.004 ms Failover mds1 to oleg219-server mount facets: mds1 1 4093 0.34 MB/sec execute 56 sec latency 11934.164 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4093 0.33 MB/sec execute 57 sec latency 12934.334 ms 1 4093 0.33 MB/sec execute 58 sec latency 13934.494 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 1 4093 0.32 MB/sec execute 59 sec latency 14934.630 ms Started lustre-MDT0000 03:43:48 (1782632628) targets are mounted 03:43:48 (1782632628) facet_failover done 1 4093 0.31 MB/sec execute 60 sec latency 15934.783 ms 1 4093 0.31 MB/sec execute 61 sec latency 16934.960 ms 1 4093 0.30 MB/sec execute 62 sec latency 17935.119 ms 1 4093 0.30 MB/sec execute 63 sec latency 18935.243 ms 1 4107 0.30 MB/sec execute 64 sec latency 19694.866 ms 1 4226 0.29 MB/sec execute 65 sec latency 70.978 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4302 0.29 MB/sec execute 66 sec latency 186.491 ms 1 4419 0.29 MB/sec execute 67 sec latency 83.314 ms 1 4524 0.29 MB/sec execute 68 sec latency 61.822 ms 1 4654 0.30 MB/sec execute 69 sec latency 45.760 ms 1 4758 0.30 MB/sec execute 70 sec latency 48.156 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 52224 3701760 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 27648 3704832 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 79872 7406592 2% /mnt/lustre 1 4840 0.30 MB/sec execute 71 sec latency 53.202 ms 1 4972 0.33 MB/sec execute 72 sec latency 416.020 ms 1 5047 0.33 MB/sec execute 73 sec latency 84.696 ms test_26 fail mds1 4 times 1 5129 0.32 MB/sec execute 74 sec latency 105.649 ms Failing mds1 on oleg219-server Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 1 5251 0.32 MB/sec execute 75 sec latency 92.605 ms 1 5260 0.32 MB/sec execute 76 sec latency 893.152 ms 03:44:05 (1782632645) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server 1 5260 0.31 MB/sec execute 77 sec latency 1893.335 ms 1 5260 0.31 MB/sec execute 78 sec latency 2893.559 ms 1 5260 0.31 MB/sec execute 79 sec latency 3893.840 ms 1 5260 0.30 MB/sec execute 80 sec latency 4894.088 ms 1 5260 0.30 MB/sec execute 81 sec latency 5894.342 ms 1 5260 0.29 MB/sec execute 82 sec latency 6894.579 ms 1 5260 0.29 MB/sec execute 83 sec latency 7894.732 ms 1 5260 0.29 MB/sec execute 84 sec latency 8894.926 ms 1 5260 0.28 MB/sec execute 85 sec latency 9895.186 ms 1 5260 0.28 MB/sec execute 86 sec latency 10895.440 ms Failover mds1 to oleg219-server mount facets: mds1 1 5260 0.28 MB/sec execute 87 sec latency 11895.614 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5260 0.27 MB/sec execute 88 sec latency 12895.758 ms 1 5260 0.27 MB/sec execute 89 sec latency 13895.902 ms 1 5260 0.27 MB/sec execute 90 sec latency 14896.036 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 03:44:19 (1782632659) targets are mounted 03:44:19 (1782632659) facet_failover done 1 5260 0.27 MB/sec execute 91 sec latency 15896.332 ms 1 5260 0.26 MB/sec execute 92 sec latency 16896.540 ms 1 5260 0.26 MB/sec execute 93 sec latency 17896.690 ms 1 5262 0.26 MB/sec execute 94 sec latency 18814.480 ms 1 5362 0.26 MB/sec execute 95 sec latency 36.631 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5453 0.25 MB/sec execute 96 sec latency 91.844 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5589 0.25 MB/sec execute 97 sec latency 49.052 ms 1 5765 0.25 MB/sec execute 98 sec latency 27.528 ms 1 5946 0.26 MB/sec execute 99 sec latency 164.450 ms 1 cleanup 100 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4608 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 52224 3653632 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 27648 3679232 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 79872 7332864 2% /mnt/lustre 1 cleanup 101 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 944 39.194 19529.737 Close 735 53.486 18814.458 Rename 41 29.380 53.152 Unlink 161 133.815 19694.850 Qpathinfo 904 6.007 63.814 Qfileinfo 158 1.346 11.842 Qfsinfo 136 0.644 4.116 Sfileinfo 94 18.480 38.277 Find 323 3.374 142.243 WriteX 490 4.225 20.761 ReadX 1361 0.546 70.124 LockX 2 3.128 3.335 UnlockX 2 3.377 4.307 Flush 71 44.106 186.479 Throughput 0.261377 MB/sec 1 clients 1 procs max_latency=19694.866 ms stopping dbench on /mnt/lustre at Sun Jun 28 03:44:30 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Jun 28 03:44:32 EDT 2026 waiting for dbench pid 48897 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients test_26 fail mds1 5 times Failing mds1 on oleg219-server 1 139 6.02 MB/sec warmup 1 sec latency 50.629 ms Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 1 196 3.92 MB/sec warmup 2 sec latency 450.269 ms 03:44:34 (1782632674) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server 1 196 2.61 MB/sec warmup 3 sec latency 1450.424 ms 1 196 1.96 MB/sec warmup 4 sec latency 2450.608 ms 1 196 1.57 MB/sec warmup 5 sec latency 3450.858 ms 1 196 1.31 MB/sec warmup 6 sec latency 4451.057 ms 1 196 1.12 MB/sec warmup 7 sec latency 5451.340 ms 1 196 0.98 MB/sec warmup 8 sec latency 6451.520 ms 1 196 0.87 MB/sec warmup 9 sec latency 7451.745 ms 1 196 0.78 MB/sec warmup 10 sec latency 8451.950 ms 1 196 0.71 MB/sec warmup 11 sec latency 9452.236 ms 1 196 0.65 MB/sec warmup 12 sec latency 10452.479 ms Failover mds1 to oleg219-server mount facets: mds1 1 196 0.60 MB/sec warmup 13 sec latency 11452.659 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 196 0.56 MB/sec warmup 14 sec latency 12452.783 ms 1 196 0.52 MB/sec warmup 15 sec latency 13452.894 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 1 196 0.49 MB/sec warmup 16 sec latency 14453.277 ms pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 03:44:48 (1782632688) targets are mounted 03:44:48 (1782632688) facet_failover done 1 196 0.46 MB/sec warmup 17 sec latency 15453.390 ms 1 196 0.44 MB/sec warmup 18 sec latency 16453.562 ms 1 196 0.41 MB/sec warmup 19 sec latency 17453.695 ms 1 196 0.00 MB/sec execute 1 sec latency 19453.998 ms 1 271 1.17 MB/sec execute 2 sec latency 19780.664 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 381 1.91 MB/sec execute 3 sec latency 43.295 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 494 2.34 MB/sec execute 4 sec latency 37.658 ms 1 636 2.64 MB/sec execute 5 sec latency 46.344 ms 1 721 2.24 MB/sec execute 6 sec latency 111.687 ms 1 860 1.96 MB/sec execute 7 sec latency 86.262 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3706880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3681280 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 61440 7388160 1% /mnt/lustre 1 972 1.75 MB/sec execute 8 sec latency 67.712 ms 1 1132 1.68 MB/sec execute 9 sec latency 71.035 ms 1 1278 1.53 MB/sec execute 10 sec latency 28.736 ms 1 1523 1.66 MB/sec execute 11 sec latency 97.917 ms test_26 fail mds1 6 times Failing mds1 on oleg219-server Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 1 1691 1.53 MB/sec execute 12 sec latency 126.764 ms 1 1762 1.42 MB/sec execute 13 sec latency 728.292 ms 03:45:05 (1782632705) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server 1 1762 1.32 MB/sec execute 14 sec latency 1728.445 ms 1 1762 1.23 MB/sec execute 15 sec latency 2728.572 ms 1 1762 1.15 MB/sec execute 16 sec latency 3728.736 ms 1 1762 1.08 MB/sec execute 17 sec latency 4728.923 ms 1 1762 1.02 MB/sec execute 18 sec latency 5729.053 ms 1 1762 0.97 MB/sec execute 19 sec latency 6729.181 ms 1 1762 0.92 MB/sec execute 20 sec latency 7729.340 ms 1 1762 0.88 MB/sec execute 21 sec latency 8729.484 ms 1 1762 0.84 MB/sec execute 22 sec latency 9729.635 ms 1 1762 0.80 MB/sec execute 23 sec latency 10729.762 ms Failover mds1 to oleg219-server mount facets: mds1 1 1762 0.77 MB/sec execute 24 sec latency 11729.940 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1762 0.74 MB/sec execute 25 sec latency 12730.103 ms 1 1762 0.71 MB/sec execute 26 sec latency 13730.232 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 1 1762 0.68 MB/sec execute 27 sec latency 14730.337 ms Started lustre-MDT0000 03:45:19 (1782632719) targets are mounted 03:45:19 (1782632719) facet_failover done 1 1762 0.66 MB/sec execute 28 sec latency 15730.535 ms 1 1762 0.64 MB/sec execute 29 sec latency 16730.756 ms 1 1762 0.61 MB/sec execute 30 sec latency 17730.999 ms 1 1762 0.59 MB/sec execute 31 sec latency 18731.136 ms 1 1884 0.58 MB/sec execute 32 sec latency 18871.980 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2083 0.57 MB/sec execute 33 sec latency 107.136 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2299 0.59 MB/sec execute 34 sec latency 26.042 ms 1 2439 0.58 MB/sec execute 35 sec latency 53.944 ms 1 2566 0.59 MB/sec execute 36 sec latency 363.622 ms 1 2566 0.57 MB/sec execute 37 sec latency 1363.783 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3717120 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3589120 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7306240 1% /mnt/lustre 1 2577 0.56 MB/sec execute 38 sec latency 2127.335 ms 1 2775 0.62 MB/sec execute 39 sec latency 53.409 ms 1 2935 0.64 MB/sec execute 40 sec latency 59.800 ms 1 3271 0.67 MB/sec execute 41 sec latency 33.544 ms test_26 fail mds1 7 times Failing mds1 on oleg219-server Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 1 3541 0.73 MB/sec execute 42 sec latency 32.357 ms 1 3586 0.71 MB/sec execute 43 sec latency 735.133 ms 03:45:35 (1782632735) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server 1 3586 0.70 MB/sec execute 44 sec latency 1735.394 ms 1 3586 0.68 MB/sec execute 45 sec latency 2735.597 ms 1 3586 0.66 MB/sec execute 46 sec latency 3735.802 ms 1 3586 0.65 MB/sec execute 47 sec latency 4735.977 ms 1 3586 0.64 MB/sec execute 48 sec latency 5736.126 ms 1 3586 0.62 MB/sec execute 49 sec latency 6736.343 ms 1 3586 0.61 MB/sec execute 50 sec latency 7736.524 ms 1 3586 0.60 MB/sec execute 51 sec latency 8736.717 ms 1 3586 0.59 MB/sec execute 52 sec latency 9736.975 ms 1 3586 0.58 MB/sec execute 53 sec latency 10737.130 ms Failover mds1 to oleg219-server mount facets: mds1 1 3586 0.57 MB/sec execute 54 sec latency 11737.343 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3586 0.56 MB/sec execute 55 sec latency 12737.580 ms 1 3586 0.55 MB/sec execute 56 sec latency 13737.750 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 1 3586 0.54 MB/sec execute 57 sec latency 14737.904 ms pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 03:45:49 (1782632749) targets are mounted 03:45:49 (1782632749) facet_failover done 1 3586 0.53 MB/sec execute 58 sec latency 15738.059 ms 1 3586 0.52 MB/sec execute 59 sec latency 16738.215 ms 1 3586 0.51 MB/sec execute 60 sec latency 17738.364 ms 1 3586 0.50 MB/sec execute 61 sec latency 18738.594 ms 1 3587 0.49 MB/sec execute 62 sec latency 19685.389 ms 1 3767 0.51 MB/sec execute 63 sec latency 57.000 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3901 0.50 MB/sec execute 64 sec latency 24.655 ms 1 4010 0.50 MB/sec execute 65 sec latency 40.998 ms 1 4133 0.49 MB/sec execute 66 sec latency 50.255 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 4257 0.48 MB/sec execute 67 sec latency 153.764 ms 1 4360 0.48 MB/sec execute 68 sec latency 81.628 ms 1 4475 0.48 MB/sec execute 69 sec latency 95.459 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sun Jun 28 03:46:02 EDT 2026 with return code 0 1 4578 0.48 MB/sec execute 70 sec latency 91.238 ms 48897 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48897 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished