== replay-dual test 26: dbench and tar with mds failover ========================================================== 01:24:22 (1780809862) Starting client oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre Started clients oleg410-client.virtnet: 192.168.204.110@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44457 Started dbench loop with 44458 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Jun 7 01:24:24 EDT 2026 waiting for dbench pid 44490 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3727360 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 27648 3721216 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 29696 7448576 1% /mnt/lustre 1 97 4.39 MB/sec warmup 1 sec latency 44.844 ms 1 189 3.79 MB/sec warmup 2 sec latency 66.470 ms 1 289 3.57 MB/sec warmup 3 sec latency 38.763 ms 1 424 3.72 MB/sec warmup 4 sec latency 40.540 ms test_26 fail mds1 1 times Failing mds1 on oleg410-server 1 539 3.79 MB/sec warmup 5 sec latency 39.169 ms Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 1 603 3.49 MB/sec warmup 6 sec latency 574.891 ms 01:24:31 (1780809871) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server 1 603 3.00 MB/sec warmup 7 sec latency 1575.065 ms 1 603 2.62 MB/sec warmup 8 sec latency 2575.280 ms 1 603 2.33 MB/sec warmup 9 sec latency 3575.518 ms 1 603 2.10 MB/sec warmup 10 sec latency 4575.714 ms 1 603 1.91 MB/sec warmup 11 sec latency 5575.946 ms 1 603 1.75 MB/sec warmup 12 sec latency 6576.149 ms 1 603 1.61 MB/sec warmup 13 sec latency 7576.388 ms 1 603 1.50 MB/sec warmup 14 sec latency 8576.576 ms 1 603 1.40 MB/sec warmup 15 sec latency 9576.705 ms 1 603 1.31 MB/sec warmup 16 sec latency 10576.845 ms Failover mds1 to oleg410-server mount facets: mds1 1 603 1.23 MB/sec warmup 17 sec latency 11576.990 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 603 1.16 MB/sec warmup 18 sec latency 12577.150 ms 1 603 1.10 MB/sec warmup 19 sec latency 13577.308 ms oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 01:24:45 (1780809885) targets are mounted 01:24:45 (1780809885) facet_failover done 1 603 0.00 MB/sec execute 1 sec latency 15578.657 ms 1 603 0.00 MB/sec execute 2 sec latency 16578.798 ms 1 603 0.00 MB/sec execute 3 sec latency 17578.945 ms 1 603 0.00 MB/sec execute 4 sec latency 18579.105 ms 1 699 0.05 MB/sec execute 5 sec latency 18945.231 ms oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 799 0.08 MB/sec execute 6 sec latency 84.671 ms 1 914 0.11 MB/sec execute 7 sec latency 49.406 ms 1 1002 0.14 MB/sec execute 8 sec latency 69.154 ms 1 1132 0.22 MB/sec execute 9 sec latency 85.572 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 9216 3671040 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 48128 3589120 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7260160 1% /mnt/lustre 1 1252 0.21 MB/sec execute 10 sec latency 34.006 ms 1 1449 0.46 MB/sec execute 11 sec latency 47.263 ms 1 1580 0.43 MB/sec execute 12 sec latency 52.497 ms 1 1719 0.41 MB/sec execute 13 sec latency 65.944 ms test_26 fail mds1 2 times Failing mds1 on oleg410-server 1 1886 0.39 MB/sec execute 14 sec latency 87.306 ms Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 1 1974 0.37 MB/sec execute 15 sec latency 569.966 ms 01:25:00 (1780809900) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server 1 1974 0.34 MB/sec execute 16 sec latency 1570.112 ms 1 1974 0.32 MB/sec execute 17 sec latency 2570.259 ms 1 1974 0.30 MB/sec execute 18 sec latency 3570.400 ms 1 1974 0.29 MB/sec execute 19 sec latency 4570.558 ms 1 1974 0.27 MB/sec execute 20 sec latency 5570.759 ms 1 1974 0.26 MB/sec execute 21 sec latency 6570.904 ms 1 1974 0.25 MB/sec execute 22 sec latency 7571.036 ms 1 1974 0.24 MB/sec execute 23 sec latency 8571.190 ms 1 1974 0.23 MB/sec execute 24 sec latency 9571.317 ms 1 1974 0.22 MB/sec execute 25 sec latency 10571.464 ms Failover mds1 to oleg410-server mount facets: mds1 1 1974 0.21 MB/sec execute 26 sec latency 11571.682 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1974 0.20 MB/sec execute 27 sec latency 12571.808 ms 1 1974 0.20 MB/sec execute 28 sec latency 13571.911 ms oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all 1 1974 0.19 MB/sec execute 29 sec latency 14572.031 ms pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 01:25:14 (1780809914) targets are mounted 01:25:14 (1780809914) facet_failover done 1 1974 0.18 MB/sec execute 30 sec latency 15572.224 ms 1 1974 0.18 MB/sec execute 31 sec latency 16572.486 ms 1 1974 0.17 MB/sec execute 32 sec latency 17572.691 ms 1 1974 0.17 MB/sec execute 33 sec latency 18572.832 ms 1 1975 0.16 MB/sec execute 34 sec latency 19563.205 ms oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2112 0.16 MB/sec execute 35 sec latency 42.977 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2291 0.19 MB/sec execute 36 sec latency 31.066 ms 1 2426 0.19 MB/sec execute 37 sec latency 80.605 ms 1 2566 0.21 MB/sec execute 38 sec latency 241.880 ms 1 2724 0.28 MB/sec execute 39 sec latency 725.445 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3631104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 59392 3539968 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7171072 2% /mnt/lustre 1 2932 0.31 MB/sec execute 40 sec latency 71.629 ms 1 3235 0.33 MB/sec execute 41 sec latency 70.755 ms 1 3530 0.41 MB/sec execute 42 sec latency 23.569 ms test_26 fail mds1 3 times 1 3697 0.41 MB/sec execute 43 sec latency 24.422 ms Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 1 3843 0.43 MB/sec execute 44 sec latency 28.495 ms 1 3856 0.42 MB/sec execute 45 sec latency 914.870 ms 01:25:29 (1780809929) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server 1 3856 0.41 MB/sec execute 46 sec latency 1914.998 ms 1 3856 0.40 MB/sec execute 47 sec latency 2915.130 ms 1 3856 0.39 MB/sec execute 48 sec latency 3915.295 ms 1 3856 0.38 MB/sec execute 49 sec latency 4915.451 ms 1 3856 0.38 MB/sec execute 50 sec latency 5915.586 ms 1 3856 0.37 MB/sec execute 51 sec latency 6915.753 ms 1 3856 0.36 MB/sec execute 52 sec latency 7915.911 ms 1 3856 0.35 MB/sec execute 53 sec latency 8916.066 ms 1 3856 0.35 MB/sec execute 54 sec latency 9916.310 ms 1 3856 0.34 MB/sec execute 55 sec latency 10916.515 ms Failover mds1 to oleg410-server mount facets: mds1 1 3856 0.34 MB/sec execute 56 sec latency 11916.744 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3856 0.33 MB/sec execute 57 sec latency 12916.947 ms 1 3856 0.32 MB/sec execute 58 sec latency 13917.159 ms oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 1 3856 0.32 MB/sec execute 59 sec latency 14917.321 ms Started lustre-MDT0000 01:25:43 (1780809943) targets are mounted 01:25:43 (1780809943) facet_failover done 1 3856 0.31 MB/sec execute 60 sec latency 15917.522 ms 1 3856 0.31 MB/sec execute 61 sec latency 16917.715 ms 1 3856 0.30 MB/sec execute 62 sec latency 17917.912 ms 1 3856 0.30 MB/sec execute 63 sec latency 18918.085 ms 1 3856 0.29 MB/sec execute 64 sec latency 19918.220 ms oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3945 0.29 MB/sec execute 65 sec latency 20229.123 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4067 0.29 MB/sec execute 66 sec latency 30.337 ms 1 4210 0.29 MB/sec execute 67 sec latency 27.219 ms 1 4299 0.29 MB/sec execute 68 sec latency 100.022 ms 1 4391 0.28 MB/sec execute 69 sec latency 55.777 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 62464 3704832 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 79872 7450624 2% /mnt/lustre 1 4493 0.28 MB/sec execute 70 sec latency 102.694 ms 1 4629 0.30 MB/sec execute 71 sec latency 80.861 ms 1 4759 0.29 MB/sec execute 72 sec latency 47.545 ms 1 4944 0.33 MB/sec execute 73 sec latency 301.831 ms test_26 fail mds1 4 times Failing mds1 on oleg410-server 1 5052 0.33 MB/sec execute 74 sec latency 152.373 ms Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 1 5129 0.32 MB/sec execute 75 sec latency 519.901 ms 01:26:00 (1780809960) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server 1 5129 0.32 MB/sec execute 76 sec latency 1520.027 ms 1 5129 0.32 MB/sec execute 77 sec latency 2520.205 ms 1 5129 0.31 MB/sec execute 78 sec latency 3520.344 ms 1 5129 0.31 MB/sec execute 79 sec latency 4520.477 ms 1 5129 0.30 MB/sec execute 80 sec latency 5520.693 ms 1 5129 0.30 MB/sec execute 81 sec latency 6520.855 ms 1 5129 0.30 MB/sec execute 82 sec latency 7521.078 ms 1 5129 0.29 MB/sec execute 83 sec latency 8521.354 ms 1 5129 0.29 MB/sec execute 84 sec latency 9521.668 ms 1 5129 0.29 MB/sec execute 85 sec latency 10521.913 ms Failover mds1 to oleg410-server mount facets: mds1 1 5129 0.28 MB/sec execute 86 sec latency 11522.071 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5129 0.28 MB/sec execute 87 sec latency 12522.208 ms 1 5129 0.28 MB/sec execute 88 sec latency 13522.331 ms oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all 1 5129 0.27 MB/sec execute 89 sec latency 14522.551 ms pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 01:26:14 (1780809974) targets are mounted 01:26:14 (1780809974) facet_failover done 1 5129 0.27 MB/sec execute 90 sec latency 15522.747 ms 1 5129 0.27 MB/sec execute 91 sec latency 16522.941 ms 1 5129 0.26 MB/sec execute 92 sec latency 17523.133 ms 1 5129 0.26 MB/sec execute 93 sec latency 18523.353 ms 1 5129 0.26 MB/sec execute 94 sec latency 19523.509 ms 1 5190 0.26 MB/sec execute 95 sec latency 20058.209 ms oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5334 0.25 MB/sec execute 96 sec latency 53.261 ms 1 5453 0.25 MB/sec execute 97 sec latency 109.884 ms 1 5631 0.25 MB/sec execute 98 sec latency 36.195 ms 1 5819 0.25 MB/sec execute 99 sec latency 27.748 ms 1 cleanup 100 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4608 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3738624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 62464 3685376 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 79872 7424000 2% /mnt/lustre 1 cleanup 101 sec 0 cleanup 102 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 941 80.864 20229.108 Close 732 2.849 24.267 Rename 39 28.032 47.523 Unlink 157 10.600 23.210 Qpathinfo 887 5.403 41.495 Qfileinfo 153 1.265 5.669 Qfsinfo 139 0.678 6.529 Sfileinfo 91 18.380 38.021 Find 321 62.097 18945.219 WriteX 476 4.299 15.791 ReadX 1343 0.629 85.558 LockX 2 2.993 3.328 UnlockX 2 2.497 2.547 Flush 75 40.470 152.357 Throughput 0.251843 MB/sec 1 clients 1 procs max_latency=20229.123 ms stopping dbench on /mnt/lustre at Sun Jun 7 01:26:26 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt test_26 fail mds1 5 times '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Jun 7 01:26:28 EDT 2026 waiting for dbench pid 48531 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 1 110 5.24 MB/sec warmup 1 sec latency 176.870 ms 01:26:30 (1780809990) shut down 1 110 2.62 MB/sec warmup 2 sec latency 1177.018 ms facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server 1 110 1.75 MB/sec warmup 3 sec latency 2177.203 ms 1 110 1.31 MB/sec warmup 4 sec latency 3177.424 ms 1 110 1.05 MB/sec warmup 5 sec latency 4177.653 ms 1 110 0.87 MB/sec warmup 6 sec latency 5177.884 ms 1 110 0.75 MB/sec warmup 7 sec latency 6178.096 ms 1 110 0.66 MB/sec warmup 8 sec latency 7178.281 ms 1 110 0.58 MB/sec warmup 9 sec latency 8178.487 ms 1 110 0.52 MB/sec warmup 10 sec latency 9178.717 ms 1 110 0.48 MB/sec warmup 11 sec latency 10178.939 ms 1 110 0.44 MB/sec warmup 12 sec latency 11179.103 ms Failover mds1 to oleg410-server mount facets: mds1 1 110 0.40 MB/sec warmup 13 sec latency 12179.254 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 110 0.37 MB/sec warmup 14 sec latency 13179.386 ms 1 110 0.35 MB/sec warmup 15 sec latency 14179.577 ms oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 1 110 0.33 MB/sec warmup 16 sec latency 15179.745 ms Started lustre-MDT0000 01:26:44 (1780810004) targets are mounted 01:26:44 (1780810004) facet_failover done 1 110 0.31 MB/sec warmup 17 sec latency 16179.973 ms 1 110 0.29 MB/sec warmup 18 sec latency 17180.203 ms 1 110 0.28 MB/sec warmup 19 sec latency 18180.448 ms 1 120 0.26 MB/sec execute 1 sec latency 20030.226 ms oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 243 1.98 MB/sec execute 2 sec latency 53.736 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 366 2.60 MB/sec execute 3 sec latency 37.397 ms 1 498 3.02 MB/sec execute 4 sec latency 41.907 ms 1 664 3.18 MB/sec execute 5 sec latency 52.486 ms 1 710 2.66 MB/sec execute 6 sec latency 273.692 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3687424 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 59392 3656704 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 99328 7344128 2% /mnt/lustre 1 803 2.32 MB/sec execute 7 sec latency 78.306 ms 1 921 2.07 MB/sec execute 8 sec latency 69.116 ms 1 1038 1.96 MB/sec execute 9 sec latency 99.574 ms test_26 fail mds1 6 times 1 1201 1.78 MB/sec execute 10 sec latency 34.789 ms Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 1 1421 1.89 MB/sec execute 11 sec latency 39.321 ms 01:27:00 (1780810020) shut down 1 1428 1.73 MB/sec execute 12 sec latency 956.513 ms facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server 1 1428 1.60 MB/sec execute 13 sec latency 1956.753 ms 1 1428 1.48 MB/sec execute 14 sec latency 2956.958 ms 1 1428 1.38 MB/sec execute 15 sec latency 3957.140 ms 1 1428 1.30 MB/sec execute 16 sec latency 4957.303 ms 1 1428 1.22 MB/sec execute 17 sec latency 5957.463 ms 1 1428 1.15 MB/sec execute 18 sec latency 6957.644 ms 1 1428 1.09 MB/sec execute 19 sec latency 7957.843 ms 1 1428 1.04 MB/sec execute 20 sec latency 8958.082 ms 1 1428 0.99 MB/sec execute 21 sec latency 9958.327 ms 1 1428 0.94 MB/sec execute 22 sec latency 10958.518 ms Failover mds1 to oleg410-server mount facets: mds1 1 1428 0.90 MB/sec execute 23 sec latency 11958.652 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1428 0.87 MB/sec execute 24 sec latency 12958.824 ms 1 1428 0.83 MB/sec execute 25 sec latency 13958.994 ms oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 01:27:14 (1780810034) targets are mounted 01:27:14 (1780810034) facet_failover done 1 1428 0.80 MB/sec execute 26 sec latency 14959.176 ms 1 1428 0.77 MB/sec execute 27 sec latency 15959.325 ms 1 1428 0.74 MB/sec execute 28 sec latency 16959.485 ms 1 1428 0.72 MB/sec execute 29 sec latency 17959.661 ms 1 1428 0.69 MB/sec execute 30 sec latency 18959.802 ms 1 1481 0.67 MB/sec execute 31 sec latency 19544.226 ms oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1591 0.65 MB/sec execute 32 sec latency 138.448 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1709 0.64 MB/sec execute 33 sec latency 56.570 ms 1 1884 0.62 MB/sec execute 34 sec latency 71.192 ms 1 2039 0.61 MB/sec execute 35 sec latency 108.619 ms 1 2225 0.60 MB/sec execute 36 sec latency 30.186 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3712000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 11264 3729408 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 48128 7441408 1% /mnt/lustre 1 2386 0.61 MB/sec execute 37 sec latency 113.147 ms 1 2494 0.62 MB/sec execute 38 sec latency 57.106 ms 1 2566 0.61 MB/sec execute 39 sec latency 835.874 ms test_26 fail mds1 7 times 1 2764 0.67 MB/sec execute 40 sec latency 1021.643 ms Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 1 2922 0.69 MB/sec execute 41 sec latency 67.747 ms 1 2932 0.67 MB/sec execute 42 sec latency 847.363 ms 01:27:30 (1780810050) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server 1 2932 0.66 MB/sec execute 43 sec latency 1847.579 ms 1 2932 0.64 MB/sec execute 44 sec latency 2847.708 ms 1 2932 0.63 MB/sec execute 45 sec latency 3848.002 ms 1 2932 0.61 MB/sec execute 46 sec latency 4848.201 ms 1 2932 0.60 MB/sec execute 47 sec latency 5848.397 ms 1 2932 0.59 MB/sec execute 48 sec latency 6848.584 ms 1 2932 0.58 MB/sec execute 49 sec latency 7848.752 ms 1 2932 0.56 MB/sec execute 50 sec latency 8848.941 ms 1 2932 0.55 MB/sec execute 51 sec latency 9849.138 ms 1 2932 0.54 MB/sec execute 52 sec latency 10849.377 ms Failover mds1 to oleg410-server mount facets: mds1 1 2932 0.53 MB/sec execute 53 sec latency 11849.571 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2932 0.52 MB/sec execute 54 sec latency 12849.785 ms 1 2932 0.51 MB/sec execute 55 sec latency 13849.955 ms oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 1 2932 0.50 MB/sec execute 56 sec latency 14850.134 ms Started lustre-MDT0000 01:27:44 (1780810064) targets are mounted 01:27:44 (1780810064) facet_failover done 1 2932 0.50 MB/sec execute 57 sec latency 15850.337 ms 1 2932 0.49 MB/sec execute 58 sec latency 16850.568 ms 1 2932 0.48 MB/sec execute 59 sec latency 17850.772 ms 1 2932 0.47 MB/sec execute 60 sec latency 18850.937 ms oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3199 0.48 MB/sec execute 61 sec latency 18922.312 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3380 0.49 MB/sec execute 62 sec latency 42.906 ms 1 3673 0.53 MB/sec execute 63 sec latency 27.159 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 3805 0.54 MB/sec execute 64 sec latency 33.570 ms 1 3920 0.54 MB/sec execute 65 sec latency 35.458 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sun Jun 7 01:27:54 EDT 2026 with return code 0 1 4039 0.53 MB/sec execute 66 sec latency 30.699 ms 48531 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48531 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished