== replay-dual test 26: dbench and tar with mds failover ========================================================== 11:51:36 (1777045896) Starting client oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 64689 Started dbench loop with 64690 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Apr 24 11:51:38 EDT 2026 waiting for dbench pid 64733 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 139 6.02 MB/sec warmup 1 sec latency 35.486 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1996 1269308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26120 3579592 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27684 7185048 1% /mnt/lustre 1 243 4.60 MB/sec warmup 2 sec latency 180.966 ms 1 397 4.69 MB/sec warmup 3 sec latency 28.611 ms 1 580 4.92 MB/sec warmup 4 sec latency 59.794 ms 1 762 4.27 MB/sec warmup 5 sec latency 27.263 ms test_26 fail mds1 1 times Failing mds1 on oleg305-server 1 884 3.61 MB/sec warmup 6 sec latency 94.261 ms Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 914 3.11 MB/sec warmup 7 sec latency 866.555 ms 1 995 2.74 MB/sec warmup 8 sec latency 596.819 ms 1 995 2.43 MB/sec warmup 9 sec latency 1596.971 ms 1 995 2.19 MB/sec warmup 10 sec latency 2597.095 ms 1 995 1.99 MB/sec warmup 11 sec latency 3597.289 ms 1 995 1.83 MB/sec warmup 12 sec latency 4597.435 ms 1 995 1.69 MB/sec warmup 13 sec latency 5597.609 ms 11:51:51 (1777045911) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 995 1.56 MB/sec warmup 14 sec latency 6597.790 ms 1 995 1.46 MB/sec warmup 15 sec latency 7597.946 ms 1 995 1.37 MB/sec warmup 16 sec latency 8598.200 ms 1 995 1.29 MB/sec warmup 17 sec latency 9598.397 ms 1 995 1.22 MB/sec warmup 18 sec latency 10598.557 ms 1 995 1.15 MB/sec warmup 19 sec latency 11598.704 ms 1 995 0.00 MB/sec execute 1 sec latency 13598.945 ms 1 995 0.00 MB/sec execute 2 sec latency 14599.132 ms 1 995 0.00 MB/sec execute 3 sec latency 15599.351 ms Failover mds1 to oleg305-server mount facets: mds1 1 995 0.00 MB/sec execute 4 sec latency 16599.500 ms 1 995 0.00 MB/sec execute 5 sec latency 17599.639 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.00 MB/sec execute 6 sec latency 18599.781 ms 1 995 0.00 MB/sec execute 7 sec latency 19599.909 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 1 995 0.00 MB/sec execute 8 sec latency 20600.026 ms Started lustre-MDT0000 11:52:06 (1777045926) targets are mounted 11:52:06 (1777045926) facet_failover done 1 995 0.00 MB/sec execute 9 sec latency 21600.197 ms 1 995 0.00 MB/sec execute 10 sec latency 22600.364 ms 1 995 0.00 MB/sec execute 11 sec latency 23600.572 ms 1 995 0.00 MB/sec execute 12 sec latency 24600.711 ms 1 1082 0.08 MB/sec execute 13 sec latency 25031.808 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1235 0.08 MB/sec execute 14 sec latency 31.827 ms 1 1466 0.27 MB/sec execute 15 sec latency 44.048 ms 1 1613 0.27 MB/sec execute 16 sec latency 141.194 ms 1 1814 0.26 MB/sec execute 17 sec latency 31.524 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1268888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1269180 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14092 3591552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 45960 3560452 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 60052 7152004 1% /mnt/lustre 1 1994 0.25 MB/sec execute 18 sec latency 37.148 ms 1 2156 0.25 MB/sec execute 19 sec latency 30.171 ms 1 2359 0.30 MB/sec execute 20 sec latency 29.030 ms 1 2479 0.32 MB/sec execute 21 sec latency 33.601 ms 1 2773 0.46 MB/sec execute 22 sec latency 44.510 ms test_26 fail mds2 2 times Failing mds2 on oleg305-server 1 3005 0.51 MB/sec execute 23 sec latency 32.003 ms Stopping /mnt/lustre-mds2 (opts:) on oleg305-server 1 3493 0.68 MB/sec execute 24 sec latency 35.187 ms 1 3766 0.71 MB/sec execute 25 sec latency 14.616 ms 1 3987 0.70 MB/sec execute 26 sec latency 17.046 ms 1 4083 0.67 MB/sec execute 27 sec latency 572.767 ms 1 4083 0.65 MB/sec execute 28 sec latency 1572.910 ms 1 4083 0.63 MB/sec execute 29 sec latency 2573.056 ms 11:52:28 (1777045948) shut down 1 4083 0.61 MB/sec execute 30 sec latency 3573.173 ms facet: mds2 facet_host: oleg305-server facet_failover_host: oleg305-server 1 4083 0.59 MB/sec execute 31 sec latency 4573.346 ms 1 4083 0.57 MB/sec execute 32 sec latency 5573.496 ms 1 4083 0.55 MB/sec execute 33 sec latency 6573.615 ms 1 4083 0.53 MB/sec execute 34 sec latency 7573.775 ms 1 4083 0.52 MB/sec execute 35 sec latency 8573.924 ms 1 4083 0.50 MB/sec execute 36 sec latency 9574.037 ms 1 4083 0.49 MB/sec execute 37 sec latency 10574.153 ms 1 4083 0.48 MB/sec execute 38 sec latency 11574.297 ms 1 4083 0.47 MB/sec execute 39 sec latency 12574.477 ms 1 4083 0.45 MB/sec execute 40 sec latency 13574.604 ms Failover mds2 to oleg305-server mount facets: mds2 1 4083 0.44 MB/sec execute 41 sec latency 14574.779 ms 1 4083 0.43 MB/sec execute 42 sec latency 15574.905 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4083 0.42 MB/sec execute 43 sec latency 16575.018 ms 1 4083 0.41 MB/sec execute 44 sec latency 17575.128 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 1 4083 0.40 MB/sec execute 45 sec latency 18575.285 ms Started lustre-MDT0001 11:52:43 (1777045963) targets are mounted 11:52:43 (1777045963) facet_failover done 1 4083 0.40 MB/sec execute 46 sec latency 19575.490 ms 1 4083 0.39 MB/sec execute 47 sec latency 20575.690 ms 1 4083 0.38 MB/sec execute 48 sec latency 21575.848 ms 1 4083 0.37 MB/sec execute 49 sec latency 22576.011 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4249 0.37 MB/sec execute 50 sec latency 22648.737 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4365 0.37 MB/sec execute 51 sec latency 65.634 ms 1 4508 0.37 MB/sec execute 52 sec latency 35.225 ms 1 4647 0.38 MB/sec execute 53 sec latency 58.860 ms 1 4786 0.38 MB/sec execute 54 sec latency 32.559 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2928 1268376 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2708 1268596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24232 3581624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 46044 3560060 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 70276 7141684 1% /mnt/lustre 1 4971 0.42 MB/sec execute 55 sec latency 40.010 ms 1 5078 0.42 MB/sec execute 56 sec latency 42.387 ms 1 5197 0.41 MB/sec execute 57 sec latency 47.479 ms 1 5329 0.40 MB/sec execute 58 sec latency 37.541 ms 1 5397 0.40 MB/sec execute 59 sec latency 100.721 ms test_26 fail mds1 3 times Failing mds1 on oleg305-server 1 5545 0.39 MB/sec execute 60 sec latency 60.333 ms Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 5700 0.39 MB/sec execute 61 sec latency 184.065 ms 11:52:59 (1777045979) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 5820 0.39 MB/sec execute 62 sec latency 295.555 ms 1 5820 0.38 MB/sec execute 63 sec latency 1295.793 ms 1 5820 0.37 MB/sec execute 64 sec latency 2296.066 ms 1 5820 0.37 MB/sec execute 65 sec latency 3296.327 ms 1 5820 0.36 MB/sec execute 66 sec latency 4296.520 ms 1 5820 0.36 MB/sec execute 67 sec latency 5296.790 ms 1 5820 0.35 MB/sec execute 68 sec latency 6296.980 ms 1 5820 0.35 MB/sec execute 69 sec latency 7297.206 ms 1 5820 0.34 MB/sec execute 70 sec latency 8297.432 ms 1 5820 0.34 MB/sec execute 71 sec latency 9297.750 ms Failover mds1 to oleg305-server 1 5820 0.33 MB/sec execute 72 sec latency 10297.943 ms mount facets: mds1 1 5820 0.33 MB/sec execute 73 sec latency 11298.073 ms 1 5820 0.32 MB/sec execute 74 sec latency 12298.322 ms 1 5820 0.32 MB/sec execute 75 sec latency 13298.568 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5820 0.32 MB/sec execute 76 sec latency 14298.834 ms 1 5820 0.31 MB/sec execute 77 sec latency 15299.101 ms 1 5820 0.31 MB/sec execute 78 sec latency 16299.269 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 1 5820 0.30 MB/sec execute 79 sec latency 17301.326 ms Started lustre-MDT0000 11:53:17 (1777045997) targets are mounted 11:53:17 (1777045997) facet_failover done 1 5820 0.30 MB/sec execute 80 sec latency 18303.676 ms 1 5820 0.30 MB/sec execute 81 sec latency 19305.565 ms 1 5820 0.29 MB/sec execute 82 sec latency 20305.804 ms 1 5832 0.29 MB/sec execute 83 sec latency 21235.614 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5967 0.30 MB/sec execute 84 sec latency 43.150 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6121 0.31 MB/sec execute 85 sec latency 40.947 ms 1 6331 0.34 MB/sec execute 86 sec latency 48.066 ms 1 6534 0.35 MB/sec execute 87 sec latency 51.887 ms 1 6752 0.36 MB/sec execute 88 sec latency 57.666 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3304 1268000 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3080 1268224 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26400 3575984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 46664 3558492 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 73064 7134476 2% /mnt/lustre 1 6914 0.37 MB/sec execute 89 sec latency 72.440 ms 1 7110 0.40 MB/sec execute 90 sec latency 46.102 ms 1 7254 0.40 MB/sec execute 91 sec latency 33.326 ms 1 7396 0.40 MB/sec execute 92 sec latency 49.687 ms 1 7502 0.40 MB/sec execute 93 sec latency 54.448 ms test_26 fail mds2 4 times 1 7590 0.40 MB/sec execute 94 sec latency 38.786 ms Failing mds2 on oleg305-server Stopping /mnt/lustre-mds2 (opts:) on oleg305-server 1 7713 0.39 MB/sec execute 95 sec latency 71.154 ms 1 7722 0.39 MB/sec execute 96 sec latency 966.234 ms 1 7722 0.38 MB/sec execute 97 sec latency 1966.412 ms 1 7722 0.38 MB/sec execute 98 sec latency 2966.558 ms 1 7722 0.38 MB/sec execute 99 sec latency 3966.817 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 11:53:40 (1777046020) shut down facet: mds2 facet_host: oleg305-server facet_failover_host: oleg305-server 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec Failover mds2 to oleg305-server mount facets: mds2 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 cleanup 116 sec 1 cleanup 117 sec oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 1 cleanup 118 sec Started lustre-MDT0001 11:53:56 (1777046036) targets are mounted 11:53:56 (1777046036) facet_failover done 1 cleanup 119 sec 1 cleanup 120 sec 1 cleanup 121 sec 1 cleanup 122 sec oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 0 cleanup 123 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1158 79.455 26883.931 Close 840 2.958 33.451 Rename 48 29.023 58.843 Unlink 238 9.505 30.644 Qpathinfo 1109 25.421 22648.719 Qfileinfo 185 1.339 7.933 Qfsinfo 191 0.575 4.899 Sfileinfo 82 17.973 36.190 Find 407 4.896 19.354 WriteX 558 4.889 184.046 ReadX 1842 0.234 57.927 LockX 4 3.462 5.063 UnlockX 4 4.171 5.694 Flush 62 15.787 64.843 Throughput 0.376683 MB/sec 1 clients 1 procs max_latency=25031.808 ms stopping dbench on /mnt/lustre at Fri Apr 24 11:54:01 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Apr 24 11:54:03 EDT 2026 waiting for dbench pid 68898 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 122 5.50 MB/sec warmup 1 sec latency 45.759 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3848 1267456 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3664 1267640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 39508 3565448 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12264 3594700 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 51772 7160148 1% /mnt/lustre 1 197 3.92 MB/sec warmup 2 sec latency 132.252 ms 1 305 3.74 MB/sec warmup 3 sec latency 48.136 ms 1 416 3.65 MB/sec warmup 4 sec latency 41.915 ms 1 522 3.61 MB/sec warmup 5 sec latency 39.150 ms 1 666 3.53 MB/sec warmup 6 sec latency 151.672 ms test_26 fail mds1 5 times 1 740 3.05 MB/sec warmup 7 sec latency 91.724 ms Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 859 2.70 MB/sec warmup 8 sec latency 156.923 ms 1 874 2.41 MB/sec warmup 9 sec latency 831.367 ms 11:54:13 (1777046053) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 874 2.17 MB/sec warmup 10 sec latency 1831.594 ms 1 874 1.97 MB/sec warmup 11 sec latency 2831.872 ms 1 874 1.80 MB/sec warmup 12 sec latency 3832.095 ms 1 874 1.67 MB/sec warmup 13 sec latency 4832.332 ms 1 874 1.55 MB/sec warmup 14 sec latency 5832.580 ms 1 874 1.44 MB/sec warmup 15 sec latency 6832.859 ms 1 874 1.35 MB/sec warmup 16 sec latency 7833.076 ms 1 874 1.27 MB/sec warmup 17 sec latency 8833.323 ms 1 874 1.20 MB/sec warmup 18 sec latency 9833.581 ms 1 874 1.14 MB/sec warmup 19 sec latency 10833.846 ms Failover mds1 to oleg305-server mount facets: mds1 1 874 0.00 MB/sec execute 1 sec latency 12834.228 ms 1 874 0.00 MB/sec execute 2 sec latency 13834.390 ms 1 874 0.00 MB/sec execute 3 sec latency 14834.571 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 874 0.00 MB/sec execute 4 sec latency 15834.830 ms 1 874 0.00 MB/sec execute 5 sec latency 16835.083 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1 874 0.00 MB/sec execute 6 sec latency 17835.337 ms pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 11:54:30 (1777046070) targets are mounted 11:54:30 (1777046070) facet_failover done 1 874 0.00 MB/sec execute 7 sec latency 18835.588 ms 1 874 0.00 MB/sec execute 8 sec latency 19835.791 ms 1 874 0.00 MB/sec execute 9 sec latency 20835.950 ms 1 874 0.00 MB/sec execute 10 sec latency 21836.081 ms 1 884 0.00 MB/sec execute 11 sec latency 22642.128 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 935 0.01 MB/sec execute 12 sec latency 493.932 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1077 0.10 MB/sec execute 13 sec latency 34.157 ms 1 1173 0.10 MB/sec execute 14 sec latency 48.397 ms 1 1268 0.10 MB/sec execute 15 sec latency 48.829 ms 1 1459 0.27 MB/sec execute 16 sec latency 48.219 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3696 1267608 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3500 1267804 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 49660 3555804 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24040 3582124 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 73700 7137928 2% /mnt/lustre 1 1538 0.26 MB/sec execute 17 sec latency 188.200 ms 1 1643 0.25 MB/sec execute 18 sec latency 62.733 ms 1 1799 0.25 MB/sec execute 19 sec latency 38.600 ms 1 1888 0.24 MB/sec execute 20 sec latency 127.367 ms 1 2004 0.23 MB/sec execute 21 sec latency 49.201 ms test_26 fail mds2 6 times Failing mds2 on oleg305-server 1 2117 0.23 MB/sec execute 22 sec latency 45.720 ms Stopping /mnt/lustre-mds2 (opts:) on oleg305-server 1 2259 0.22 MB/sec execute 23 sec latency 143.483 ms 11:54:47 (1777046087) shut down facet: mds2 facet_host: oleg305-server facet_failover_host: oleg305-server 1 2259 0.21 MB/sec execute 24 sec latency 1143.732 ms 1 2259 0.21 MB/sec execute 25 sec latency 2143.958 ms 1 2259 0.20 MB/sec execute 26 sec latency 3144.196 ms 1 2259 0.19 MB/sec execute 27 sec latency 4144.430 ms 1 2259 0.18 MB/sec execute 28 sec latency 5144.721 ms 1 2259 0.18 MB/sec execute 29 sec latency 6144.977 ms 1 2259 0.17 MB/sec execute 30 sec latency 7145.222 ms 1 2259 0.17 MB/sec execute 31 sec latency 8145.473 ms 1 2259 0.16 MB/sec execute 32 sec latency 9145.693 ms 1 2259 0.16 MB/sec execute 33 sec latency 10145.892 ms Failover mds2 to oleg305-server mount facets: mds2 1 2259 0.15 MB/sec execute 34 sec latency 11146.165 ms 1 2259 0.15 MB/sec execute 35 sec latency 12146.460 ms 1 2259 0.14 MB/sec execute 36 sec latency 13146.709 ms 1 2259 0.14 MB/sec execute 37 sec latency 14146.869 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.13 MB/sec execute 38 sec latency 15147.081 ms 1 2259 0.13 MB/sec execute 39 sec latency 16147.245 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1 2259 0.13 MB/sec execute 40 sec latency 17147.440 ms pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0001 11:55:04 (1777046104) targets are mounted 11:55:04 (1777046104) facet_failover done 1 2259 0.13 MB/sec execute 41 sec latency 18147.703 ms 1 2259 0.12 MB/sec execute 42 sec latency 19147.911 ms 1 2259 0.12 MB/sec execute 43 sec latency 20148.241 ms 1 2259 0.12 MB/sec execute 44 sec latency 21148.479 ms 1 2373 0.14 MB/sec execute 45 sec latency 21563.383 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2494 0.15 MB/sec execute 46 sec latency 36.163 ms 1 2770 0.22 MB/sec execute 47 sec latency 95.582 ms 1 2984 0.25 MB/sec execute 48 sec latency 57.427 ms 1 3209 0.26 MB/sec execute 49 sec latency 80.682 ms 1 3374 0.28 MB/sec execute 50 sec latency 47.165 ms 1 3577 0.33 MB/sec execute 51 sec latency 40.398 ms 1 3695 0.33 MB/sec execute 52 sec latency 37.972 ms 1 3801 0.34 MB/sec execute 53 sec latency 43.107 ms 1 3884 0.34 MB/sec execute 54 sec latency 84.861 ms 1 3971 0.33 MB/sec execute 55 sec latency 117.593 ms 1 4070 0.33 MB/sec execute 56 sec latency 40.406 ms 1 4213 0.32 MB/sec execute 57 sec latency 42.721 ms 1 4301 0.32 MB/sec execute 58 sec latency 61.321 ms 1 4419 0.32 MB/sec execute 59 sec latency 45.764 ms 1 4494 0.32 MB/sec execute 60 sec latency 89.653 ms 1 4631 0.33 MB/sec execute 61 sec latency 47.656 ms 1 4811 0.33 MB/sec execute 62 sec latency 28.051 ms 1 4978 0.37 MB/sec execute 63 sec latency 41.565 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Apr 24 11:55:27 EDT 2026 with return code 0 68898 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 68898 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished