== replay-dual test 26: dbench and tar with mds failover ========================================================== 09:33:30 (1773236010) Starting client oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre Started clients oleg124-client.virtnet: 192.168.201.124@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65355 Started dbench loop with 65356 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Mar 11 09:33:33 EDT 2026 waiting for dbench pid 65399 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1269304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1269524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3579592 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3108 7185048 1% /mnt/lustre 1 64 2.75 MB/sec warmup 1 sec latency 53.715 ms 1 142 3.01 MB/sec warmup 2 sec latency 107.577 ms 1 197 2.61 MB/sec warmup 3 sec latency 213.996 ms 1 267 2.50 MB/sec warmup 4 sec latency 65.572 ms 1 323 2.35 MB/sec warmup 5 sec latency 80.699 ms 1 389 2.30 MB/sec warmup 6 sec latency 87.054 ms test_26 fail mds1 1 times 1 461 2.31 MB/sec warmup 7 sec latency 92.365 ms Failing mds1 on oleg124-server 1 521 2.26 MB/sec warmup 8 sec latency 88.236 ms Stopping /mnt/lustre-mds1 (opts:) on oleg124-server 1 649 2.34 MB/sec warmup 9 sec latency 73.771 ms 1 660 2.11 MB/sec warmup 10 sec latency 887.486 ms 1 660 1.92 MB/sec warmup 11 sec latency 1887.621 ms 1 660 1.76 MB/sec warmup 12 sec latency 2887.789 ms 1 660 1.63 MB/sec warmup 13 sec latency 3888.054 ms 1 660 1.51 MB/sec warmup 14 sec latency 4888.318 ms 1 660 1.41 MB/sec warmup 15 sec latency 5888.468 ms 09:33:49 (1773236029) shut down 1 660 1.32 MB/sec warmup 16 sec latency 6888.626 ms facet: mds1 facet_host: oleg124-server facet_failover_host: oleg124-server 1 660 1.24 MB/sec warmup 17 sec latency 7889.121 ms 1 660 1.17 MB/sec warmup 18 sec latency 8889.332 ms 1 660 1.11 MB/sec warmup 19 sec latency 9889.703 ms 1 660 0.00 MB/sec execute 1 sec latency 11890.037 ms 1 660 0.00 MB/sec execute 2 sec latency 12890.535 ms 1 660 0.00 MB/sec execute 3 sec latency 13890.688 ms 1 660 0.00 MB/sec execute 4 sec latency 14890.870 ms 1 660 0.00 MB/sec execute 5 sec latency 15891.027 ms 1 660 0.00 MB/sec execute 6 sec latency 16891.582 ms Failover mds1 to oleg124-server mount facets: mds1 1 660 0.00 MB/sec execute 7 sec latency 17892.111 ms 1 660 0.00 MB/sec execute 8 sec latency 18892.337 ms 1 660 0.00 MB/sec execute 9 sec latency 19893.148 ms 1 660 0.00 MB/sec execute 10 sec latency 20895.210 ms 1 660 0.00 MB/sec execute 11 sec latency 21896.369 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 660 0.00 MB/sec execute 12 sec latency 22897.106 ms 1 660 0.00 MB/sec execute 13 sec latency 23897.581 ms 1 660 0.00 MB/sec execute 14 sec latency 24898.091 ms oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 1 660 0.00 MB/sec execute 15 sec latency 25898.398 ms pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 1 660 0.00 MB/sec execute 16 sec latency 26898.617 ms Started lustre-MDT0000 09:34:09 (1773236049) targets are mounted 09:34:09 (1773236049) facet_failover done 1 660 0.00 MB/sec execute 17 sec latency 27898.901 ms 1 660 0.00 MB/sec execute 18 sec latency 28899.134 ms 1 660 0.00 MB/sec execute 19 sec latency 29899.752 ms 1 678 0.00 MB/sec execute 20 sec latency 30608.888 ms oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 740 0.01 MB/sec execute 21 sec latency 79.003 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 804 0.01 MB/sec execute 22 sec latency 66.321 ms 1 874 0.02 MB/sec execute 23 sec latency 67.153 ms 1 884 0.02 MB/sec execute 24 sec latency 824.037 ms 1 902 0.02 MB/sec execute 25 sec latency 1321.692 ms 1 960 0.03 MB/sec execute 26 sec latency 75.583 ms 1 1042 0.06 MB/sec execute 27 sec latency 62.237 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1940 1269364 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19440 3587268 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 39608 3566512 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 59048 7153780 1% /mnt/lustre 1 1124 0.07 MB/sec execute 28 sec latency 55.226 ms 1 1190 0.07 MB/sec execute 29 sec latency 54.786 ms 1 1252 0.06 MB/sec execute 30 sec latency 69.803 ms 1 1403 0.16 MB/sec execute 31 sec latency 62.085 ms 1 1467 0.15 MB/sec execute 32 sec latency 92.762 ms 1 1518 0.15 MB/sec execute 33 sec latency 228.679 ms 1 1588 0.15 MB/sec execute 34 sec latency 59.550 ms test_26 fail mds2 2 times 1 1653 0.14 MB/sec execute 35 sec latency 69.788 ms Failing mds2 on oleg124-server 1 1711 0.14 MB/sec execute 36 sec latency 79.636 ms Stopping /mnt/lustre-mds2 (opts:) on oleg124-server 1 1805 0.14 MB/sec execute 37 sec latency 61.497 ms 1 1864 0.14 MB/sec execute 38 sec latency 68.457 ms 1 1942 0.14 MB/sec execute 39 sec latency 81.908 ms 1 2010 0.13 MB/sec execute 40 sec latency 67.372 ms 1 2098 0.13 MB/sec execute 41 sec latency 57.332 ms 1 2170 0.13 MB/sec execute 42 sec latency 65.014 ms 1 2259 0.13 MB/sec execute 43 sec latency 308.299 ms 09:34:37 (1773236077) shut down facet: mds2 facet_host: oleg124-server facet_failover_host: oleg124-server 1 2259 0.13 MB/sec execute 44 sec latency 1308.575 ms 1 2259 0.13 MB/sec execute 45 sec latency 2308.818 ms 1 2259 0.12 MB/sec execute 46 sec latency 3309.734 ms 1 2259 0.12 MB/sec execute 47 sec latency 4310.258 ms 1 2259 0.12 MB/sec execute 48 sec latency 5310.716 ms 1 2259 0.11 MB/sec execute 49 sec latency 6310.931 ms 1 2259 0.11 MB/sec execute 50 sec latency 7311.129 ms 1 2259 0.11 MB/sec execute 51 sec latency 8311.330 ms 1 2259 0.11 MB/sec execute 52 sec latency 9313.223 ms 1 2259 0.11 MB/sec execute 53 sec latency 10313.538 ms Failover mds2 to oleg124-server mount facets: mds2 1 2259 0.10 MB/sec execute 54 sec latency 11313.701 ms 1 2259 0.10 MB/sec execute 55 sec latency 12313.873 ms 1 2259 0.10 MB/sec execute 56 sec latency 13314.118 ms 1 2259 0.10 MB/sec execute 57 sec latency 14314.268 ms 1 2259 0.10 MB/sec execute 58 sec latency 15314.791 ms 1 2259 0.10 MB/sec execute 59 sec latency 16314.991 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.09 MB/sec execute 60 sec latency 17315.180 ms 1 2259 0.09 MB/sec execute 61 sec latency 18315.434 ms 1 2259 0.09 MB/sec execute 62 sec latency 19316.111 ms 1 2259 0.09 MB/sec execute 63 sec latency 20316.337 ms oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 1 2259 0.09 MB/sec execute 64 sec latency 21316.492 ms Started lustre-MDT0001 09:34:58 (1773236098) targets are mounted 09:34:58 (1773236098) facet_failover done 1 2259 0.09 MB/sec execute 65 sec latency 22316.729 ms 1 2259 0.09 MB/sec execute 66 sec latency 23316.990 ms 1 2290 0.10 MB/sec execute 67 sec latency 24067.959 ms 1 2358 0.10 MB/sec execute 68 sec latency 73.288 ms oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 2412 0.10 MB/sec execute 69 sec latency 67.365 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2466 0.11 MB/sec execute 70 sec latency 63.973 ms 1 2562 0.11 MB/sec execute 71 sec latency 69.081 ms 1 2655 0.12 MB/sec execute 72 sec latency 70.532 ms 1 2902 0.16 MB/sec execute 73 sec latency 65.454 ms 1 3010 0.17 MB/sec execute 74 sec latency 66.546 ms 1 3232 0.18 MB/sec execute 75 sec latency 86.557 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2496 1268808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2264 1269040 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 29132 3576876 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 40428 3563284 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 69560 7140160 1% /mnt/lustre 1 3352 0.19 MB/sec execute 76 sec latency 66.242 ms 1 3523 0.22 MB/sec execute 77 sec latency 53.090 ms 1 3628 0.22 MB/sec execute 78 sec latency 52.715 ms 1 3720 0.22 MB/sec execute 79 sec latency 53.710 ms 1 3797 0.23 MB/sec execute 80 sec latency 86.970 ms 1 3860 0.23 MB/sec execute 81 sec latency 61.342 ms test_26 fail mds1 3 times 1 3939 0.23 MB/sec execute 82 sec latency 56.687 ms Failing mds1 on oleg124-server 1 3991 0.23 MB/sec execute 83 sec latency 68.102 ms Stopping /mnt/lustre-mds1 (opts:) on oleg124-server 1 4049 0.23 MB/sec execute 84 sec latency 70.257 ms 1 4083 0.22 MB/sec execute 85 sec latency 383.521 ms 09:35:19 (1773236119) shut down facet: mds1 facet_host: oleg124-server facet_failover_host: oleg124-server 1 4083 0.22 MB/sec execute 86 sec latency 1384.124 ms 1 4083 0.22 MB/sec execute 87 sec latency 2384.383 ms 1 4083 0.22 MB/sec execute 88 sec latency 3384.751 ms 1 4083 0.21 MB/sec execute 89 sec latency 4386.566 ms 1 4083 0.21 MB/sec execute 90 sec latency 5387.690 ms 1 4083 0.21 MB/sec execute 91 sec latency 6390.679 ms 1 4083 0.21 MB/sec execute 92 sec latency 7391.488 ms 1 4083 0.20 MB/sec execute 93 sec latency 8392.203 ms 1 4083 0.20 MB/sec execute 94 sec latency 9393.011 ms 1 4083 0.20 MB/sec execute 95 sec latency 10393.361 ms 1 4083 0.20 MB/sec execute 96 sec latency 11395.017 ms Failover mds1 to oleg124-server mount facets: mds1 1 4083 0.20 MB/sec execute 97 sec latency 12395.742 ms 1 4083 0.19 MB/sec execute 98 sec latency 13396.253 ms 1 4083 0.19 MB/sec execute 99 sec latency 14396.448 ms 1 cleanup 100 sec 1 cleanup 101 sec Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 1 cleanup 114 sec pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 1 cleanup 115 sec Started lustre-MDT0000 09:35:48 (1773236148) targets are mounted 09:35:48 (1773236148) facet_failover done 1 cleanup 116 sec 1 cleanup 117 sec 1 cleanup 118 sec 1 cleanup 119 sec oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 cleanup 120 sec 1 cleanup 121 sec 0 cleanup 121 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 605 71.155 24067.938 Close 443 76.930 30608.857 Rename 26 54.923 86.444 Unlink 95 17.723 47.332 Qpathinfo 530 73.750 33553.320 Qfileinfo 96 1.705 9.268 Qfsinfo 96 1.428 12.209 Sfileinfo 48 38.078 83.162 Find 199 7.075 20.056 WriteX 299 8.180 54.469 ReadX 940 0.724 41.204 LockX 2 5.409 5.882 UnlockX 2 6.240 7.145 Flush 43 34.014 81.894 Throughput 0.191201 MB/sec 1 clients 1 procs max_latency=30608.888 ms stopping dbench on /mnt/lustre at Wed Mar 11 09:35:54 EDT 2026 with return code 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Mar 11 09:36:00 EDT 2026 waiting for dbench pid 68585 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2892 1268412 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2660 1268644 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 45788 3559628 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 3160 3603220 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 48948 7162848 1% /mnt/lustre 1 44 1.70 MB/sec warmup 1 sec latency 68.959 ms 1 113 2.62 MB/sec warmup 2 sec latency 141.999 ms 1 157 2.18 MB/sec warmup 3 sec latency 110.666 ms 1 207 2.03 MB/sec warmup 4 sec latency 238.425 ms 1 282 2.09 MB/sec warmup 5 sec latency 67.448 ms 1 349 2.09 MB/sec warmup 6 sec latency 75.568 ms test_26 fail mds2 4 times 1 421 2.12 MB/sec warmup 7 sec latency 65.365 ms 1 477 2.08 MB/sec warmup 8 sec latency 89.917 ms Failing mds2 on oleg124-server Stopping /mnt/lustre-mds2 (opts:) on oleg124-server 1 538 2.10 MB/sec warmup 9 sec latency 85.158 ms 1 615 2.10 MB/sec warmup 10 sec latency 470.469 ms 1 615 1.91 MB/sec warmup 11 sec latency 1471.078 ms 09:36:11 (1773236171) shut down facet: mds2 facet_host: oleg124-server facet_failover_host: oleg124-server 1 615 1.75 MB/sec warmup 12 sec latency 2471.334 ms 1 615 1.61 MB/sec warmup 13 sec latency 3471.586 ms 1 615 1.50 MB/sec warmup 14 sec latency 4472.073 ms 1 615 1.40 MB/sec warmup 15 sec latency 5472.294 ms 1 615 1.31 MB/sec warmup 16 sec latency 6472.606 ms 1 615 1.23 MB/sec warmup 17 sec latency 7473.025 ms 1 615 1.16 MB/sec warmup 18 sec latency 8474.003 ms 1 615 1.10 MB/sec warmup 19 sec latency 9474.654 ms 1 615 0.00 MB/sec execute 1 sec latency 11477.384 ms Failover mds2 to oleg124-server mount facets: mds2 1 615 0.00 MB/sec execute 2 sec latency 12478.081 ms 1 615 0.00 MB/sec execute 3 sec latency 13478.481 ms 1 615 0.00 MB/sec execute 4 sec latency 14478.771 ms 1 615 0.00 MB/sec execute 5 sec latency 15479.086 ms 1 615 0.00 MB/sec execute 6 sec latency 16479.346 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 615 0.00 MB/sec execute 7 sec latency 17479.719 ms 1 615 0.00 MB/sec execute 8 sec latency 18480.199 ms 1 615 0.00 MB/sec execute 9 sec latency 19481.029 ms oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 1 615 0.00 MB/sec execute 10 sec latency 20481.238 ms pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0001 09:36:31 (1773236191) targets are mounted 09:36:31 (1773236191) facet_failover done 1 615 0.00 MB/sec execute 11 sec latency 21481.522 ms 1 615 0.00 MB/sec execute 12 sec latency 22481.723 ms 1 615 0.00 MB/sec execute 13 sec latency 23482.065 ms 1 615 0.00 MB/sec execute 14 sec latency 24482.237 ms 1 661 0.01 MB/sec execute 15 sec latency 24819.588 ms 1 713 0.01 MB/sec execute 16 sec latency 78.047 ms oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 767 0.02 MB/sec execute 17 sec latency 115.078 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 804 0.03 MB/sec execute 18 sec latency 79.443 ms 1 872 0.04 MB/sec execute 19 sec latency 81.298 ms 1 884 0.03 MB/sec execute 20 sec latency 683.003 ms 1 926 0.04 MB/sec execute 21 sec latency 753.618 ms 1 990 0.04 MB/sec execute 22 sec latency 116.067 ms 1 1100 0.09 MB/sec execute 23 sec latency 53.951 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3228 1268076 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3020 1268284 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 56632 3547132 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 16512 3590060 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 73144 7137192 2% /mnt/lustre 1 1176 0.08 MB/sec execute 24 sec latency 54.684 ms 1 1252 0.08 MB/sec execute 25 sec latency 57.988 ms 1 1405 0.19 MB/sec execute 26 sec latency 61.267 ms 1 1455 0.19 MB/sec execute 27 sec latency 304.714 ms 1 1519 0.18 MB/sec execute 28 sec latency 193.219 ms 1 1603 0.18 MB/sec execute 29 sec latency 37.848 ms test_26 fail mds1 5 times 1 1686 0.17 MB/sec execute 30 sec latency 56.774 ms Failing mds1 on oleg124-server 1 1773 0.17 MB/sec execute 31 sec latency 63.597 ms Stopping /mnt/lustre-mds1 (opts:) on oleg124-server 1 1861 0.17 MB/sec execute 32 sec latency 65.722 ms 1 1984 0.17 MB/sec execute 33 sec latency 47.593 ms 1 2083 0.17 MB/sec execute 34 sec latency 49.103 ms 1 2177 0.16 MB/sec execute 35 sec latency 57.684 ms 1 2259 0.16 MB/sec execute 36 sec latency 418.745 ms 1 2259 0.16 MB/sec execute 37 sec latency 1420.306 ms 1 2259 0.15 MB/sec execute 38 sec latency 2420.507 ms 09:36:58 (1773236218) shut down facet: mds1 facet_host: oleg124-server facet_failover_host: oleg124-server 1 2259 0.15 MB/sec execute 39 sec latency 3421.023 ms 1 2259 0.15 MB/sec execute 40 sec latency 4421.239 ms 1 2259 0.14 MB/sec execute 41 sec latency 5421.722 ms 1 2259 0.14 MB/sec execute 42 sec latency 6422.245 ms 1 2259 0.14 MB/sec execute 43 sec latency 7422.456 ms 1 2259 0.13 MB/sec execute 44 sec latency 8423.266 ms 1 2259 0.13 MB/sec execute 45 sec latency 9423.482 ms 1 2259 0.13 MB/sec execute 46 sec latency 10423.847 ms 1 2259 0.12 MB/sec execute 47 sec latency 11424.130 ms 1 2259 0.12 MB/sec execute 48 sec latency 12426.049 ms Failover mds1 to oleg124-server mount facets: mds1 1 2259 0.12 MB/sec execute 49 sec latency 13426.337 ms 1 2259 0.12 MB/sec execute 50 sec latency 14428.177 ms 1 2259 0.11 MB/sec execute 51 sec latency 15428.805 ms 1 2259 0.11 MB/sec execute 52 sec latency 16429.445 ms 1 2259 0.11 MB/sec execute 53 sec latency 17431.302 ms 1 2259 0.11 MB/sec execute 54 sec latency 18432.766 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 2259 0.11 MB/sec execute 55 sec latency 19433.195 ms 1 2259 0.10 MB/sec execute 56 sec latency 20433.482 ms 1 2259 0.10 MB/sec execute 57 sec latency 21433.767 ms 1 2259 0.10 MB/sec execute 58 sec latency 22434.356 ms 1 2259 0.10 MB/sec execute 59 sec latency 23434.678 ms 1 2259 0.10 MB/sec execute 60 sec latency 24434.885 ms 1 2259 0.10 MB/sec execute 61 sec latency 25435.501 ms 1 2259 0.09 MB/sec execute 62 sec latency 26435.951 ms 1 2259 0.09 MB/sec execute 63 sec latency 27436.176 ms 1 2259 0.09 MB/sec execute 64 sec latency 28436.359 ms 1 2259 0.09 MB/sec execute 65 sec latency 29436.895 ms oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 1 2259 0.09 MB/sec execute 66 sec latency 30437.742 ms pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 1 2259 0.09 MB/sec execute 67 sec latency 31438.164 ms 09:37:27 (1773236247) targets are mounted 09:37:27 (1773236247) facet_failover done 1 2259 0.09 MB/sec execute 68 sec latency 32438.376 ms 1 2259 0.08 MB/sec execute 69 sec latency 33439.289 ms 1 2259 0.08 MB/sec execute 70 sec latency 34439.916 ms 1 2331 0.10 MB/sec execute 71 sec latency 34940.011 ms oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2389 0.10 MB/sec execute 72 sec latency 81.293 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2447 0.10 MB/sec execute 73 sec latency 67.143 ms 1 2549 0.11 MB/sec execute 74 sec latency 61.471 ms 1 2621 0.11 MB/sec execute 75 sec latency 88.399 ms 1 2902 0.16 MB/sec execute 76 sec latency 57.128 ms 1 3000 0.16 MB/sec execute 77 sec latency 67.981 ms 1 3235 0.17 MB/sec execute 78 sec latency 61.389 ms 1 3365 0.18 MB/sec execute 79 sec latency 46.550 ms 1 3543 0.22 MB/sec execute 80 sec latency 48.805 ms 1 3644 0.22 MB/sec execute 81 sec latency 47.137 ms 1 3738 0.22 MB/sec execute 82 sec latency 49.772 ms 1 3820 0.23 MB/sec execute 83 sec latency 65.321 ms 1 3871 0.22 MB/sec execute 84 sec latency 135.405 ms 1 3944 0.22 MB/sec execute 85 sec latency 55.532 ms 1 4014 0.22 MB/sec execute 86 sec latency 52.784 ms 1 4073 0.22 MB/sec execute 87 sec latency 103.487 ms 1 4147 0.22 MB/sec execute 88 sec latency 65.389 ms 1 4236 0.22 MB/sec execute 89 sec latency 52.898 ms 1 4301 0.22 MB/sec execute 90 sec latency 83.469 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Mar 11 09:37:50 EDT 2026 with return code 0 68585 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 68585 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished