== replay-dual test 26: dbench and tar with mds failover ========================================================== 06:14:25 (1762168465) Starting client oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre Started clients oleg117-client.virtnet: 192.168.201.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44514 Started dbench loop with 44515 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Nov 3 06:14:28 EST 2025 waiting for dbench pid 44548 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3726336 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 32768 7472128 1% /mnt/lustre 1 111 5.24 MB/sec warmup 1 sec latency 54.683 ms 1 166 3.40 MB/sec warmup 2 sec latency 99.737 ms 1 237 3.02 MB/sec warmup 3 sec latency 109.404 ms 1 307 2.80 MB/sec warmup 4 sec latency 66.594 ms test_26 fail mds1 1 times 1 381 2.71 MB/sec warmup 5 sec latency 70.327 ms Failing mds1 on oleg117-server 1 469 2.74 MB/sec warmup 6 sec latency 58.013 ms Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 1 513 2.53 MB/sec warmup 7 sec latency 388.815 ms 1 513 2.21 MB/sec warmup 8 sec latency 1389.554 ms 06:14:36 (1762168476) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server 1 513 1.97 MB/sec warmup 9 sec latency 2389.775 ms 1 513 1.77 MB/sec warmup 10 sec latency 3389.965 ms 1 513 1.61 MB/sec warmup 11 sec latency 4390.495 ms 1 513 1.48 MB/sec warmup 12 sec latency 5391.220 ms 1 513 1.36 MB/sec warmup 13 sec latency 6392.457 ms 1 513 1.27 MB/sec warmup 14 sec latency 7392.683 ms 1 513 1.18 MB/sec warmup 15 sec latency 8393.776 ms 1 513 1.11 MB/sec warmup 16 sec latency 9394.017 ms 1 513 1.04 MB/sec warmup 17 sec latency 10394.195 ms 1 513 0.98 MB/sec warmup 18 sec latency 11394.359 ms Failover mds1 to oleg117-server mount facets: mds1 1 513 0.93 MB/sec warmup 19 sec latency 12394.572 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 513 0.00 MB/sec execute 1 sec latency 14394.924 ms 1 513 0.00 MB/sec execute 2 sec latency 15395.222 ms oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 1 520 0.09 MB/sec execute 3 sec latency 16347.844 ms Started lustre-MDT0000 06:14:51 (1762168491) targets are mounted 06:14:51 (1762168491) facet_failover done 1 692 0.87 MB/sec execute 4 sec latency 66.121 ms 1 784 0.74 MB/sec execute 5 sec latency 82.232 ms oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 870 0.65 MB/sec execute 6 sec latency 111.583 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 923 0.58 MB/sec execute 7 sec latency 75.215 ms 1 1033 0.64 MB/sec execute 8 sec latency 87.512 ms 1 1154 0.59 MB/sec execute 9 sec latency 47.330 ms 1 1257 0.54 MB/sec execute 10 sec latency 43.551 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3724288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 49152 3664896 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7389184 1% /mnt/lustre 1 1446 0.75 MB/sec execute 11 sec latency 40.009 ms 1 1542 0.70 MB/sec execute 12 sec latency 158.911 ms 1 1619 0.65 MB/sec execute 13 sec latency 41.945 ms 1 1709 0.61 MB/sec execute 14 sec latency 72.503 ms test_26 fail mds1 2 times 1 1843 0.58 MB/sec execute 15 sec latency 36.816 ms Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 1 1973 0.55 MB/sec execute 16 sec latency 70.297 ms 1 1973 0.51 MB/sec execute 17 sec latency 1052.575 ms 06:15:05 (1762168505) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server 1 1973 0.48 MB/sec execute 18 sec latency 2052.823 ms 1 1973 0.46 MB/sec execute 19 sec latency 3053.043 ms 1 1973 0.44 MB/sec execute 20 sec latency 4053.271 ms 1 1973 0.42 MB/sec execute 21 sec latency 5053.802 ms 1 1973 0.40 MB/sec execute 22 sec latency 6054.028 ms 1 1973 0.38 MB/sec execute 23 sec latency 7054.359 ms 1 1973 0.36 MB/sec execute 24 sec latency 8054.564 ms 1 1973 0.35 MB/sec execute 25 sec latency 9054.783 ms 1 1973 0.34 MB/sec execute 26 sec latency 10054.960 ms 1 1973 0.32 MB/sec execute 27 sec latency 11055.733 ms Failover mds1 to oleg117-server mount facets: mds1 1 1973 0.31 MB/sec execute 28 sec latency 12055.927 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1973 0.30 MB/sec execute 29 sec latency 13056.161 ms 1 1973 0.29 MB/sec execute 30 sec latency 14056.889 ms oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all 1 1973 0.28 MB/sec execute 31 sec latency 15057.109 ms pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 06:15:20 (1762168520) targets are mounted 06:15:20 (1762168520) facet_failover done 1 1973 0.27 MB/sec execute 32 sec latency 16059.989 ms 1 1973 0.26 MB/sec execute 33 sec latency 17060.161 ms 1 2004 0.26 MB/sec execute 34 sec latency 17537.703 ms oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2095 0.26 MB/sec execute 35 sec latency 50.218 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2196 0.25 MB/sec execute 36 sec latency 47.002 ms 1 2349 0.27 MB/sec execute 37 sec latency 161.822 ms 1 2416 0.27 MB/sec execute 38 sec latency 108.841 ms 1 2500 0.28 MB/sec execute 39 sec latency 112.667 ms 1 2569 0.28 MB/sec execute 40 sec latency 737.249 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4352 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3631104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 62464 3571712 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7202816 2% /mnt/lustre 1 2770 0.35 MB/sec execute 41 sec latency 115.808 ms 1 2973 0.38 MB/sec execute 42 sec latency 98.367 ms 1 3197 0.38 MB/sec execute 43 sec latency 69.564 ms test_26 fail mds1 3 times 1 3353 0.40 MB/sec execute 44 sec latency 87.932 ms Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 1 3543 0.46 MB/sec execute 45 sec latency 42.948 ms 1 3579 0.45 MB/sec execute 46 sec latency 686.967 ms 1 3579 0.44 MB/sec execute 47 sec latency 1687.153 ms 1 3579 0.43 MB/sec execute 48 sec latency 2687.362 ms 1 3579 0.42 MB/sec execute 49 sec latency 3688.102 ms 1 3579 0.41 MB/sec execute 50 sec latency 4689.370 ms 1 3579 0.41 MB/sec execute 51 sec latency 5689.661 ms 1 3579 0.40 MB/sec execute 52 sec latency 6689.898 ms 06:15:41 (1762168541) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server 1 3579 0.39 MB/sec execute 53 sec latency 7690.087 ms 1 3579 0.38 MB/sec execute 54 sec latency 8690.341 ms 1 3579 0.38 MB/sec execute 55 sec latency 9690.549 ms 1 3579 0.37 MB/sec execute 56 sec latency 10690.854 ms 1 3579 0.36 MB/sec execute 57 sec latency 11691.136 ms 1 3579 0.36 MB/sec execute 58 sec latency 12693.197 ms 1 3579 0.35 MB/sec execute 59 sec latency 13693.408 ms 1 3579 0.34 MB/sec execute 60 sec latency 14693.910 ms 1 3579 0.34 MB/sec execute 61 sec latency 15694.109 ms 1 3579 0.33 MB/sec execute 62 sec latency 16694.422 ms Failover mds1 to oleg117-server mount facets: mds1 1 3579 0.33 MB/sec execute 63 sec latency 17694.809 ms 1 3579 0.32 MB/sec execute 64 sec latency 18698.011 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3579 0.32 MB/sec execute 65 sec latency 19698.865 ms 1 3579 0.31 MB/sec execute 66 sec latency 20699.122 ms oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all 1 3579 0.31 MB/sec execute 67 sec latency 21699.324 ms pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 06:15:56 (1762168556) targets are mounted 06:15:56 (1762168556) facet_failover done 1 3579 0.30 MB/sec execute 68 sec latency 22699.533 ms 1 3579 0.30 MB/sec execute 69 sec latency 23699.783 ms 1 3579 0.30 MB/sec execute 70 sec latency 24699.984 ms 1 3579 0.29 MB/sec execute 71 sec latency 25700.290 ms 1 3579 0.29 MB/sec execute 72 sec latency 26700.613 ms 1 3579 0.28 MB/sec execute 73 sec latency 27700.962 ms 1 3579 0.28 MB/sec execute 74 sec latency 28701.273 ms 1 3579 0.28 MB/sec execute 75 sec latency 29702.669 ms 1 3640 0.27 MB/sec execute 76 sec latency 30013.667 ms oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3729 0.28 MB/sec execute 77 sec latency 46.632 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3836 0.28 MB/sec execute 78 sec latency 46.720 ms 1 3942 0.28 MB/sec execute 79 sec latency 38.105 ms 1 4042 0.28 MB/sec execute 80 sec latency 37.917 ms 1 4136 0.28 MB/sec execute 81 sec latency 87.251 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5248 2203264 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3720192 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 67584 3652608 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 84992 7372800 2% /mnt/lustre 1 4248 0.28 MB/sec execute 82 sec latency 147.476 ms 1 4343 0.27 MB/sec execute 83 sec latency 92.488 ms 1 4448 0.27 MB/sec execute 84 sec latency 76.459 ms 1 4549 0.27 MB/sec execute 85 sec latency 60.244 ms test_26 fail mds1 4 times Failing mds1 on oleg117-server 1 4684 0.28 MB/sec execute 86 sec latency 114.512 ms Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 1 4762 0.28 MB/sec execute 87 sec latency 230.360 ms 1 4762 0.28 MB/sec execute 88 sec latency 1230.763 ms 06:16:16 (1762168576) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server 1 4762 0.27 MB/sec execute 89 sec latency 2235.059 ms 1 4762 0.27 MB/sec execute 90 sec latency 3235.330 ms 1 4762 0.27 MB/sec execute 91 sec latency 4235.638 ms 1 4762 0.27 MB/sec execute 92 sec latency 5235.861 ms 1 4762 0.26 MB/sec execute 93 sec latency 6236.147 ms 1 4762 0.26 MB/sec execute 94 sec latency 7236.468 ms 1 4762 0.26 MB/sec execute 95 sec latency 8236.750 ms 1 4762 0.25 MB/sec execute 96 sec latency 9237.027 ms 1 4762 0.25 MB/sec execute 97 sec latency 10237.245 ms 1 4762 0.25 MB/sec execute 98 sec latency 11237.434 ms Failover mds1 to oleg117-server mount facets: mds1 1 4762 0.25 MB/sec execute 99 sec latency 12237.748 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all 1 cleanup 103 sec pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 06:16:32 (1762168592) targets are mounted 06:16:32 (1762168592) facet_failover done 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 0 cleanup 108 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 710 87.750 30013.653 Close 537 4.949 18.638 Rename 33 37.162 69.224 Unlink 137 10.770 19.610 Qpathinfo 648 64.716 19574.173 Qfileinfo 127 0.975 6.372 Qfsinfo 130 1.203 54.303 Sfileinfo 77 23.253 56.304 Find 251 3.059 26.872 WriteX 444 4.663 17.995 ReadX 1085 0.825 41.129 LockX 2 4.603 5.506 UnlockX 2 4.473 4.826 Flush 67 53.366 161.807 Throughput 0.246757 MB/sec 1 clients 1 procs max_latency=30013.667 ms stopping dbench on /mnt/lustre at Mon Nov 3 06:16:36 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Nov 3 06:16:39 EST 2025 waiting for dbench pid 48423 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 113 5.24 MB/sec warmup 1 sec latency 68.531 ms 1 184 3.69 MB/sec warmup 2 sec latency 92.971 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6144 2202368 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3617792 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 68608 3602432 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 108544 7220224 2% /mnt/lustre 1 271 3.39 MB/sec warmup 3 sec latency 55.066 ms 1 348 3.13 MB/sec warmup 4 sec latency 69.706 ms 1 446 3.13 MB/sec warmup 5 sec latency 59.010 ms 1 542 3.18 MB/sec warmup 6 sec latency 69.540 ms test_26 fail mds1 5 times 1 674 3.02 MB/sec warmup 7 sec latency 170.477 ms Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 1 738 2.66 MB/sec warmup 8 sec latency 82.480 ms 1 740 2.37 MB/sec warmup 9 sec latency 982.995 ms 06:16:48 (1762168608) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server 1 740 2.13 MB/sec warmup 10 sec latency 1984.761 ms 1 740 1.94 MB/sec warmup 11 sec latency 2985.613 ms 1 740 1.78 MB/sec warmup 12 sec latency 3985.980 ms 1 740 1.64 MB/sec warmup 13 sec latency 4987.019 ms 1 740 1.52 MB/sec warmup 14 sec latency 5987.266 ms 1 740 1.42 MB/sec warmup 15 sec latency 6988.130 ms 1 740 1.33 MB/sec warmup 16 sec latency 7988.337 ms 1 740 1.25 MB/sec warmup 17 sec latency 8988.654 ms 1 740 1.18 MB/sec warmup 18 sec latency 9989.250 ms 1 740 1.12 MB/sec warmup 19 sec latency 10989.547 ms Failover mds1 to oleg117-server mount facets: mds1 1 740 0.00 MB/sec execute 1 sec latency 12989.852 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 740 0.00 MB/sec execute 2 sec latency 13990.041 ms 1 740 0.00 MB/sec execute 3 sec latency 14990.992 ms 1 740 0.00 MB/sec execute 4 sec latency 15991.727 ms oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 1 740 0.00 MB/sec execute 5 sec latency 16992.349 ms Started lustre-MDT0000 06:17:04 (1762168624) targets are mounted 06:17:04 (1762168624) facet_failover done 1 789 0.02 MB/sec execute 6 sec latency 17453.334 ms 1 873 0.05 MB/sec execute 7 sec latency 151.033 ms oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 930 0.06 MB/sec execute 8 sec latency 121.204 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1002 0.08 MB/sec execute 9 sec latency 87.071 ms 1 1122 0.17 MB/sec execute 10 sec latency 89.252 ms 1 1224 0.16 MB/sec execute 11 sec latency 50.188 ms 1 1405 0.39 MB/sec execute 12 sec latency 38.672 ms 1 1497 0.36 MB/sec execute 13 sec latency 113.973 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6656 2201856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 38912 3686400 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3687424 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 71680 7373824 1% /mnt/lustre 1 1573 0.34 MB/sec execute 14 sec latency 53.351 ms 1 1642 0.32 MB/sec execute 15 sec latency 90.827 ms 1 1751 0.31 MB/sec execute 16 sec latency 55.488 ms test_26 fail mds1 6 times 1 1877 0.30 MB/sec execute 17 sec latency 38.812 ms Failing mds1 on oleg117-server 1 1984 0.29 MB/sec execute 18 sec latency 177.405 ms Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 1 2030 0.28 MB/sec execute 19 sec latency 422.475 ms 06:17:19 (1762168639) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server 1 2030 0.26 MB/sec execute 20 sec latency 1422.721 ms 1 2030 0.25 MB/sec execute 21 sec latency 2422.922 ms 1 2030 0.24 MB/sec execute 22 sec latency 3423.114 ms 1 2030 0.23 MB/sec execute 23 sec latency 4423.935 ms 1 2030 0.22 MB/sec execute 24 sec latency 5424.111 ms 1 2030 0.21 MB/sec execute 25 sec latency 6424.307 ms 1 2030 0.20 MB/sec execute 26 sec latency 7424.526 ms 1 2030 0.19 MB/sec execute 27 sec latency 8425.471 ms 1 2030 0.19 MB/sec execute 28 sec latency 9425.694 ms 1 2030 0.18 MB/sec execute 29 sec latency 10425.936 ms 1 2030 0.18 MB/sec execute 30 sec latency 11426.456 ms Failover mds1 to oleg117-server mount facets: mds1 1 2030 0.17 MB/sec execute 31 sec latency 12426.643 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2030 0.16 MB/sec execute 32 sec latency 13426.800 ms 1 2030 0.16 MB/sec execute 33 sec latency 14427.239 ms 1 2030 0.15 MB/sec execute 34 sec latency 15427.985 ms oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all 1 2030 0.15 MB/sec execute 35 sec latency 16428.146 ms pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 06:17:35 (1762168655) targets are mounted 06:17:35 (1762168655) facet_failover done 1 2030 0.15 MB/sec execute 36 sec latency 17429.278 ms 1 2030 0.14 MB/sec execute 37 sec latency 18429.945 ms 1 2030 0.14 MB/sec execute 38 sec latency 19432.817 ms 1 2030 0.13 MB/sec execute 39 sec latency 20432.983 ms 1 2034 0.13 MB/sec execute 40 sec latency 21389.244 ms oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2156 0.13 MB/sec execute 41 sec latency 39.766 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2295 0.15 MB/sec execute 42 sec latency 45.149 ms 1 2394 0.15 MB/sec execute 43 sec latency 146.562 ms 1 2478 0.17 MB/sec execute 44 sec latency 85.679 ms 1 2566 0.17 MB/sec execute 45 sec latency 574.869 ms 1 2758 0.23 MB/sec execute 46 sec latency 804.472 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 44032 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 36864 3708928 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 80896 7421952 2% /mnt/lustre 1 2922 0.26 MB/sec execute 47 sec latency 145.378 ms 1 3013 0.25 MB/sec execute 48 sec latency 141.064 ms 1 3277 0.29 MB/sec execute 49 sec latency 93.469 ms 1 3504 0.34 MB/sec execute 50 sec latency 54.070 ms test_26 fail mds1 7 times 1 3632 0.34 MB/sec execute 51 sec latency 37.486 ms Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 1 3755 0.35 MB/sec execute 52 sec latency 70.854 ms 1 3755 0.35 MB/sec execute 53 sec latency 1071.713 ms 06:17:52 (1762168672) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server 1 3755 0.34 MB/sec execute 54 sec latency 2071.906 ms 1 3755 0.34 MB/sec execute 55 sec latency 3072.223 ms 1 3755 0.33 MB/sec execute 56 sec latency 4072.409 ms 1 3755 0.32 MB/sec execute 57 sec latency 5072.901 ms 1 3755 0.32 MB/sec execute 58 sec latency 6073.090 ms 1 3755 0.31 MB/sec execute 59 sec latency 7073.470 ms 1 3755 0.31 MB/sec execute 60 sec latency 8074.385 ms 1 3755 0.30 MB/sec execute 61 sec latency 9075.392 ms 1 3755 0.30 MB/sec execute 62 sec latency 10076.042 ms 1 3755 0.29 MB/sec execute 63 sec latency 11076.238 ms Failover mds1 to oleg117-server mount facets: mds1 1 3755 0.29 MB/sec execute 64 sec latency 12076.409 ms 1 3755 0.28 MB/sec execute 65 sec latency 13076.573 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3755 0.28 MB/sec execute 66 sec latency 14076.812 ms 1 3755 0.28 MB/sec execute 67 sec latency 15079.360 ms oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all 1 3755 0.27 MB/sec execute 68 sec latency 16079.572 ms pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 06:18:07 (1762168687) targets are mounted 06:18:07 (1762168687) facet_failover done 1 3755 0.27 MB/sec execute 69 sec latency 17080.191 ms 1 3755 0.26 MB/sec execute 70 sec latency 18080.388 ms 1 3802 0.26 MB/sec execute 71 sec latency 18333.446 ms oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3900 0.26 MB/sec execute 72 sec latency 38.545 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3988 0.26 MB/sec execute 73 sec latency 46.491 ms 1 4079 0.25 MB/sec execute 74 sec latency 48.857 ms 1 4168 0.25 MB/sec execute 75 sec latency 40.360 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Nov 3 06:18:15 EST 2025 with return code 0 1 4243 0.25 MB/sec execute 76 sec latency 150.196 ms 48423 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48423 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished