== replay-dual test 26: dbench and tar with mds failover ========================================================== 03:34:24 (1759822464) Starting client oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre Started clients oleg146-client.virtnet: 192.168.201.146@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 64908 Started dbench loop with 64909 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Oct 7 03:34:26 EDT 2025 waiting for dbench pid 64953 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 111 5.24 MB/sec warmup 1 sec latency 50.630 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26120 3577472 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3604436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27656 7181908 1% /mnt/lustre 1 203 3.95 MB/sec warmup 2 sec latency 172.391 ms 1 323 3.91 MB/sec warmup 3 sec latency 52.912 ms 1 441 3.86 MB/sec warmup 4 sec latency 65.136 ms 1 597 4.16 MB/sec warmup 5 sec latency 32.077 ms test_26 fail mds1 1 times Failing mds1 on oleg146-server 1 721 3.55 MB/sec warmup 6 sec latency 40.891 ms Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 1 789 3.07 MB/sec warmup 7 sec latency 262.516 ms 03:34:34 (1759822474) shut down facet: mds1 facet_host: oleg146-server facet_failover_host: oleg146-server 1 884 2.71 MB/sec warmup 8 sec latency 53.380 ms 1 974 2.42 MB/sec warmup 9 sec latency 452.660 ms 1 995 2.19 MB/sec warmup 10 sec latency 862.029 ms 1 995 1.99 MB/sec warmup 11 sec latency 1862.212 ms 1 995 1.83 MB/sec warmup 12 sec latency 2862.438 ms 1 995 1.69 MB/sec warmup 13 sec latency 3862.680 ms 1 995 1.56 MB/sec warmup 14 sec latency 4862.904 ms 1 995 1.46 MB/sec warmup 15 sec latency 5863.116 ms 1 995 1.37 MB/sec warmup 16 sec latency 6863.375 ms 1 995 1.29 MB/sec warmup 17 sec latency 7863.579 ms Failover mds1 to oleg146-server mount facets: mds1 1 995 1.22 MB/sec warmup 18 sec latency 8863.750 ms 1 995 1.15 MB/sec warmup 19 sec latency 9863.925 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.00 MB/sec execute 1 sec latency 11864.265 ms 1 995 0.00 MB/sec execute 2 sec latency 12864.452 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 1 995 0.00 MB/sec execute 3 sec latency 13864.651 ms Started lustre-MDT0000 03:34:49 (1759822489) targets are mounted 03:34:49 (1759822489) facet_failover done 1 995 0.00 MB/sec execute 4 sec latency 14864.825 ms 1 995 0.00 MB/sec execute 5 sec latency 15865.022 ms 1 995 0.00 MB/sec execute 6 sec latency 16865.211 ms 1 995 0.00 MB/sec execute 7 sec latency 17865.348 ms 1 1062 0.12 MB/sec execute 8 sec latency 18091.033 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1204 0.13 MB/sec execute 9 sec latency 50.694 ms 1 1431 0.41 MB/sec execute 10 sec latency 41.538 ms 1 1578 0.38 MB/sec execute 11 sec latency 146.881 ms 1 1731 0.36 MB/sec execute 12 sec latency 30.485 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2212 1285476 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 47460 3558108 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12212 3593120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 59672 7151228 1% /mnt/lustre 1 1902 0.35 MB/sec execute 13 sec latency 35.601 ms 1 2043 0.33 MB/sec execute 14 sec latency 36.046 ms 1 2185 0.32 MB/sec execute 15 sec latency 35.184 ms 1 2378 0.38 MB/sec execute 16 sec latency 26.504 ms 1 2489 0.40 MB/sec execute 17 sec latency 39.868 ms test_26 fail mds2 2 times 1 2768 0.56 MB/sec execute 18 sec latency 43.062 ms Failing mds2 on oleg146-server Stopping /mnt/lustre-mds2 (opts:) on oleg146-server 1 2993 0.61 MB/sec execute 19 sec latency 68.134 ms 03:35:06 (1759822506) shut down facet: mds2 facet_host: oleg146-server facet_failover_host: oleg146-server 1 3379 0.68 MB/sec execute 20 sec latency 32.395 ms 1 3695 0.80 MB/sec execute 21 sec latency 17.718 ms 1 3910 0.82 MB/sec execute 22 sec latency 20.610 ms 1 4083 0.79 MB/sec execute 23 sec latency 54.925 ms 1 4083 0.76 MB/sec execute 24 sec latency 1055.128 ms 1 4083 0.73 MB/sec execute 25 sec latency 2055.331 ms 1 4083 0.70 MB/sec execute 26 sec latency 3055.500 ms 1 4083 0.67 MB/sec execute 27 sec latency 4055.632 ms 1 4083 0.65 MB/sec execute 28 sec latency 5055.818 ms 1 4083 0.63 MB/sec execute 29 sec latency 6055.994 ms Failover mds2 to oleg146-server mount facets: mds2 1 4083 0.61 MB/sec execute 30 sec latency 7056.131 ms 1 4083 0.59 MB/sec execute 31 sec latency 8056.273 ms 1 4083 0.57 MB/sec execute 32 sec latency 9056.455 ms 1 4083 0.55 MB/sec execute 33 sec latency 10056.636 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4083 0.53 MB/sec execute 34 sec latency 11056.808 ms 1 4083 0.52 MB/sec execute 35 sec latency 12056.987 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0001 03:35:22 (1759822522) targets are mounted 03:35:22 (1759822522) facet_failover done 1 4083 0.50 MB/sec execute 36 sec latency 13057.171 ms 1 4083 0.49 MB/sec execute 37 sec latency 14057.297 ms 1 4083 0.48 MB/sec execute 38 sec latency 15057.493 ms 1 4083 0.47 MB/sec execute 39 sec latency 16057.654 ms 1 4160 0.45 MB/sec execute 40 sec latency 16387.104 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4302 0.45 MB/sec execute 41 sec latency 48.885 ms 1 4433 0.45 MB/sec execute 42 sec latency 41.325 ms 1 4550 0.44 MB/sec execute 43 sec latency 32.447 ms 1 4705 0.46 MB/sec execute 44 sec latency 37.824 ms 1 4814 0.45 MB/sec execute 45 sec latency 60.223 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2932 1284756 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2616 1285072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 56004 3547320 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12508 3592476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 68512 7139796 1% /mnt/lustre 1 5017 0.50 MB/sec execute 46 sec latency 50.664 ms 1 5151 0.50 MB/sec execute 47 sec latency 38.018 ms 1 5290 0.49 MB/sec execute 48 sec latency 47.173 ms 1 5414 0.48 MB/sec execute 49 sec latency 31.072 ms test_26 fail mds1 3 times 1 5575 0.47 MB/sec execute 50 sec latency 34.569 ms Failing mds1 on oleg146-server Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 1 5724 0.47 MB/sec execute 51 sec latency 33.883 ms 1 5820 0.46 MB/sec execute 52 sec latency 328.874 ms 03:35:38 (1759822538) shut down facet: mds1 facet_host: oleg146-server facet_failover_host: oleg146-server 1 5820 0.45 MB/sec execute 53 sec latency 1329.027 ms 1 5820 0.44 MB/sec execute 54 sec latency 2329.236 ms 1 5820 0.44 MB/sec execute 55 sec latency 3329.448 ms 1 5820 0.43 MB/sec execute 56 sec latency 4329.709 ms 1 5820 0.42 MB/sec execute 57 sec latency 5329.909 ms 1 5820 0.41 MB/sec execute 58 sec latency 6330.038 ms 1 5820 0.41 MB/sec execute 59 sec latency 7330.267 ms 1 5820 0.40 MB/sec execute 60 sec latency 8330.589 ms 1 5820 0.39 MB/sec execute 61 sec latency 9330.816 ms 1 5820 0.39 MB/sec execute 62 sec latency 10331.084 ms Failover mds1 to oleg146-server mount facets: mds1 1 5820 0.38 MB/sec execute 63 sec latency 11331.233 ms 1 5820 0.37 MB/sec execute 64 sec latency 12331.396 ms 1 5820 0.37 MB/sec execute 65 sec latency 13331.593 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5820 0.36 MB/sec execute 66 sec latency 14331.765 ms 1 5820 0.36 MB/sec execute 67 sec latency 15331.931 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 1 5820 0.35 MB/sec execute 68 sec latency 16332.101 ms Started lustre-MDT0000 03:35:54 (1759822554) targets are mounted 03:35:54 (1759822554) facet_failover done 1 5820 0.35 MB/sec execute 69 sec latency 17332.253 ms 1 5820 0.34 MB/sec execute 70 sec latency 18332.451 ms 1 5820 0.34 MB/sec execute 71 sec latency 19332.645 ms 1 5867 0.35 MB/sec execute 72 sec latency 20094.133 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5988 0.35 MB/sec execute 73 sec latency 53.243 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6144 0.35 MB/sec execute 74 sec latency 43.361 ms 1 6462 0.40 MB/sec execute 75 sec latency 38.197 ms 1 6759 0.41 MB/sec execute 76 sec latency 42.925 ms 1 6937 0.42 MB/sec execute 77 sec latency 36.070 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3408 1284280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3176 1284512 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 58128 3544112 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12824 3590764 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 70952 7134876 1% /mnt/lustre 1 7185 0.46 MB/sec execute 78 sec latency 58.671 ms 1 7332 0.47 MB/sec execute 79 sec latency 45.959 ms 1 7477 0.47 MB/sec execute 80 sec latency 41.212 ms 1 7593 0.46 MB/sec execute 81 sec latency 38.711 ms 1 7729 0.45 MB/sec execute 82 sec latency 42.794 ms test_26 fail mds2 4 times Failing mds2 on oleg146-server Stopping /mnt/lustre-mds2 (opts:) on oleg146-server 1 7880 0.45 MB/sec execute 83 sec latency 34.742 ms 1 7912 0.45 MB/sec execute 84 sec latency 753.020 ms 03:36:11 (1759822571) shut down facet: mds2 facet_host: oleg146-server facet_failover_host: oleg146-server 1 7912 0.44 MB/sec execute 85 sec latency 1753.283 ms 1 7912 0.44 MB/sec execute 86 sec latency 2753.513 ms 1 7912 0.43 MB/sec execute 87 sec latency 3753.751 ms 1 7912 0.43 MB/sec execute 88 sec latency 4754.015 ms 1 7912 0.42 MB/sec execute 89 sec latency 5754.193 ms 1 7912 0.42 MB/sec execute 90 sec latency 6754.426 ms 1 7912 0.42 MB/sec execute 91 sec latency 7754.627 ms 1 7912 0.41 MB/sec execute 92 sec latency 8754.781 ms 1 7912 0.41 MB/sec execute 93 sec latency 9754.979 ms 1 7912 0.40 MB/sec execute 94 sec latency 10755.136 ms Failover mds2 to oleg146-server mount facets: mds2 1 7912 0.40 MB/sec execute 95 sec latency 11755.252 ms 1 7912 0.39 MB/sec execute 96 sec latency 12755.499 ms 1 7912 0.39 MB/sec execute 97 sec latency 13755.687 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 7912 0.39 MB/sec execute 98 sec latency 14755.839 ms 1 7912 0.38 MB/sec execute 99 sec latency 15755.992 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 1 cleanup 100 sec Started lustre-MDT0001 03:36:26 (1759822586) targets are mounted 03:36:26 (1759822586) facet_failover done 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 cleanup 105 sec 0 cleanup 105 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1188 47.705 20094.118 Close 866 2.964 41.191 Rename 50 27.244 47.162 Unlink 242 10.149 58.657 Qpathinfo 1118 19.718 16387.077 Qfileinfo 191 0.955 6.344 Qfsinfo 200 0.560 5.728 Sfileinfo 90 248.723 20732.004 Find 416 5.232 24.903 WriteX 587 4.647 214.710 ReadX 1883 0.417 106.882 LockX 4 2.502 3.145 UnlockX 4 2.737 3.284 Flush 79 14.304 48.872 Throughput 0.381797 MB/sec 1 clients 1 procs max_latency=20094.133 ms stopping dbench on /mnt/lustre at Tue Oct 7 03:36:32 EDT 2025 with return code 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Oct 7 03:36:34 EDT 2025 waiting for dbench pid 69127 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 121 5.50 MB/sec warmup 1 sec latency 82.329 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3916 1283772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3740 1283948 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 49032 3553324 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5432 3600276 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 54464 7153600 1% /mnt/lustre 1 207 4.05 MB/sec warmup 2 sec latency 163.581 ms 1 316 3.83 MB/sec warmup 3 sec latency 82.098 ms 1 448 3.91 MB/sec warmup 4 sec latency 35.930 ms 1 612 4.19 MB/sec warmup 5 sec latency 36.995 ms 1 735 3.55 MB/sec warmup 6 sec latency 130.702 ms test_26 fail mds1 5 times Failing mds1 on oleg146-server 1 863 3.08 MB/sec warmup 7 sec latency 69.883 ms Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 1 885 2.71 MB/sec warmup 8 sec latency 435.182 ms 03:36:43 (1759822603) shut down facet: mds1 facet_host: oleg146-server facet_failover_host: oleg146-server 1 885 2.41 MB/sec warmup 9 sec latency 1378.005 ms 1 885 2.17 MB/sec warmup 10 sec latency 2378.260 ms 1 885 1.97 MB/sec warmup 11 sec latency 3378.461 ms 1 885 1.81 MB/sec warmup 12 sec latency 4378.683 ms 1 885 1.67 MB/sec warmup 13 sec latency 5378.889 ms 1 885 1.55 MB/sec warmup 14 sec latency 6379.108 ms 1 885 1.44 MB/sec warmup 15 sec latency 7379.303 ms 1 885 1.35 MB/sec warmup 16 sec latency 8379.487 ms 1 885 1.27 MB/sec warmup 17 sec latency 9379.670 ms 1 885 1.20 MB/sec warmup 18 sec latency 10379.861 ms Failover mds1 to oleg146-server mount facets: mds1 1 885 1.14 MB/sec warmup 19 sec latency 11380.000 ms 1 885 0.00 MB/sec execute 1 sec latency 13380.309 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 885 0.00 MB/sec execute 2 sec latency 14380.540 ms 1 885 0.00 MB/sec execute 3 sec latency 15380.759 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 1 885 0.00 MB/sec execute 4 sec latency 16380.886 ms Started lustre-MDT0000 03:36:59 (1759822619) targets are mounted 03:36:59 (1759822619) facet_failover done 1 885 0.00 MB/sec execute 5 sec latency 17381.139 ms 1 885 0.00 MB/sec execute 6 sec latency 18381.337 ms 1 885 0.00 MB/sec execute 7 sec latency 19381.508 ms 1 885 0.00 MB/sec execute 8 sec latency 20381.686 ms 1 927 0.01 MB/sec execute 9 sec latency 20815.740 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1068 0.12 MB/sec execute 10 sec latency 37.362 ms 1 1211 0.13 MB/sec execute 11 sec latency 35.242 ms 1 1380 0.36 MB/sec execute 12 sec latency 58.386 ms 1 1489 0.34 MB/sec execute 13 sec latency 201.389 ms 1 1610 0.32 MB/sec execute 14 sec latency 30.260 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3676 1284012 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3436 1284252 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 58780 3548212 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13772 3586376 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 72552 7134588 2% /mnt/lustre 1 1711 0.30 MB/sec execute 15 sec latency 175.749 ms 1 1877 0.29 MB/sec execute 16 sec latency 31.281 ms 1 2033 0.29 MB/sec execute 17 sec latency 57.948 ms 1 2193 0.28 MB/sec execute 18 sec latency 42.908 ms 1 2377 0.33 MB/sec execute 19 sec latency 90.216 ms test_26 fail mds2 6 times Failing mds2 on oleg146-server 1 2489 0.35 MB/sec execute 20 sec latency 43.915 ms Stopping /mnt/lustre-mds2 (opts:) on oleg146-server 1 2586 0.35 MB/sec execute 21 sec latency 555.631 ms 03:37:16 (1759822636) shut down facet: mds2 facet_host: oleg146-server facet_failover_host: oleg146-server 1 2586 0.33 MB/sec execute 22 sec latency 1555.835 ms 1 2586 0.32 MB/sec execute 23 sec latency 2556.072 ms 1 2586 0.30 MB/sec execute 24 sec latency 3556.358 ms 1 2586 0.29 MB/sec execute 25 sec latency 4556.603 ms 1 2586 0.28 MB/sec execute 26 sec latency 5556.836 ms 1 2586 0.27 MB/sec execute 27 sec latency 6557.031 ms 1 2586 0.26 MB/sec execute 28 sec latency 7557.211 ms 1 2586 0.25 MB/sec execute 29 sec latency 8557.424 ms 1 2586 0.24 MB/sec execute 30 sec latency 9557.609 ms 1 2586 0.24 MB/sec execute 31 sec latency 10557.744 ms Failover mds2 to oleg146-server mount facets: mds2 1 2586 0.23 MB/sec execute 32 sec latency 11557.901 ms 1 2586 0.22 MB/sec execute 33 sec latency 12558.137 ms 1 2586 0.21 MB/sec execute 34 sec latency 13558.332 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2586 0.21 MB/sec execute 35 sec latency 14558.559 ms 1 2586 0.20 MB/sec execute 36 sec latency 15558.792 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 1 2586 0.20 MB/sec execute 37 sec latency 16558.940 ms Started lustre-MDT0001 03:37:32 (1759822652) targets are mounted 03:37:32 (1759822652) facet_failover done 1 2586 0.19 MB/sec execute 38 sec latency 17559.123 ms 1 2586 0.19 MB/sec execute 39 sec latency 18559.319 ms 1 2586 0.18 MB/sec execute 40 sec latency 19559.461 ms 1 2586 0.18 MB/sec execute 41 sec latency 20559.593 ms 1 2749 0.24 MB/sec execute 42 sec latency 21105.279 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 2984 0.28 MB/sec execute 43 sec latency 53.410 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3278 0.31 MB/sec execute 44 sec latency 41.159 ms 1 3557 0.37 MB/sec execute 45 sec latency 26.839 ms 1 3729 0.38 MB/sec execute 46 sec latency 32.447 ms 1 3847 0.38 MB/sec execute 47 sec latency 66.373 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2940 1284748 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2524 1285164 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 56748 3549992 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13540 3592092 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 70288 7142084 1% /mnt/lustre 1 3920 0.38 MB/sec execute 48 sec latency 98.419 ms 1 4039 0.38 MB/sec execute 49 sec latency 39.956 ms 1 4227 0.37 MB/sec execute 50 sec latency 30.861 ms 1 4349 0.37 MB/sec execute 51 sec latency 41.749 ms 1 4472 0.37 MB/sec execute 52 sec latency 53.614 ms test_26 fail mds1 7 times 1 4569 0.37 MB/sec execute 53 sec latency 123.049 ms Failing mds1 on oleg146-server Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 1 4716 0.38 MB/sec execute 54 sec latency 181.312 ms 03:37:49 (1759822669) shut down facet: mds1 facet_host: oleg146-server facet_failover_host: oleg146-server 1 4716 0.37 MB/sec execute 55 sec latency 1181.519 ms 1 4716 0.37 MB/sec execute 56 sec latency 2181.725 ms 1 4716 0.36 MB/sec execute 57 sec latency 3181.897 ms 1 4716 0.35 MB/sec execute 58 sec latency 4182.066 ms 1 4716 0.35 MB/sec execute 59 sec latency 5182.290 ms 1 4716 0.34 MB/sec execute 60 sec latency 6182.458 ms 1 4716 0.34 MB/sec execute 61 sec latency 7182.680 ms 1 4716 0.33 MB/sec execute 62 sec latency 8182.872 ms 1 4716 0.32 MB/sec execute 63 sec latency 9183.060 ms 1 4716 0.32 MB/sec execute 64 sec latency 10183.270 ms Failover mds1 to oleg146-server 1 4716 0.31 MB/sec execute 65 sec latency 11183.412 ms mount facets: mds1 1 4716 0.31 MB/sec execute 66 sec latency 12183.578 ms 1 4716 0.31 MB/sec execute 67 sec latency 13183.853 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4716 0.30 MB/sec execute 68 sec latency 14183.997 ms 1 4716 0.30 MB/sec execute 69 sec latency 15184.179 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all 1 4716 0.29 MB/sec execute 70 sec latency 16184.320 ms pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 03:38:05 (1759822685) targets are mounted 03:38:05 (1759822685) facet_failover done 1 4716 0.29 MB/sec execute 71 sec latency 17184.494 ms 1 4716 0.28 MB/sec execute 72 sec latency 18184.652 ms 1 4716 0.28 MB/sec execute 73 sec latency 19184.838 ms 1 4716 0.28 MB/sec execute 74 sec latency 20185.022 ms 1 4739 0.27 MB/sec execute 75 sec latency 20938.042 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4946 0.31 MB/sec execute 76 sec latency 35.077 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5058 0.31 MB/sec execute 77 sec latency 62.704 ms striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar 1 5174 0.30 MB/sec execute 78 sec latency 59.248 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Oct 7 03:38:13 EDT 2025 with return code 0 69127 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 69127 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished