== replay-dual test 26: dbench and tar with mds failover ========================================================== 22:00:02 (1749607202) Starting client oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre Started clients oleg448-client.virtnet: 192.168.204.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 15559 Started dbench loop with 15560 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 10 22:00:03 EDT 2025 waiting for dbench pid 15591 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 199 7.84 MB/sec warmup 1 sec latency 32.744 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3707904 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3698688 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7406592 1% /mnt/lustre 1 382 6.78 MB/sec warmup 2 sec latency 30.232 ms 1 616 6.99 MB/sec warmup 3 sec latency 31.829 ms 1 789 5.36 MB/sec warmup 4 sec latency 49.502 ms test_26 fail mds1 1 times Failing mds1 on oleg448-server 1 957 4.36 MB/sec warmup 5 sec latency 43.324 ms Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 1 1045 3.82 MB/sec warmup 6 sec latency 698.222 ms 22:00:09 (1749607209) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server 1 1045 3.27 MB/sec warmup 7 sec latency 1698.502 ms 1 1045 2.86 MB/sec warmup 8 sec latency 2698.675 ms 1 1045 2.54 MB/sec warmup 9 sec latency 3698.962 ms 1 1045 2.29 MB/sec warmup 10 sec latency 4699.215 ms 1 1045 2.08 MB/sec warmup 11 sec latency 5699.471 ms 1 1045 1.91 MB/sec warmup 12 sec latency 6699.776 ms 1 1045 1.76 MB/sec warmup 13 sec latency 7700.033 ms 1 1045 1.63 MB/sec warmup 14 sec latency 8700.288 ms 1 1045 1.53 MB/sec warmup 15 sec latency 9700.526 ms 1 1045 1.43 MB/sec warmup 16 sec latency 10700.809 ms Failover mds1 to oleg448-server mount facets: mds1 1 1045 1.35 MB/sec warmup 17 sec latency 11700.999 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1045 1.27 MB/sec warmup 18 sec latency 12701.178 ms oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all 1 1045 1.20 MB/sec warmup 19 sec latency 13701.307 ms pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 22:00:23 (1749607223) targets are mounted 22:00:23 (1749607223) facet_failover done 1 1045 0.00 MB/sec execute 1 sec latency 15701.736 ms 1 1045 0.00 MB/sec execute 2 sec latency 16701.941 ms 1 1045 0.00 MB/sec execute 3 sec latency 17702.097 ms 1 1045 0.00 MB/sec execute 4 sec latency 18702.250 ms 1 1045 0.00 MB/sec execute 5 sec latency 19702.376 ms 1 1045 0.00 MB/sec execute 6 sec latency 20702.503 ms 1 1133 0.01 MB/sec execute 7 sec latency 21426.160 ms oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1467 0.39 MB/sec execute 8 sec latency 34.318 ms 1 1700 0.37 MB/sec execute 9 sec latency 40.041 ms 1 2030 0.37 MB/sec execute 10 sec latency 46.258 ms 1 2361 0.46 MB/sec execute 11 sec latency 46.211 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4736 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 19456 3644416 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 36864 3577856 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7222272 1% /mnt/lustre 1 2758 0.76 MB/sec execute 12 sec latency 43.113 ms 1 3246 0.91 MB/sec execute 13 sec latency 33.829 ms 1 3666 1.13 MB/sec execute 14 sec latency 16.986 ms test_26 fail mds1 2 times Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 1 3916 1.14 MB/sec execute 15 sec latency 18.076 ms 22:00:39 (1749607239) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server 1 3939 1.07 MB/sec execute 16 sec latency 908.024 ms 1 3939 1.01 MB/sec execute 17 sec latency 1908.174 ms 1 3939 0.95 MB/sec execute 18 sec latency 2908.319 ms 1 3939 0.90 MB/sec execute 19 sec latency 3908.495 ms 1 3939 0.86 MB/sec execute 20 sec latency 4908.621 ms 1 3939 0.82 MB/sec execute 21 sec latency 5908.748 ms 1 3939 0.78 MB/sec execute 22 sec latency 6908.932 ms 1 3939 0.75 MB/sec execute 23 sec latency 7909.056 ms 1 3939 0.72 MB/sec execute 24 sec latency 8909.224 ms 1 3939 0.69 MB/sec execute 25 sec latency 9909.378 ms Failover mds1 to oleg448-server mount facets: mds1 1 3939 0.66 MB/sec execute 26 sec latency 10909.492 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3939 0.64 MB/sec execute 27 sec latency 11909.608 ms 1 3939 0.61 MB/sec execute 28 sec latency 12909.767 ms oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 22:00:52 (1749607252) targets are mounted 22:00:52 (1749607252) facet_failover done 1 3939 0.59 MB/sec execute 29 sec latency 13909.897 ms 1 3939 0.57 MB/sec execute 30 sec latency 14910.042 ms 1 3939 0.55 MB/sec execute 31 sec latency 15910.230 ms 1 3939 0.54 MB/sec execute 32 sec latency 16910.382 ms 1 3939 0.52 MB/sec execute 33 sec latency 17910.532 ms 1 3939 0.51 MB/sec execute 34 sec latency 18910.690 ms 1 3939 0.49 MB/sec execute 35 sec latency 19910.832 ms 1 3939 0.48 MB/sec execute 36 sec latency 20910.953 ms 1 3941 0.46 MB/sec execute 37 sec latency 21905.433 ms oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4149 0.45 MB/sec execute 38 sec latency 21.880 ms 1 4369 0.45 MB/sec execute 39 sec latency 49.216 ms 1 4577 0.46 MB/sec execute 40 sec latency 47.005 ms 1 4825 0.47 MB/sec execute 41 sec latency 22.380 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6144 2202368 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3531776 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 39936 3499008 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 63488 7030784 1% /mnt/lustre 1 5145 0.53 MB/sec execute 42 sec latency 43.988 ms 1 5397 0.53 MB/sec execute 43 sec latency 38.851 ms 1 5670 0.52 MB/sec execute 44 sec latency 42.142 ms test_26 fail mds1 3 times Failing mds1 on oleg448-server 1 5966 0.54 MB/sec execute 45 sec latency 45.881 ms Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 1 6041 0.54 MB/sec execute 46 sec latency 642.213 ms 22:01:09 (1749607269) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server 1 6041 0.53 MB/sec execute 47 sec latency 1642.317 ms 1 6041 0.52 MB/sec execute 48 sec latency 2642.451 ms 1 6041 0.51 MB/sec execute 49 sec latency 3642.570 ms 1 6041 0.50 MB/sec execute 50 sec latency 4642.703 ms 1 6041 0.49 MB/sec execute 51 sec latency 5642.820 ms 1 6041 0.48 MB/sec execute 52 sec latency 6642.954 ms 1 6041 0.47 MB/sec execute 53 sec latency 7643.071 ms 1 6041 0.46 MB/sec execute 54 sec latency 8643.183 ms 1 6041 0.45 MB/sec execute 55 sec latency 9643.361 ms 1 6041 0.45 MB/sec execute 56 sec latency 10643.495 ms Failover mds1 to oleg448-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 6041 0.44 MB/sec execute 57 sec latency 11643.620 ms 1 6041 0.43 MB/sec execute 58 sec latency 12643.720 ms oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 1 6041 0.42 MB/sec execute 59 sec latency 13643.825 ms Started lustre-MDT0000 22:01:22 (1749607282) targets are mounted 22:01:22 (1749607282) facet_failover done 1 6041 0.42 MB/sec execute 60 sec latency 14643.939 ms 1 6041 0.41 MB/sec execute 61 sec latency 15644.050 ms 1 6041 0.40 MB/sec execute 62 sec latency 16644.162 ms oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6486 0.47 MB/sec execute 63 sec latency 16748.690 ms 1 7053 0.54 MB/sec execute 64 sec latency 33.578 ms 1 7349 0.55 MB/sec execute 65 sec latency 18.784 ms 1 7570 0.55 MB/sec execute 66 sec latency 14.742 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 7680 2200832 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3535872 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 41984 3514368 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 65536 7050240 1% /mnt/lustre 1 7822 0.55 MB/sec execute 67 sec latency 77.087 ms 1 8047 0.55 MB/sec execute 68 sec latency 48.935 ms 1 8339 0.56 MB/sec execute 69 sec latency 38.041 ms test_26 fail mds1 4 times 1 8661 0.59 MB/sec execute 70 sec latency 49.541 ms Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 1 8786 0.59 MB/sec execute 71 sec latency 482.085 ms 22:01:34 (1749607294) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server 1 8786 0.58 MB/sec execute 72 sec latency 1482.231 ms 1 8786 0.57 MB/sec execute 73 sec latency 2482.406 ms 1 8786 0.56 MB/sec execute 74 sec latency 3482.567 ms 1 8786 0.55 MB/sec execute 75 sec latency 4482.693 ms 1 8786 0.55 MB/sec execute 76 sec latency 5482.834 ms 1 8786 0.54 MB/sec execute 77 sec latency 6482.959 ms 1 8786 0.53 MB/sec execute 78 sec latency 7483.167 ms 1 8786 0.53 MB/sec execute 79 sec latency 8483.297 ms 1 8786 0.52 MB/sec execute 80 sec latency 9483.418 ms 1 8786 0.51 MB/sec execute 81 sec latency 10483.566 ms Failover mds1 to oleg448-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 8786 0.51 MB/sec execute 82 sec latency 11483.679 ms 1 8786 0.50 MB/sec execute 83 sec latency 12483.820 ms oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 22:01:47 (1749607307) targets are mounted 22:01:47 (1749607307) facet_failover done 1 8786 0.49 MB/sec execute 84 sec latency 13483.961 ms 1 8786 0.49 MB/sec execute 85 sec latency 14484.096 ms 1 8786 0.48 MB/sec execute 86 sec latency 15484.190 ms 1 8786 0.48 MB/sec execute 87 sec latency 16484.285 ms oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 8933 0.47 MB/sec execute 88 sec latency 16802.149 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 9216 0.47 MB/sec execute 89 sec latency 34.906 ms 1 9493 0.48 MB/sec execute 90 sec latency 51.445 ms 1 9818 0.52 MB/sec execute 91 sec latency 44.388 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 9216 2199296 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3300352 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 43008 3273728 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 6574080 2% /mnt/lustre 1 10296 0.54 MB/sec execute 92 sec latency 51.314 ms 1 10732 0.58 MB/sec execute 93 sec latency 33.372 ms 1 10959 0.59 MB/sec execute 94 sec latency 17.223 ms test_26 fail mds1 5 times 1 11155 0.58 MB/sec execute 95 sec latency 20.210 ms Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 1 11349 0.58 MB/sec execute 96 sec latency 219.366 ms 22:02:00 (1749607320) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server 1 11349 0.57 MB/sec execute 97 sec latency 1219.501 ms 1 11349 0.57 MB/sec execute 98 sec latency 2219.645 ms 1 11349 0.56 MB/sec execute 99 sec latency 3219.806 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec Failover mds1 to oleg448-server mount facets: mds1 1 cleanup 107 sec Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 108 sec oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all 1 cleanup 109 sec pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 22:02:13 (1749607333) targets are mounted 22:02:13 (1749607333) facet_failover done 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec 1 cleanup 116 sec 1 cleanup 117 sec oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 0 cleanup 118 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1777 33.836 21905.425 Close 1298 44.183 21426.144 Rename 74 15.460 23.359 Unlink 363 4.899 28.249 Qpathinfo 1664 2.882 30.337 Qfileinfo 283 0.365 3.156 Qfsinfo 293 0.440 2.727 Sfileinfo 135 8.773 14.337 Find 627 1.101 30.784 WriteX 855 1.748 7.217 ReadX 2813 0.121 29.516 LockX 6 1.680 2.263 UnlockX 6 1.750 2.245 Flush 111 23.912 85.177 Throughput 0.562092 MB/sec 1 clients 1 procs max_latency=21905.433 ms stopping dbench on /mnt/lustre at Tue Jun 10 22:02:21 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 10 22:02:22 EDT 2025 waiting for dbench pid 20780 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 276 10.31 MB/sec warmup 1 sec latency 24.568 ms 1 542 9.55 MB/sec warmup 2 sec latency 21.383 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10496 2198016 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 47104 3505152 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3449856 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 69632 6955008 1% /mnt/lustre 1 809 7.16 MB/sec warmup 3 sec latency 60.412 ms 1 1099 5.74 MB/sec warmup 4 sec latency 29.939 ms 1 1446 5.20 MB/sec warmup 5 sec latency 19.870 ms test_26 fail mds1 6 times 1 1721 4.38 MB/sec warmup 6 sec latency 47.482 ms Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 1 1898 3.77 MB/sec warmup 7 sec latency 445.546 ms 22:02:29 (1749607349) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server 1 1898 3.30 MB/sec warmup 8 sec latency 1445.646 ms 1 1898 2.93 MB/sec warmup 9 sec latency 2445.781 ms 1 1898 2.64 MB/sec warmup 10 sec latency 3445.913 ms 1 1898 2.40 MB/sec warmup 11 sec latency 4446.040 ms 1 1898 2.20 MB/sec warmup 12 sec latency 5446.164 ms 1 1898 2.03 MB/sec warmup 13 sec latency 6446.284 ms 1 1898 1.89 MB/sec warmup 14 sec latency 7446.401 ms 1 1898 1.76 MB/sec warmup 15 sec latency 8446.517 ms 1 1898 1.65 MB/sec warmup 16 sec latency 9446.659 ms 1 1898 1.55 MB/sec warmup 17 sec latency 10446.784 ms Failover mds1 to oleg448-server mount facets: mds1 1 1898 1.47 MB/sec warmup 18 sec latency 11446.906 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1898 1.39 MB/sec warmup 19 sec latency 12447.018 ms oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 22:02:42 (1749607362) targets are mounted 22:02:42 (1749607362) facet_failover done 1 1898 0.00 MB/sec execute 1 sec latency 14447.190 ms 1 1898 0.00 MB/sec execute 2 sec latency 15447.278 ms 1 1898 0.00 MB/sec execute 3 sec latency 16447.429 ms oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2083 0.06 MB/sec execute 4 sec latency 16880.699 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2424 0.34 MB/sec execute 5 sec latency 36.911 ms 1 2906 1.07 MB/sec execute 6 sec latency 49.477 ms 1 3378 1.29 MB/sec execute 7 sec latency 43.442 ms 1 3785 1.67 MB/sec execute 8 sec latency 16.505 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 11392 2197120 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 49152 3473408 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3484672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 72704 6958080 2% /mnt/lustre 1 4014 1.52 MB/sec execute 9 sec latency 17.526 ms 1 4266 1.39 MB/sec execute 10 sec latency 51.743 ms test_26 fail mds1 7 times 1 4483 1.32 MB/sec execute 11 sec latency 50.828 ms Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 1 4701 1.31 MB/sec execute 12 sec latency 249.775 ms 22:02:55 (1749607375) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server 1 4701 1.21 MB/sec execute 13 sec latency 1249.877 ms 1 4701 1.12 MB/sec execute 14 sec latency 2250.017 ms 1 4701 1.05 MB/sec execute 15 sec latency 3250.134 ms 1 4701 0.98 MB/sec execute 16 sec latency 4250.252 ms 1 4701 0.92 MB/sec execute 17 sec latency 5250.391 ms 1 4701 0.87 MB/sec execute 18 sec latency 6250.511 ms 1 4701 0.83 MB/sec execute 19 sec latency 7250.685 ms 1 4701 0.79 MB/sec execute 20 sec latency 8250.821 ms 1 4701 0.75 MB/sec execute 21 sec latency 9250.959 ms 1 4701 0.71 MB/sec execute 22 sec latency 10251.077 ms Failover mds1 to oleg448-server mount facets: mds1 1 4701 0.68 MB/sec execute 23 sec latency 11251.190 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4701 0.65 MB/sec execute 24 sec latency 12251.300 ms oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 22:03:07 (1749607387) targets are mounted 22:03:07 (1749607387) facet_failover done 1 4701 0.63 MB/sec execute 25 sec latency 13251.411 ms 1 4701 0.60 MB/sec execute 26 sec latency 14251.546 ms 1 4701 0.58 MB/sec execute 27 sec latency 15251.630 ms 1 4701 0.56 MB/sec execute 28 sec latency 16251.721 ms oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5054 0.65 MB/sec execute 29 sec latency 16263.131 ms 1 5326 0.63 MB/sec execute 30 sec latency 61.100 ms 1 5581 0.62 MB/sec execute 31 sec latency 53.655 ms 1 5933 0.64 MB/sec execute 32 sec latency 54.263 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 8448 2199936 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 51200 3680256 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3705856 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7386112 1% /mnt/lustre 1 6320 0.75 MB/sec execute 33 sec latency 51.088 ms 1 6860 0.83 MB/sec execute 34 sec latency 42.775 ms 1 7269 0.90 MB/sec execute 35 sec latency 14.566 ms test_26 fail mds1 8 times Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 1 7534 0.91 MB/sec execute 36 sec latency 14.083 ms 22:03:19 (1749607399) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server 1 7588 0.89 MB/sec execute 37 sec latency 784.852 ms 1 7588 0.86 MB/sec execute 38 sec latency 1784.988 ms 1 7588 0.84 MB/sec execute 39 sec latency 2785.119 ms 1 7588 0.82 MB/sec execute 40 sec latency 3785.317 ms 1 7588 0.80 MB/sec execute 41 sec latency 4785.553 ms 1 7588 0.78 MB/sec execute 42 sec latency 5785.799 ms 1 7588 0.76 MB/sec execute 43 sec latency 6786.025 ms 1 7588 0.75 MB/sec execute 44 sec latency 7786.256 ms 1 7588 0.73 MB/sec execute 45 sec latency 8786.529 ms 1 7588 0.71 MB/sec execute 46 sec latency 9786.754 ms Failover mds1 to oleg448-server mount facets: mds1 1 7588 0.70 MB/sec execute 47 sec latency 10786.930 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7588 0.68 MB/sec execute 48 sec latency 11787.054 ms 1 7588 0.67 MB/sec execute 49 sec latency 12787.204 ms oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 22:03:32 (1749607412) targets are mounted 1 7588 0.66 MB/sec execute 50 sec latency 13787.372 ms 22:03:32 (1749607412) facet_failover done 1 7588 0.64 MB/sec execute 51 sec latency 14787.550 ms 1 7588 0.63 MB/sec execute 52 sec latency 15787.737 ms 1 7588 0.62 MB/sec execute 53 sec latency 16787.963 ms 1 7590 0.61 MB/sec execute 54 sec latency 17783.725 ms oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7790 0.60 MB/sec execute 55 sec latency 86.089 ms 1 7963 0.60 MB/sec execute 56 sec latency 66.356 ms 1 8099 0.59 MB/sec execute 57 sec latency 50.840 ms 1 8300 0.60 MB/sec execute 58 sec latency 25.383 ms 1 8537 0.64 MB/sec execute 59 sec latency 30.315 ms 1 8688 0.63 MB/sec execute 60 sec latency 69.868 ms 1 8837 0.62 MB/sec execute 61 sec latency 69.894 ms 1 8993 0.62 MB/sec execute 62 sec latency 47.350 ms 1 9189 0.61 MB/sec execute 63 sec latency 58.439 ms 1 9430 0.62 MB/sec execute 64 sec latency 21.357 ms 1 9698 0.63 MB/sec execute 65 sec latency 52.412 ms 1 10330 0.70 MB/sec execute 66 sec latency 43.218 ms 1 10783 0.76 MB/sec execute 67 sec latency 12.719 ms 1 11049 0.76 MB/sec execute 68 sec latency 15.625 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Jun 10 22:03:51 EDT 2025 with return code 0 20780 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 20780 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished