== replay-dual test 26: dbench and tar with mds failover ========================================================== 13:16:42 (1750180602) Starting client oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre Started clients oleg203-client.virtnet: 192.168.202.103@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 15642 Started dbench loop with 15643 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 17 13:16:43 EDT 2025 waiting for dbench pid 15674 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3724288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7461888 1% /mnt/lustre 1 155 6.54 MB/sec warmup 1 sec latency 66.087 ms 1 293 5.38 MB/sec warmup 2 sec latency 46.034 ms 1 445 5.21 MB/sec warmup 3 sec latency 34.961 ms 1 639 5.26 MB/sec warmup 4 sec latency 41.199 ms test_26 fail mds1 1 times Failing mds1 on oleg203-server 1 721 4.26 MB/sec warmup 5 sec latency 124.195 ms Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 759 3.56 MB/sec warmup 6 sec latency 477.396 ms 13:16:50 (1750180610) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 759 3.05 MB/sec warmup 7 sec latency 1477.562 ms 1 759 2.67 MB/sec warmup 8 sec latency 2477.782 ms 1 759 2.37 MB/sec warmup 9 sec latency 3477.992 ms 1 759 2.13 MB/sec warmup 10 sec latency 4478.209 ms 1 759 1.94 MB/sec warmup 11 sec latency 5478.461 ms 1 759 1.78 MB/sec warmup 12 sec latency 6478.691 ms 1 759 1.64 MB/sec warmup 13 sec latency 7478.893 ms 1 759 1.52 MB/sec warmup 14 sec latency 8479.111 ms 1 759 1.42 MB/sec warmup 15 sec latency 9479.337 ms 1 759 1.33 MB/sec warmup 16 sec latency 10479.515 ms Failover mds1 to oleg203-server mount facets: mds1 1 759 1.26 MB/sec warmup 17 sec latency 11479.826 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 759 1.19 MB/sec warmup 18 sec latency 12480.917 ms 1 759 1.12 MB/sec warmup 19 sec latency 13481.549 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:17:04 (1750180624) targets are mounted 13:17:04 (1750180624) facet_failover done 1 759 0.00 MB/sec execute 1 sec latency 15481.893 ms 1 759 0.00 MB/sec execute 2 sec latency 16482.089 ms 1 759 0.00 MB/sec execute 3 sec latency 17482.281 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 884 0.08 MB/sec execute 4 sec latency 17649.311 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 996 0.11 MB/sec execute 5 sec latency 67.811 ms 1 1172 0.28 MB/sec execute 6 sec latency 41.766 ms 1 1450 0.67 MB/sec execute 7 sec latency 22.882 ms 1 1634 0.60 MB/sec execute 8 sec latency 58.124 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4352 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3597312 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 12288 3641344 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 49152 7238656 1% /mnt/lustre 1 1861 0.56 MB/sec execute 9 sec latency 26.231 ms 1 2060 0.53 MB/sec execute 10 sec latency 38.494 ms 1 2272 0.53 MB/sec execute 11 sec latency 26.632 ms test_26 fail mds1 2 times 1 2435 0.56 MB/sec execute 12 sec latency 86.700 ms Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 2592 0.59 MB/sec execute 13 sec latency 95.091 ms 13:17:17 (1750180637) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 2592 0.55 MB/sec execute 14 sec latency 1095.376 ms 1 2592 0.51 MB/sec execute 15 sec latency 2095.586 ms 1 2592 0.48 MB/sec execute 16 sec latency 3095.784 ms 1 2592 0.45 MB/sec execute 17 sec latency 4096.418 ms 1 2592 0.43 MB/sec execute 18 sec latency 5096.864 ms 1 2592 0.40 MB/sec execute 19 sec latency 6097.291 ms 1 2592 0.38 MB/sec execute 20 sec latency 7097.541 ms 1 2592 0.36 MB/sec execute 21 sec latency 8097.835 ms 1 2592 0.35 MB/sec execute 22 sec latency 9098.019 ms 1 2592 0.33 MB/sec execute 23 sec latency 10098.208 ms Failover mds1 to oleg203-server mount facets: mds1 1 2592 0.32 MB/sec execute 24 sec latency 11098.717 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2592 0.31 MB/sec execute 25 sec latency 12098.883 ms 1 2592 0.29 MB/sec execute 26 sec latency 13099.119 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 1 2592 0.28 MB/sec execute 27 sec latency 14099.338 ms Started lustre-MDT0000 13:17:31 (1750180651) targets are mounted 13:17:31 (1750180651) facet_failover done 1 2592 0.27 MB/sec execute 28 sec latency 15100.234 ms 1 2592 0.26 MB/sec execute 29 sec latency 16100.483 ms 1 2592 0.26 MB/sec execute 30 sec latency 17100.859 ms 1 2592 0.25 MB/sec execute 31 sec latency 18101.207 ms 1 2592 0.24 MB/sec execute 32 sec latency 19101.410 ms 1 2592 0.23 MB/sec execute 33 sec latency 20101.629 ms 1 2764 0.31 MB/sec execute 34 sec latency 20580.244 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2993 0.35 MB/sec execute 35 sec latency 81.958 ms 1 3342 0.39 MB/sec execute 36 sec latency 67.910 ms 1 3626 0.47 MB/sec execute 37 sec latency 24.886 ms 1 3780 0.48 MB/sec execute 38 sec latency 37.306 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3535872 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3567616 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 59392 7103488 1% /mnt/lustre 1 3884 0.47 MB/sec execute 39 sec latency 42.339 ms 1 4006 0.47 MB/sec execute 40 sec latency 42.315 ms 1 4117 0.46 MB/sec execute 41 sec latency 36.665 ms 1 4245 0.45 MB/sec execute 42 sec latency 112.450 ms test_26 fail mds1 3 times Failing mds1 on oleg203-server 1 4330 0.44 MB/sec execute 43 sec latency 72.318 ms Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 4370 0.44 MB/sec execute 44 sec latency 465.226 ms 13:17:48 (1750180668) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 4370 0.43 MB/sec execute 45 sec latency 1465.431 ms 1 4370 0.42 MB/sec execute 46 sec latency 2465.615 ms 1 4370 0.41 MB/sec execute 47 sec latency 3466.132 ms 1 4370 0.40 MB/sec execute 48 sec latency 4466.501 ms 1 4370 0.39 MB/sec execute 49 sec latency 5467.686 ms 1 4370 0.38 MB/sec execute 50 sec latency 6468.364 ms 1 4370 0.38 MB/sec execute 51 sec latency 7468.688 ms 1 4370 0.37 MB/sec execute 52 sec latency 8468.980 ms 1 4370 0.36 MB/sec execute 53 sec latency 9469.975 ms 1 4370 0.36 MB/sec execute 54 sec latency 10470.165 ms Failover mds1 to oleg203-server mount facets: mds1 1 4370 0.35 MB/sec execute 55 sec latency 11470.462 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4370 0.34 MB/sec execute 56 sec latency 12471.436 ms 1 4370 0.34 MB/sec execute 57 sec latency 13471.657 ms 1 4370 0.33 MB/sec execute 58 sec latency 14471.853 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:18:02 (1750180682) targets are mounted 13:18:02 (1750180682) facet_failover done 1 4370 0.33 MB/sec execute 59 sec latency 15472.159 ms 1 4370 0.32 MB/sec execute 60 sec latency 16472.740 ms 1 4370 0.32 MB/sec execute 61 sec latency 17473.005 ms 1 4370 0.31 MB/sec execute 62 sec latency 18473.371 ms 1 4370 0.31 MB/sec execute 63 sec latency 19473.559 ms 1 4395 0.30 MB/sec execute 64 sec latency 20155.530 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4523 0.30 MB/sec execute 65 sec latency 74.063 ms 1 4680 0.31 MB/sec execute 66 sec latency 89.090 ms 1 4812 0.31 MB/sec execute 67 sec latency 49.815 ms 1 5011 0.35 MB/sec execute 68 sec latency 42.155 ms 1 5117 0.35 MB/sec execute 69 sec latency 105.775 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6016 2202496 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 45056 3584000 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3555328 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 65536 7139328 1% /mnt/lustre 1 5199 0.34 MB/sec execute 70 sec latency 95.686 ms 1 5292 0.34 MB/sec execute 71 sec latency 57.760 ms 1 5385 0.33 MB/sec execute 72 sec latency 52.619 ms test_26 fail mds1 4 times 1 5478 0.33 MB/sec execute 73 sec latency 101.600 ms Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 5587 0.33 MB/sec execute 74 sec latency 43.618 ms 1 5616 0.33 MB/sec execute 75 sec latency 890.896 ms 13:18:19 (1750180699) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 5616 0.32 MB/sec execute 76 sec latency 1892.824 ms 1 5616 0.32 MB/sec execute 77 sec latency 2893.784 ms 1 5616 0.31 MB/sec execute 78 sec latency 3894.076 ms 1 5616 0.31 MB/sec execute 79 sec latency 4894.505 ms 1 5616 0.30 MB/sec execute 80 sec latency 5894.726 ms 1 5616 0.30 MB/sec execute 81 sec latency 6894.900 ms 1 5616 0.30 MB/sec execute 82 sec latency 7895.308 ms 1 5616 0.29 MB/sec execute 83 sec latency 8895.573 ms 1 5616 0.29 MB/sec execute 84 sec latency 9895.764 ms 1 5616 0.29 MB/sec execute 85 sec latency 10895.944 ms Failover mds1 to oleg203-server mount facets: mds1 1 5616 0.28 MB/sec execute 86 sec latency 11896.061 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5616 0.28 MB/sec execute 87 sec latency 12897.060 ms 1 5616 0.28 MB/sec execute 88 sec latency 13897.287 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:18:32 (1750180712) targets are mounted 13:18:32 (1750180712) facet_failover done 1 5616 0.27 MB/sec execute 89 sec latency 14897.548 ms 1 5616 0.27 MB/sec execute 90 sec latency 15897.832 ms 1 5616 0.27 MB/sec execute 91 sec latency 16898.129 ms 1 5616 0.27 MB/sec execute 92 sec latency 17898.575 ms 1 5616 0.26 MB/sec execute 93 sec latency 18898.861 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5718 0.26 MB/sec execute 94 sec latency 19157.495 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5820 0.26 MB/sec execute 95 sec latency 48.240 ms 1 5928 0.27 MB/sec execute 96 sec latency 119.731 ms 1 6001 0.27 MB/sec execute 97 sec latency 131.002 ms 1 6129 0.27 MB/sec execute 98 sec latency 111.428 ms 1 6324 0.30 MB/sec execute 99 sec latency 99.968 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6912 2201600 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 46080 3596288 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 21504 3609600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 7205888 1% /mnt/lustre 1 cleanup 100 sec 1 cleanup 101 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 971 36.764 17649.301 Close 751 3.816 16.517 Rename 43 27.080 56.504 Unlink 165 8.393 19.427 Qpathinfo 924 48.219 20580.218 Qfileinfo 171 0.638 6.178 Qfsinfo 144 1.112 16.996 Sfileinfo 94 232.127 20155.519 Find 333 1.674 18.989 WriteX 513 3.320 22.554 ReadX 1539 0.357 39.402 LockX 4 4.717 8.901 UnlockX 4 4.457 6.686 Flush 67 56.911 168.601 Throughput 0.300252 MB/sec 1 clients 1 procs max_latency=20580.244 ms stopping dbench on /mnt/lustre at Tue Jun 17 13:18:45 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt test_26 fail mds1 5 times Failing mds1 on oleg203-server '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 17 13:18:47 EDT 2025 waiting for dbench pid 20127 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 90 4.19 MB/sec warmup 1 sec latency 304.268 ms 13:18:49 (1750180729) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 90 2.10 MB/sec warmup 2 sec latency 1304.491 ms 1 90 1.40 MB/sec warmup 3 sec latency 2304.679 ms 1 90 1.05 MB/sec warmup 4 sec latency 3304.894 ms 1 90 0.84 MB/sec warmup 5 sec latency 4305.077 ms 1 90 0.70 MB/sec warmup 6 sec latency 5306.706 ms 1 90 0.60 MB/sec warmup 7 sec latency 6307.371 ms 1 90 0.52 MB/sec warmup 8 sec latency 7307.586 ms 1 90 0.47 MB/sec warmup 9 sec latency 8308.074 ms 1 90 0.42 MB/sec warmup 10 sec latency 9309.421 ms 1 90 0.38 MB/sec warmup 11 sec latency 10309.625 ms 1 90 0.35 MB/sec warmup 12 sec latency 11309.840 ms Failover mds1 to oleg203-server mount facets: mds1 1 90 0.32 MB/sec warmup 13 sec latency 12310.247 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 90 0.30 MB/sec warmup 14 sec latency 13310.458 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all 1 90 0.28 MB/sec warmup 15 sec latency 14310.762 ms pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:19:03 (1750180743) targets are mounted 13:19:03 (1750180743) facet_failover done 1 90 0.26 MB/sec warmup 16 sec latency 15311.188 ms 1 90 0.25 MB/sec warmup 17 sec latency 16311.727 ms 1 90 0.23 MB/sec warmup 18 sec latency 17312.050 ms 1 90 0.22 MB/sec warmup 19 sec latency 18312.170 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 289 3.64 MB/sec execute 1 sec latency 18606.045 ms 1 397 3.51 MB/sec execute 2 sec latency 47.750 ms 1 513 3.55 MB/sec execute 3 sec latency 47.749 ms 1 669 3.52 MB/sec execute 4 sec latency 108.998 ms 1 736 2.85 MB/sec execute 5 sec latency 102.356 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 7680 2200832 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3590144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3569664 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7159808 1% /mnt/lustre 1 815 2.40 MB/sec execute 6 sec latency 75.187 ms 1 916 2.10 MB/sec execute 7 sec latency 99.049 ms 1 990 1.86 MB/sec execute 8 sec latency 99.916 ms test_26 fail mds1 6 times 1 1145 1.77 MB/sec execute 9 sec latency 46.667 ms Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 1245 1.60 MB/sec execute 10 sec latency 247.592 ms 13:19:18 (1750180758) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 1245 1.46 MB/sec execute 11 sec latency 1247.963 ms 1 1245 1.33 MB/sec execute 12 sec latency 2248.574 ms 1 1245 1.23 MB/sec execute 13 sec latency 3248.823 ms 1 1245 1.14 MB/sec execute 14 sec latency 4249.042 ms 1 1245 1.07 MB/sec execute 15 sec latency 5249.275 ms 1 1245 1.00 MB/sec execute 16 sec latency 6249.495 ms 1 1245 0.94 MB/sec execute 17 sec latency 7249.744 ms 1 1245 0.89 MB/sec execute 18 sec latency 8250.106 ms 1 1245 0.84 MB/sec execute 19 sec latency 9251.727 ms 1 1245 0.80 MB/sec execute 20 sec latency 10251.928 ms Failover mds1 to oleg203-server mount facets: mds1 1 1245 0.76 MB/sec execute 21 sec latency 11252.145 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1245 0.73 MB/sec execute 22 sec latency 12252.336 ms 1 1245 0.70 MB/sec execute 23 sec latency 13252.582 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 1 1245 0.67 MB/sec execute 24 sec latency 14252.946 ms Started lustre-MDT0000 13:19:32 (1750180772) targets are mounted 13:19:32 (1750180772) facet_failover done 1 1245 0.64 MB/sec execute 25 sec latency 15253.157 ms 1 1245 0.62 MB/sec execute 26 sec latency 16253.347 ms 1 1245 0.59 MB/sec execute 27 sec latency 17253.624 ms 1 1245 0.57 MB/sec execute 28 sec latency 18253.828 ms 1 1245 0.55 MB/sec execute 29 sec latency 19254.482 ms 1 1277 0.54 MB/sec execute 30 sec latency 19922.432 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1492 0.61 MB/sec execute 31 sec latency 137.802 ms 1 1643 0.60 MB/sec execute 32 sec latency 84.589 ms 1 1834 0.58 MB/sec execute 33 sec latency 35.957 ms 1 1982 0.57 MB/sec execute 34 sec latency 85.236 ms 1 2150 0.56 MB/sec execute 35 sec latency 40.499 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 8448 2200064 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3562496 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 46080 3559424 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 62464 7121920 1% /mnt/lustre 1 2287 0.57 MB/sec execute 36 sec latency 44.058 ms 1 2389 0.56 MB/sec execute 37 sec latency 96.403 ms 1 2499 0.57 MB/sec execute 38 sec latency 85.132 ms test_26 fail mds1 7 times 1 2878 0.64 MB/sec execute 39 sec latency 65.801 ms Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 3013 0.66 MB/sec execute 40 sec latency 131.856 ms 1 3013 0.65 MB/sec execute 41 sec latency 1132.328 ms 1 3013 0.63 MB/sec execute 42 sec latency 2132.525 ms 1 3013 0.62 MB/sec execute 43 sec latency 3133.309 ms 1 3013 0.60 MB/sec execute 44 sec latency 4133.494 ms 1 3013 0.59 MB/sec execute 45 sec latency 5136.302 ms 1 3013 0.58 MB/sec execute 46 sec latency 6137.490 ms 13:19:54 (1750180794) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 3013 0.56 MB/sec execute 47 sec latency 7137.724 ms 1 3013 0.55 MB/sec execute 48 sec latency 8137.890 ms 1 3013 0.54 MB/sec execute 49 sec latency 9139.249 ms 1 3013 0.53 MB/sec execute 50 sec latency 10139.887 ms 1 3013 0.52 MB/sec execute 51 sec latency 11140.160 ms 1 3013 0.51 MB/sec execute 52 sec latency 12143.219 ms 1 3013 0.50 MB/sec execute 53 sec latency 13143.466 ms 1 3013 0.49 MB/sec execute 54 sec latency 14143.689 ms 1 3013 0.48 MB/sec execute 55 sec latency 15143.882 ms 1 3013 0.47 MB/sec execute 56 sec latency 16144.082 ms Failover mds1 to oleg203-server mount facets: mds1 1 3013 0.46 MB/sec execute 57 sec latency 17144.434 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3013 0.46 MB/sec execute 58 sec latency 18144.613 ms 1 3013 0.45 MB/sec execute 59 sec latency 19144.779 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 1 3013 0.44 MB/sec execute 60 sec latency 20145.324 ms Started lustre-MDT0000 13:20:08 (1750180808) targets are mounted 13:20:08 (1750180808) facet_failover done 1 3013 0.43 MB/sec execute 61 sec latency 21145.588 ms 1 3013 0.43 MB/sec execute 62 sec latency 22145.772 ms 1 3013 0.42 MB/sec execute 63 sec latency 23145.955 ms 1 3013 0.41 MB/sec execute 64 sec latency 24146.187 ms 1 3013 0.41 MB/sec execute 65 sec latency 25146.375 ms 1 3013 0.40 MB/sec execute 66 sec latency 26147.437 ms 1 3013 0.39 MB/sec execute 67 sec latency 27147.760 ms 1 3013 0.39 MB/sec execute 68 sec latency 28147.912 ms 1 3013 0.38 MB/sec execute 69 sec latency 29148.142 ms 1 3024 0.38 MB/sec execute 70 sec latency 30003.324 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3326 0.40 MB/sec execute 71 sec latency 118.983 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3537 0.43 MB/sec execute 72 sec latency 47.462 ms 1 3689 0.43 MB/sec execute 73 sec latency 31.456 ms 1 3820 0.44 MB/sec execute 74 sec latency 35.729 ms 1 3949 0.44 MB/sec execute 75 sec latency 26.348 ms 1 4054 0.43 MB/sec execute 76 sec latency 28.269 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Jun 17 13:20:24 EDT 2025 with return code 0 20127 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 20127 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished