== replay-dual test 26: dbench and tar with mds failover ========================================================== 15:47:50 (1713210470) Starting client oleg218-client.virtnet: -o user_xattr,flock oleg218-server@tcp:/lustre /mnt/lustre Started clients oleg218-client.virtnet: 192.168.202.118@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 4288 Started dbench loop with 4289 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Apr 15 15:47:51 EDT 2024 waiting for dbench pid 4330 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 221 8.56 MB/sec warmup 1 sec latency 30.469 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4096 2204544 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 4096 3698688 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 5120 3733504 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 9216 7432192 1% /mnt/lustre 1 477 8.34 MB/sec warmup 2 sec latency 17.351 ms 1 733 7.10 MB/sec warmup 3 sec latency 51.337 ms test_26 fail mds1 1 times 1 983 5.48 MB/sec warmup 4 sec latency 35.744 ms Failing mds1 on oleg218-server Stopping /mnt/lustre-mds1 (opts:) on oleg218-server 1 1242 4.62 MB/sec warmup 5 sec latency 237.018 ms 15:47:57 (1713210477) shut down 1 1242 3.85 MB/sec warmup 6 sec latency 1237.279 ms 1 1242 3.30 MB/sec warmup 7 sec latency 2237.463 ms 1 1242 2.89 MB/sec warmup 8 sec latency 3237.686 ms 1 1242 2.56 MB/sec warmup 9 sec latency 4237.868 ms 1 1242 2.31 MB/sec warmup 10 sec latency 5238.101 ms 1 1242 2.10 MB/sec warmup 11 sec latency 6238.256 ms 1 1242 1.92 MB/sec warmup 12 sec latency 7238.404 ms 1 1242 1.78 MB/sec warmup 13 sec latency 8238.565 ms 1 1242 1.65 MB/sec warmup 14 sec latency 9238.683 ms 1 1242 1.54 MB/sec warmup 15 sec latency 10238.850 ms Failover mds1 to oleg218-server mount facets: mds1 1 1242 1.44 MB/sec warmup 16 sec latency 11239.000 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1242 1.36 MB/sec warmup 17 sec latency 12239.265 ms oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 15:48:09 (1713210489) targets are mounted 15:48:09 (1713210489) facet_failover done 1 1242 1.28 MB/sec warmup 18 sec latency 13239.426 ms 1 1242 1.21 MB/sec warmup 19 sec latency 14239.596 ms 1 1242 0.00 MB/sec execute 1 sec latency 16240.011 ms 1 1242 0.00 MB/sec execute 2 sec latency 17240.303 ms oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1463 0.98 MB/sec execute 3 sec latency 17723.532 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1685 0.78 MB/sec execute 4 sec latency 48.724 ms 1 1974 0.67 MB/sec execute 5 sec latency 26.424 ms 1 2162 0.61 MB/sec execute 6 sec latency 24.824 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 5376 2203264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 4992 2203648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 38912 3592192 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3660800 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 52224 7252992 1% /mnt/lustre 1 2416 0.72 MB/sec execute 7 sec latency 42.428 ms 1 2881 1.14 MB/sec execute 8 sec latency 39.220 ms 1 3237 1.27 MB/sec execute 9 sec latency 64.339 ms test_26 fail mds2 2 times 1 3584 1.53 MB/sec execute 10 sec latency 40.376 ms Failing mds2 on oleg218-server Stopping /mnt/lustre-mds2 (opts:) on oleg218-server 1 3806 1.52 MB/sec execute 11 sec latency 290.304 ms 15:48:23 (1713210503) shut down 1 3806 1.39 MB/sec execute 12 sec latency 1290.557 ms 1 3806 1.28 MB/sec execute 13 sec latency 2290.806 ms 1 3806 1.19 MB/sec execute 14 sec latency 3291.032 ms 1 3806 1.11 MB/sec execute 15 sec latency 4291.190 ms 1 3806 1.04 MB/sec execute 16 sec latency 5291.453 ms 1 3806 0.98 MB/sec execute 17 sec latency 6291.686 ms 1 3806 0.93 MB/sec execute 18 sec latency 7291.955 ms 1 3806 0.88 MB/sec execute 19 sec latency 8292.227 ms 1 3806 0.83 MB/sec execute 20 sec latency 9292.405 ms 1 3806 0.79 MB/sec execute 21 sec latency 10292.681 ms Failover mds2 to oleg218-server mount facets: mds2 1 3806 0.76 MB/sec execute 22 sec latency 11293.021 ms Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 1 3806 0.73 MB/sec execute 23 sec latency 12293.182 ms oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 1 3806 0.69 MB/sec execute 24 sec latency 13293.321 ms Started lustre-MDT0001 15:48:36 (1713210516) targets are mounted 15:48:36 (1713210516) facet_failover done 1 3806 0.67 MB/sec execute 25 sec latency 14293.438 ms 1 3806 0.64 MB/sec execute 26 sec latency 15293.622 ms 1 3806 0.62 MB/sec execute 27 sec latency 16293.888 ms 1 3806 0.60 MB/sec execute 28 sec latency 17294.114 ms 1 3819 0.58 MB/sec execute 29 sec latency 18237.497 ms oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4040 0.57 MB/sec execute 30 sec latency 14.343 ms 1 4218 0.55 MB/sec execute 31 sec latency 33.637 ms 1 4387 0.55 MB/sec execute 32 sec latency 46.725 ms 1 4648 0.57 MB/sec execute 33 sec latency 40.081 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5504 2203008 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 5376 2203264 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 47104 3530752 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3589120 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 65536 7119872 1% /mnt/lustre 1 5007 0.65 MB/sec execute 34 sec latency 25.763 ms 1 5282 0.64 MB/sec execute 35 sec latency 44.223 ms 1 5569 0.63 MB/sec execute 36 sec latency 43.834 ms test_26 fail mds1 3 times Failing mds1 on oleg218-server Stopping /mnt/lustre-mds1 (opts:) on oleg218-server 1 5905 0.65 MB/sec execute 37 sec latency 68.607 ms 15:48:49 (1713210529) shut down 1 6167 0.66 MB/sec execute 38 sec latency 83.070 ms 1 6787 0.78 MB/sec execute 39 sec latency 39.078 ms 1 7241 0.87 MB/sec execute 40 sec latency 15.006 ms 1 7430 0.87 MB/sec execute 41 sec latency 19.955 ms 1 7587 0.86 MB/sec execute 42 sec latency 24.245 ms 1 7627 0.84 MB/sec execute 43 sec latency 726.757 ms 1 7627 0.82 MB/sec execute 44 sec latency 1726.850 ms 1 7627 0.80 MB/sec execute 45 sec latency 2726.939 ms 1 7627 0.79 MB/sec execute 46 sec latency 3727.077 ms 1 7627 0.77 MB/sec execute 47 sec latency 4727.216 ms Failover mds1 to oleg218-server mount facets: mds1 1 7627 0.75 MB/sec execute 48 sec latency 5727.302 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7627 0.74 MB/sec execute 49 sec latency 6727.390 ms oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 1 7627 0.72 MB/sec execute 50 sec latency 7727.511 ms Started lustre-MDT0000 15:49:02 (1713210542) targets are mounted 15:49:02 (1713210542) facet_failover done 1 7627 0.71 MB/sec execute 51 sec latency 8727.728 ms 1 7627 0.69 MB/sec execute 52 sec latency 9727.913 ms 1 7627 0.68 MB/sec execute 53 sec latency 10728.092 ms 1 7627 0.67 MB/sec execute 54 sec latency 11728.364 ms oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7818 0.66 MB/sec execute 55 sec latency 12032.878 ms 1 8053 0.66 MB/sec execute 56 sec latency 34.560 ms 1 8367 0.67 MB/sec execute 57 sec latency 33.432 ms 1 8733 0.71 MB/sec execute 58 sec latency 44.630 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 5248 2203392 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 9088 2199552 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 47104 3222528 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 21504 3319808 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 68608 6542336 2% /mnt/lustre 1 9045 0.71 MB/sec execute 59 sec latency 32.125 ms 1 9395 0.72 MB/sec execute 60 sec latency 21.334 ms 1 9697 0.72 MB/sec execute 61 sec latency 36.963 ms test_26 fail mds2 4 times 1 10279 0.79 MB/sec execute 62 sec latency 35.952 ms Failing mds2 on oleg218-server Stopping /mnt/lustre-mds2 (opts:) on oleg218-server 1 10477 0.80 MB/sec execute 63 sec latency 496.256 ms 1 10477 0.79 MB/sec execute 64 sec latency 1496.446 ms 1 10477 0.78 MB/sec execute 65 sec latency 2496.705 ms 1 10477 0.77 MB/sec execute 66 sec latency 3496.931 ms 1 10477 0.76 MB/sec execute 67 sec latency 4497.157 ms 1 10477 0.74 MB/sec execute 68 sec latency 5497.366 ms 15:49:20 (1713210560) shut down 1 10477 0.73 MB/sec execute 69 sec latency 6497.543 ms 1 10477 0.72 MB/sec execute 70 sec latency 7497.746 ms 1 10477 0.71 MB/sec execute 71 sec latency 8497.936 ms 1 10477 0.70 MB/sec execute 72 sec latency 9498.083 ms 1 10477 0.69 MB/sec execute 73 sec latency 10498.303 ms 1 10477 0.68 MB/sec execute 74 sec latency 11498.496 ms 1 10477 0.67 MB/sec execute 75 sec latency 12498.671 ms 1 10477 0.67 MB/sec execute 76 sec latency 13498.925 ms 1 10477 0.66 MB/sec execute 77 sec latency 14499.231 ms 1 10477 0.65 MB/sec execute 78 sec latency 15499.476 ms Failover mds2 to oleg218-server mount facets: mds2 1 10477 0.64 MB/sec execute 79 sec latency 16499.651 ms Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 1 10477 0.63 MB/sec execute 80 sec latency 17499.744 ms oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0001 15:49:32 (1713210572) targets are mounted 15:49:32 (1713210572) facet_failover done 1 10477 0.62 MB/sec execute 81 sec latency 18499.886 ms 1 10477 0.62 MB/sec execute 82 sec latency 19500.080 ms 1 10477 0.61 MB/sec execute 83 sec latency 20500.236 ms 1 10477 0.60 MB/sec execute 84 sec latency 21500.453 ms 1 10477 0.60 MB/sec execute 85 sec latency 22500.595 ms oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 10709 0.62 MB/sec execute 86 sec latency 23011.139 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 10958 0.63 MB/sec execute 87 sec latency 17.770 ms 1 11132 0.63 MB/sec execute 88 sec latency 24.048 ms 1 11371 0.62 MB/sec execute 89 sec latency 34.409 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 6144 2202496 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 11008 2197504 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 52224 3567616 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3598336 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 86016 7165952 2% /mnt/lustre 1 11615 0.62 MB/sec execute 90 sec latency 29.798 ms 1 11919 0.63 MB/sec execute 91 sec latency 23.941 ms 1 12271 0.66 MB/sec execute 92 sec latency 40.216 ms test_26 fail mds1 5 times 1 12481 0.65 MB/sec execute 93 sec latency 59.816 ms Failing mds1 on oleg218-server Stopping /mnt/lustre-mds1 (opts:) on oleg218-server 1 12688 0.65 MB/sec execute 94 sec latency 184.204 ms 15:49:46 (1713210586) shut down 1 12688 0.64 MB/sec execute 95 sec latency 1184.341 ms 1 12688 0.63 MB/sec execute 96 sec latency 2184.521 ms 1 12688 0.63 MB/sec execute 97 sec latency 3184.784 ms 1 12688 0.62 MB/sec execute 98 sec latency 4185.000 ms 1 12688 0.61 MB/sec execute 99 sec latency 5185.289 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec Failover mds1 to oleg218-server mount facets: mds1 1 cleanup 105 sec Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 106 sec 1 cleanup 107 sec oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 1 cleanup 108 sec Started lustre-MDT0000 15:50:00 (1713210600) targets are mounted 15:50:00 (1713210600) facet_failover done 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 0 cleanup 113 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2010 29.551 23011.122 Close 1478 1.848 6.010 Rename 80 12.469 31.345 Unlink 385 5.163 20.787 Qpathinfo 1839 22.542 18359.758 Qfileinfo 319 0.449 2.465 Qfsinfo 311 0.941 10.062 Sfileinfo 154 6.315 13.386 Find 685 20.218 12032.864 WriteX 962 1.963 9.442 ReadX 3077 0.104 12.421 LockX 6 1.380 1.618 UnlockX 6 1.492 1.680 Flush 135 24.306 83.059 Throughput 0.614617 MB/sec 1 clients 1 procs max_latency=23011.139 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec stopping dbench on /mnt/lustre at Mon Apr 15 15:50:04 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Apr 15 15:50:06 EDT 2024 waiting for dbench pid 9476 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 279 10.44 MB/sec warmup 1 sec latency 20.985 ms 1 484 8.47 MB/sec warmup 2 sec latency 28.057 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 7552 2201088 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 12160 2196480 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 45056 3572736 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3621888 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 7194624 1% /mnt/lustre 1 765 7.12 MB/sec warmup 3 sec latency 38.155 ms 1 1035 5.72 MB/sec warmup 4 sec latency 27.521 ms test_26 fail mds2 6 times 1 1244 4.62 MB/sec warmup 5 sec latency 26.120 ms Failing mds2 on oleg218-server Stopping /mnt/lustre-mds2 (opts:) on oleg218-server 1 1588 4.36 MB/sec warmup 6 sec latency 54.760 ms 15:50:13 (1713210613) shut down 1 1589 3.74 MB/sec warmup 7 sec latency 991.357 ms 1 1589 3.27 MB/sec warmup 8 sec latency 1991.583 ms 1 1589 2.91 MB/sec warmup 9 sec latency 2991.859 ms 1 1589 2.62 MB/sec warmup 10 sec latency 3992.108 ms 1 1589 2.38 MB/sec warmup 11 sec latency 4992.294 ms 1 1589 2.18 MB/sec warmup 12 sec latency 5992.458 ms 1 1589 2.01 MB/sec warmup 13 sec latency 6992.721 ms 1 1589 1.87 MB/sec warmup 14 sec latency 7992.977 ms 1 1589 1.74 MB/sec warmup 15 sec latency 8993.161 ms 1 1589 1.63 MB/sec warmup 16 sec latency 9993.393 ms Failover mds2 to oleg218-server mount facets: mds2 1 1589 1.54 MB/sec warmup 17 sec latency 10993.566 ms Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 1 1589 1.45 MB/sec warmup 18 sec latency 11993.741 ms 1 1589 1.38 MB/sec warmup 19 sec latency 12993.909 ms oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0001 15:50:26 (1713210626) targets are mounted 15:50:26 (1713210626) facet_failover done 1 1589 0.00 MB/sec execute 1 sec latency 14994.332 ms 1 1589 0.00 MB/sec execute 2 sec latency 15994.571 ms 1 1589 0.00 MB/sec execute 3 sec latency 16994.766 ms 1 1589 0.00 MB/sec execute 4 sec latency 17994.982 ms oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1736 0.03 MB/sec execute 5 sec latency 18503.695 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2042 0.07 MB/sec execute 6 sec latency 26.042 ms 1 2389 0.26 MB/sec execute 7 sec latency 46.965 ms 1 2584 0.35 MB/sec execute 8 sec latency 57.821 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 7808 2200832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 11776 2196736 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 47104 3564544 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3555328 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 77824 7119872 2% /mnt/lustre 1 3028 0.82 MB/sec execute 9 sec latency 51.485 ms 1 3590 1.23 MB/sec execute 10 sec latency 31.428 ms 1 3789 1.24 MB/sec execute 11 sec latency 26.561 ms test_26 fail mds1 7 times 1 3960 1.16 MB/sec execute 12 sec latency 19.437 ms Failing mds1 on oleg218-server Stopping /mnt/lustre-mds1 (opts:) on oleg218-server 1 4083 1.07 MB/sec execute 13 sec latency 380.627 ms 15:50:39 (1713210639) shut down 1 4083 1.00 MB/sec execute 14 sec latency 1380.749 ms 1 4083 0.93 MB/sec execute 15 sec latency 2380.929 ms 1 4083 0.87 MB/sec execute 16 sec latency 3381.086 ms 1 4083 0.82 MB/sec execute 17 sec latency 4381.201 ms 1 4083 0.77 MB/sec execute 18 sec latency 5381.359 ms 1 4083 0.73 MB/sec execute 19 sec latency 6381.503 ms 1 4083 0.70 MB/sec execute 20 sec latency 7381.680 ms 1 4083 0.66 MB/sec execute 21 sec latency 8381.780 ms 1 4083 0.63 MB/sec execute 22 sec latency 9381.925 ms 1 4083 0.61 MB/sec execute 23 sec latency 10382.106 ms Failover mds1 to oleg218-server mount facets: mds1 1 4083 0.58 MB/sec execute 24 sec latency 11382.317 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4083 0.56 MB/sec execute 25 sec latency 12382.426 ms oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all 1 4083 0.54 MB/sec execute 26 sec latency 13382.589 ms pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 15:50:53 (1713210653) targets are mounted 15:50:53 (1713210653) facet_failover done 1 4083 0.52 MB/sec execute 27 sec latency 14382.814 ms 1 4083 0.50 MB/sec execute 28 sec latency 15382.998 ms 1 4083 0.48 MB/sec execute 29 sec latency 16383.146 ms 1 4083 0.46 MB/sec execute 30 sec latency 17383.327 ms oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4279 0.46 MB/sec execute 31 sec latency 17645.015 ms 1 4488 0.46 MB/sec execute 32 sec latency 40.529 ms 1 4723 0.48 MB/sec execute 33 sec latency 33.466 ms 1 4969 0.56 MB/sec execute 34 sec latency 44.785 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6784 2201728 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 6656 2201856 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 48128 3716096 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 27648 3728384 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 75776 7444480 2% /mnt/lustre 1 5147 0.55 MB/sec execute 35 sec latency 67.365 ms 1 5364 0.54 MB/sec execute 36 sec latency 78.439 ms 1 5569 0.53 MB/sec execute 37 sec latency 88.654 ms test_26 fail mds2 8 times 1 5894 0.55 MB/sec execute 38 sec latency 17.205 ms Failing mds2 on oleg218-server Stopping /mnt/lustre-mds2 (opts:) on oleg218-server 1 6039 0.56 MB/sec execute 39 sec latency 69.010 ms 15:51:06 (1713210666) shut down 1 6039 0.54 MB/sec execute 40 sec latency 1033.832 ms 1 6039 0.53 MB/sec execute 41 sec latency 2034.013 ms 1 6039 0.52 MB/sec execute 42 sec latency 3034.212 ms 1 6039 0.50 MB/sec execute 43 sec latency 4034.392 ms 1 6039 0.49 MB/sec execute 44 sec latency 5034.584 ms 1 6039 0.48 MB/sec execute 45 sec latency 6034.744 ms 1 6039 0.47 MB/sec execute 46 sec latency 7034.952 ms 1 6039 0.46 MB/sec execute 47 sec latency 8035.159 ms 1 6039 0.45 MB/sec execute 48 sec latency 9035.342 ms 1 6039 0.44 MB/sec execute 49 sec latency 10035.601 ms Failover mds2 to oleg218-server mount facets: mds2 1 6039 0.43 MB/sec execute 50 sec latency 11035.745 ms Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 1 6039 0.43 MB/sec execute 51 sec latency 12035.901 ms 1 6039 0.42 MB/sec execute 52 sec latency 13036.093 ms oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0001 15:51:19 (1713210679) targets are mounted 15:51:19 (1713210679) facet_failover done 1 6039 0.41 MB/sec execute 53 sec latency 14036.384 ms 1 6039 0.40 MB/sec execute 54 sec latency 15036.542 ms 1 6039 0.39 MB/sec execute 55 sec latency 16036.726 ms 1 6039 0.39 MB/sec execute 56 sec latency 17037.004 ms 1 6039 0.38 MB/sec execute 57 sec latency 18037.209 ms oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 6324 0.43 MB/sec execute 58 sec latency 18456.033 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6742 0.46 MB/sec execute 59 sec latency 63.207 ms 1 7060 0.52 MB/sec execute 60 sec latency 76.232 ms 1 7306 0.54 MB/sec execute 61 sec latency 21.016 ms 1 7558 0.53 MB/sec execute 62 sec latency 12.705 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Apr 15 15:51:29 EDT 2024 with return code 0 9476 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 9476 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished