== replay-dual test 26: dbench and tar with mds failover ========================================================== 06:50:45 (1733399445) Starting client oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre Started clients oleg146-client.virtnet: 192.168.201.146@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 3775 Started dbench loop with 3776 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Dec 5 06:50:47 EST 2024 waiting for dbench pid 3817 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 223 8.62 MB/sec warmup 1 sec latency 24.962 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2268 1285420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2068 1285620 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26120 3580648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27656 7186132 1% /mnt/lustre 1 475 8.31 MB/sec warmup 2 sec latency 20.994 ms 1 689 7.06 MB/sec warmup 3 sec latency 391.225 ms 1 960 5.45 MB/sec warmup 4 sec latency 18.842 ms test_26 fail mds1 1 times 1 1281 4.63 MB/sec warmup 5 sec latency 16.652 ms Failing mds1 on oleg146-server Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 1 1589 4.36 MB/sec warmup 6 sec latency 265.427 ms 06:50:53 (1733399453) shut down 1 1589 3.74 MB/sec warmup 7 sec latency 1265.661 ms 1 1589 3.27 MB/sec warmup 8 sec latency 2265.805 ms 1 1589 2.91 MB/sec warmup 9 sec latency 3266.000 ms 1 1589 2.62 MB/sec warmup 10 sec latency 4266.177 ms 1 1589 2.38 MB/sec warmup 11 sec latency 5266.322 ms 1 1589 2.18 MB/sec warmup 12 sec latency 6266.543 ms 1 1589 2.01 MB/sec warmup 13 sec latency 7266.740 ms 1 1589 1.87 MB/sec warmup 14 sec latency 8266.919 ms 1 1589 1.74 MB/sec warmup 15 sec latency 9267.129 ms 1 1589 1.63 MB/sec warmup 16 sec latency 10267.369 ms Failover mds1 to oleg146-server mount facets: mds1 1 1589 1.54 MB/sec warmup 17 sec latency 11267.541 ms 1 1589 1.45 MB/sec warmup 18 sec latency 12267.673 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1589 1.38 MB/sec warmup 19 sec latency 13267.819 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 06:51:07 (1733399467) targets are mounted 06:51:07 (1733399467) facet_failover done 1 1589 0.00 MB/sec execute 1 sec latency 15268.035 ms 1 1589 0.00 MB/sec execute 2 sec latency 16268.276 ms 1 1589 0.00 MB/sec execute 3 sec latency 17268.467 ms 1 1589 0.00 MB/sec execute 4 sec latency 18268.683 ms 1 1670 0.01 MB/sec execute 5 sec latency 18969.409 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1989 0.05 MB/sec execute 6 sec latency 18.173 ms 1 2275 0.15 MB/sec execute 7 sec latency 45.201 ms 1 2597 0.36 MB/sec execute 8 sec latency 39.163 ms 1 3245 0.95 MB/sec execute 9 sec latency 16.656 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3004 1284684 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2596 1285092 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 32096 3565668 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5984 3583264 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38080 7148932 1% /mnt/lustre 1 3688 1.26 MB/sec execute 10 sec latency 29.078 ms 1 3955 1.26 MB/sec execute 11 sec latency 14.212 ms 1 4233 1.18 MB/sec execute 12 sec latency 17.965 ms 1 4499 1.13 MB/sec execute 13 sec latency 17.582 ms test_26 fail mds2 2 times Failing mds2 on oleg146-server Stopping /mnt/lustre-mds2 (opts:) on oleg146-server 1 4810 1.15 MB/sec execute 14 sec latency 17.038 ms 06:51:22 (1733399482) shut down 1 4812 1.07 MB/sec execute 15 sec latency 993.919 ms 1 4812 1.00 MB/sec execute 16 sec latency 1994.027 ms 1 4812 0.94 MB/sec execute 17 sec latency 2994.128 ms 1 4812 0.89 MB/sec execute 18 sec latency 3994.252 ms 1 4812 0.84 MB/sec execute 19 sec latency 4994.379 ms 1 4812 0.80 MB/sec execute 20 sec latency 5994.481 ms 1 4812 0.76 MB/sec execute 21 sec latency 6994.589 ms 1 4812 0.73 MB/sec execute 22 sec latency 7994.702 ms 1 4812 0.70 MB/sec execute 23 sec latency 8994.798 ms 1 4812 0.67 MB/sec execute 24 sec latency 9994.943 ms Failover mds2 to oleg146-server mount facets: mds2 1 4812 0.64 MB/sec execute 25 sec latency 10995.099 ms 1 4812 0.62 MB/sec execute 26 sec latency 11995.222 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4812 0.59 MB/sec execute 27 sec latency 12995.332 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all 1 4812 0.57 MB/sec execute 28 sec latency 13995.427 ms pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0001 06:51:36 (1733399496) targets are mounted 06:51:36 (1733399496) facet_failover done 1 4812 0.55 MB/sec execute 29 sec latency 14995.593 ms 1 4812 0.53 MB/sec execute 30 sec latency 15995.806 ms 1 4812 0.52 MB/sec execute 31 sec latency 16995.999 ms 1 4812 0.50 MB/sec execute 32 sec latency 17996.200 ms 1 5013 0.58 MB/sec execute 33 sec latency 18511.100 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5276 0.56 MB/sec execute 34 sec latency 17.043 ms 1 5575 0.56 MB/sec execute 35 sec latency 13.186 ms 1 5919 0.58 MB/sec execute 36 sec latency 48.288 ms 1 6329 0.67 MB/sec execute 37 sec latency 18.400 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4556 1283132 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37208 3564324 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12772 3582520 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49980 7146844 1% /mnt/lustre 1 6897 0.75 MB/sec execute 38 sec latency 19.402 ms 1 7273 0.82 MB/sec execute 39 sec latency 13.284 ms 1 7519 0.83 MB/sec execute 40 sec latency 15.180 ms 1 7788 0.81 MB/sec execute 41 sec latency 14.572 ms test_26 fail mds1 3 times Failing mds1 on oleg146-server Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 1 8035 0.81 MB/sec execute 42 sec latency 29.143 ms 1 8106 0.79 MB/sec execute 43 sec latency 712.889 ms 1 8106 0.77 MB/sec execute 44 sec latency 1713.099 ms 1 8106 0.76 MB/sec execute 45 sec latency 2713.261 ms 1 8106 0.74 MB/sec execute 46 sec latency 3713.404 ms 1 8106 0.72 MB/sec execute 47 sec latency 4713.548 ms 06:51:55 (1733399515) shut down 1 8106 0.71 MB/sec execute 48 sec latency 5713.660 ms 1 8106 0.69 MB/sec execute 49 sec latency 6713.790 ms 1 8106 0.68 MB/sec execute 50 sec latency 7713.973 ms 1 8106 0.67 MB/sec execute 51 sec latency 8714.114 ms 1 8106 0.65 MB/sec execute 52 sec latency 9714.208 ms 1 8106 0.64 MB/sec execute 53 sec latency 10714.423 ms 1 8106 0.63 MB/sec execute 54 sec latency 11714.607 ms 1 8106 0.62 MB/sec execute 55 sec latency 12714.844 ms 1 8106 0.61 MB/sec execute 56 sec latency 13715.073 ms 1 8106 0.60 MB/sec execute 57 sec latency 14715.248 ms 1 8106 0.59 MB/sec execute 58 sec latency 15715.475 ms Failover mds1 to oleg146-server mount facets: mds1 1 8106 0.58 MB/sec execute 59 sec latency 16715.621 ms 1 8106 0.57 MB/sec execute 60 sec latency 17715.760 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 8106 0.56 MB/sec execute 61 sec latency 18715.913 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 06:52:09 (1733399529) targets are mounted 06:52:09 (1733399529) facet_failover done 1 8106 0.55 MB/sec execute 62 sec latency 19716.032 ms 1 8106 0.54 MB/sec execute 63 sec latency 20716.151 ms 1 8106 0.53 MB/sec execute 64 sec latency 21716.302 ms 1 8106 0.52 MB/sec execute 65 sec latency 22716.479 ms 1 8106 0.51 MB/sec execute 66 sec latency 23716.615 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 8287 0.52 MB/sec execute 67 sec latency 24138.509 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8608 0.56 MB/sec execute 68 sec latency 19.016 ms 1 8885 0.56 MB/sec execute 69 sec latency 15.945 ms 1 9195 0.55 MB/sec execute 70 sec latency 13.417 ms 1 9531 0.57 MB/sec execute 71 sec latency 15.662 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5132 1282556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3128 1284560 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 48768 3550540 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24416 3580332 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 73184 7130872 2% /mnt/lustre 1 10006 0.62 MB/sec execute 72 sec latency 16.607 ms 1 10466 0.65 MB/sec execute 73 sec latency 16.307 ms 1 10863 0.70 MB/sec execute 74 sec latency 14.289 ms test_26 fail mds2 4 times 1 11082 0.70 MB/sec execute 75 sec latency 14.551 ms Failing mds2 on oleg146-server Stopping /mnt/lustre-mds2 (opts:) on oleg146-server 1 11277 0.69 MB/sec execute 76 sec latency 220.119 ms 06:52:23 (1733399543) shut down 1 11277 0.68 MB/sec execute 77 sec latency 1220.232 ms 1 11277 0.67 MB/sec execute 78 sec latency 2220.408 ms 1 11277 0.66 MB/sec execute 79 sec latency 3220.588 ms 1 11277 0.65 MB/sec execute 80 sec latency 4220.793 ms 1 11277 0.64 MB/sec execute 81 sec latency 5220.989 ms 1 11277 0.64 MB/sec execute 82 sec latency 6221.100 ms 1 11277 0.63 MB/sec execute 83 sec latency 7221.269 ms 1 11277 0.62 MB/sec execute 84 sec latency 8221.541 ms 1 11277 0.61 MB/sec execute 85 sec latency 9221.703 ms 1 11277 0.61 MB/sec execute 86 sec latency 10221.897 ms Failover mds2 to oleg146-server mount facets: mds2 1 11277 0.60 MB/sec execute 87 sec latency 11222.016 ms 1 11277 0.59 MB/sec execute 88 sec latency 12222.199 ms 1 11277 0.59 MB/sec execute 89 sec latency 13222.358 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 11277 0.58 MB/sec execute 90 sec latency 14222.602 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0001 06:52:38 (1733399558) targets are mounted 06:52:38 (1733399558) facet_failover done 1 11277 0.57 MB/sec execute 91 sec latency 15222.887 ms 1 11277 0.57 MB/sec execute 92 sec latency 16223.010 ms 1 11277 0.56 MB/sec execute 93 sec latency 17223.205 ms 1 11277 0.56 MB/sec execute 94 sec latency 18223.451 ms 1 11277 0.55 MB/sec execute 95 sec latency 19223.650 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 11482 0.55 MB/sec execute 96 sec latency 19465.978 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11782 0.56 MB/sec execute 97 sec latency 16.790 ms 1 12126 0.58 MB/sec execute 98 sec latency 23.524 ms 1 12404 0.58 MB/sec execute 99 sec latency 17.548 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5176 1282512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3256 1284432 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 53908 3549324 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25320 3578048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 79228 7127372 2% /mnt/lustre 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1944 40.824 24138.498 Close 1436 1.575 5.309 Rename 79 13.983 23.501 Unlink 378 4.473 17.946 Qpathinfo 1790 2.445 12.336 Qfileinfo 309 0.451 2.507 Qfsinfo 306 0.905 13.515 Sfileinfo 150 8.370 22.032 Find 667 30.757 18969.401 WriteX 941 1.983 10.651 ReadX 2955 0.109 39.153 LockX 6 1.368 1.529 UnlockX 6 1.456 1.721 Flush 134 7.539 48.272 Throughput 0.580586 MB/sec 1 clients 1 procs max_latency=24138.509 ms stopping dbench on /mnt/lustre at Thu Dec 5 06:52:48 EST 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Dec 5 06:52:49 EST 2024 waiting for dbench pid 8409 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 250 9.40 MB/sec warmup 1 sec latency 23.338 ms test_26 fail mds1 5 times Failing mds1 on oleg146-server 1 512 8.86 MB/sec warmup 2 sec latency 17.183 ms Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 1 652 7.02 MB/sec warmup 3 sec latency 689.159 ms 1 652 5.27 MB/sec warmup 4 sec latency 1689.367 ms 1 652 4.21 MB/sec warmup 5 sec latency 2689.605 ms 1 652 3.51 MB/sec warmup 6 sec latency 3689.773 ms 1 652 3.01 MB/sec warmup 7 sec latency 4689.938 ms 1 652 2.63 MB/sec warmup 8 sec latency 5690.113 ms 06:52:58 (1733399578) shut down 1 652 2.34 MB/sec warmup 9 sec latency 6690.276 ms 1 652 2.11 MB/sec warmup 10 sec latency 7690.525 ms 1 652 1.91 MB/sec warmup 11 sec latency 8690.765 ms 1 652 1.76 MB/sec warmup 12 sec latency 9691.106 ms 1 652 1.62 MB/sec warmup 13 sec latency 10691.350 ms 1 652 1.50 MB/sec warmup 14 sec latency 11691.628 ms 1 652 1.40 MB/sec warmup 15 sec latency 12691.851 ms 1 652 1.32 MB/sec warmup 16 sec latency 13692.072 ms 1 652 1.24 MB/sec warmup 17 sec latency 14692.290 ms 1 652 1.17 MB/sec warmup 18 sec latency 15692.613 ms Failover mds1 to oleg146-server mount facets: mds1 1 652 1.11 MB/sec warmup 19 sec latency 16692.816 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 652 0.00 MB/sec execute 1 sec latency 18693.092 ms 1 652 0.00 MB/sec execute 2 sec latency 19693.335 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 06:53:12 (1733399592) targets are mounted 06:53:12 (1733399592) facet_failover done 1 652 0.00 MB/sec execute 3 sec latency 20693.633 ms 1 652 0.00 MB/sec execute 4 sec latency 21693.765 ms 1 652 0.00 MB/sec execute 5 sec latency 22693.996 ms 1 652 0.00 MB/sec execute 6 sec latency 23694.256 ms 1 652 0.00 MB/sec execute 7 sec latency 24694.403 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 848 0.06 MB/sec execute 8 sec latency 24889.260 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1102 0.21 MB/sec execute 9 sec latency 26.542 ms 1 1451 0.50 MB/sec execute 10 sec latency 16.697 ms 1 1689 0.47 MB/sec execute 11 sec latency 23.187 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4004 1283684 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2548 1285140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 33416 3565548 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15704 3583664 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49120 7149212 1% /mnt/lustre 1 2012 0.45 MB/sec execute 12 sec latency 28.359 ms 1 2358 0.53 MB/sec execute 13 sec latency 16.397 ms 1 2562 0.56 MB/sec execute 14 sec latency 376.620 ms 1 3189 0.88 MB/sec execute 15 sec latency 17.276 ms test_26 fail mds2 6 times 1 3667 1.10 MB/sec execute 16 sec latency 15.487 ms Failing mds2 on oleg146-server Stopping /mnt/lustre-mds2 (opts:) on oleg146-server 1 3821 1.10 MB/sec execute 17 sec latency 432.520 ms 06:53:27 (1733399607) shut down 1 3821 1.04 MB/sec execute 18 sec latency 1432.733 ms 1 3821 0.98 MB/sec execute 19 sec latency 2434.213 ms 1 3821 0.93 MB/sec execute 20 sec latency 3434.296 ms 1 3821 0.89 MB/sec execute 21 sec latency 4434.436 ms 1 3821 0.85 MB/sec execute 22 sec latency 5434.579 ms 1 3821 0.81 MB/sec execute 23 sec latency 6434.706 ms 1 3821 0.78 MB/sec execute 24 sec latency 7434.836 ms 1 3821 0.75 MB/sec execute 25 sec latency 8434.931 ms 1 3821 0.72 MB/sec execute 26 sec latency 9435.086 ms 1 3821 0.69 MB/sec execute 27 sec latency 10435.237 ms Failover mds2 to oleg146-server mount facets: mds2 1 3821 0.67 MB/sec execute 28 sec latency 11435.348 ms 1 3821 0.64 MB/sec execute 29 sec latency 12435.536 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 3821 0.62 MB/sec execute 30 sec latency 13435.667 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 1 3821 0.60 MB/sec execute 31 sec latency 14435.829 ms Started lustre-MDT0001 06:53:40 (1733399620) targets are mounted 06:53:40 (1733399620) facet_failover done 1 3821 0.58 MB/sec execute 32 sec latency 15436.026 ms 1 3821 0.57 MB/sec execute 33 sec latency 16436.180 ms 1 3821 0.55 MB/sec execute 34 sec latency 17436.348 ms 1 3821 0.53 MB/sec execute 35 sec latency 18436.503 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4006 0.53 MB/sec execute 36 sec latency 18603.247 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar 1 4248 0.52 MB/sec execute 37 sec latency 44.745 ms 1 4489 0.52 MB/sec execute 38 sec latency 27.901 ms 1 4787 0.54 MB/sec execute 39 sec latency 16.446 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2652 1285036 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2240 1285448 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37464 3569132 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11956 3594152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49420 7163284 1% /mnt/lustre 1 5129 0.60 MB/sec execute 40 sec latency 16.247 ms 1 5352 0.59 MB/sec execute 41 sec latency 158.072 ms 1 5573 0.58 MB/sec execute 42 sec latency 428.171 ms 1 5922 0.60 MB/sec execute 43 sec latency 12.955 ms test_26 fail mds1 7 times Failing mds1 on oleg146-server 1 6312 0.68 MB/sec execute 44 sec latency 17.971 ms Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 1 6832 0.74 MB/sec execute 45 sec latency 18.527 ms 06:53:54 (1733399634) shut down 1 7176 0.80 MB/sec execute 46 sec latency 20.632 ms 1 7341 0.80 MB/sec execute 47 sec latency 25.822 ms 1 7467 0.79 MB/sec execute 48 sec latency 26.309 ms 1 7590 0.78 MB/sec execute 49 sec latency 26.045 ms 1 7627 0.76 MB/sec execute 50 sec latency 723.502 ms 1 7627 0.75 MB/sec execute 51 sec latency 1723.704 ms 1 7627 0.73 MB/sec execute 52 sec latency 2723.963 ms 1 7627 0.72 MB/sec execute 53 sec latency 3724.293 ms 1 7627 0.71 MB/sec execute 54 sec latency 4724.622 ms 1 7627 0.69 MB/sec execute 55 sec latency 5724.955 ms Failover mds1 to oleg146-server mount facets: mds1 1 7627 0.68 MB/sec execute 56 sec latency 6725.199 ms 1 7627 0.67 MB/sec execute 57 sec latency 7725.318 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7627 0.66 MB/sec execute 58 sec latency 8725.513 ms oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 1 7627 0.65 MB/sec execute 59 sec latency 9725.636 ms Started lustre-MDT0000 06:54:09 (1733399649) targets are mounted 06:54:09 (1733399649) facet_failover done 1 7627 0.64 MB/sec execute 60 sec latency 10725.780 ms 1 7627 0.63 MB/sec execute 61 sec latency 11725.937 ms 1 7627 0.62 MB/sec execute 62 sec latency 12726.084 ms 1 7627 0.61 MB/sec execute 63 sec latency 13726.226 ms 1 7703 0.60 MB/sec execute 64 sec latency 14476.662 ms oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7994 0.60 MB/sec execute 65 sec latency 18.829 ms 1 8287 0.61 MB/sec execute 66 sec latency 18.325 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 8570 0.64 MB/sec execute 67 sec latency 25.126 ms 1 8770 0.64 MB/sec execute 68 sec latency 21.250 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Dec 5 06:54:18 EST 2024 with return code 0 8409 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 8409 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished