== replay-dual test 26: dbench and tar with mds failover ========================================================== 00:33:50 (1734672830) Starting client oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre Started clients oleg426-client.virtnet: 192.168.204.126@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 15255 Started dbench loop with 15257 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Dec 20 00:33:52 EST 2024 waiting for dbench pid 15286 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3742720 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3727360 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7470080 1% /mnt/lustre 1 121 5.50 MB/sec warmup 1 sec latency 83.709 ms 1 207 4.05 MB/sec warmup 2 sec latency 69.874 ms 1 305 3.74 MB/sec warmup 3 sec latency 55.643 ms 1 390 3.45 MB/sec warmup 4 sec latency 71.051 ms test_26 fail mds1 1 times 1 493 3.44 MB/sec warmup 5 sec latency 57.912 ms Failing mds1 on oleg426-server Stopping /mnt/lustre-mds1 (opts:) on oleg426-server 1 642 3.51 MB/sec warmup 6 sec latency 67.389 ms 1 642 3.01 MB/sec warmup 7 sec latency 1015.221 ms 00:34:00 (1734672840) shut down facet: mds1 facet_host: oleg426-server facet_failover_host: oleg426-server 1 642 2.63 MB/sec warmup 8 sec latency 2015.634 ms 1 642 2.34 MB/sec warmup 9 sec latency 3016.525 ms 1 642 2.10 MB/sec warmup 10 sec latency 4016.760 ms 1 642 1.91 MB/sec warmup 11 sec latency 5016.936 ms 1 642 1.75 MB/sec warmup 12 sec latency 6019.301 ms 1 642 1.62 MB/sec warmup 13 sec latency 7019.483 ms 1 642 1.50 MB/sec warmup 14 sec latency 8020.821 ms 1 642 1.40 MB/sec warmup 15 sec latency 9021.099 ms 1 642 1.32 MB/sec warmup 16 sec latency 10021.940 ms 1 642 1.24 MB/sec warmup 17 sec latency 11023.177 ms Failover mds1 to oleg426-server mount facets: mds1 1 642 1.17 MB/sec warmup 18 sec latency 12023.357 ms 1 642 1.11 MB/sec warmup 19 sec latency 13023.557 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 642 0.00 MB/sec execute 1 sec latency 15024.010 ms 1 642 0.00 MB/sec execute 2 sec latency 16025.386 ms oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 1 642 0.00 MB/sec execute 3 sec latency 17026.821 ms Started lustre-MDT0000 00:34:15 (1734672855) targets are mounted 00:34:15 (1734672855) facet_failover done 1 642 0.00 MB/sec execute 4 sec latency 18028.420 ms 1 642 0.00 MB/sec execute 5 sec latency 19028.564 ms 1 642 0.00 MB/sec execute 6 sec latency 20030.100 ms 1 642 0.00 MB/sec execute 7 sec latency 21030.238 ms 1 649 0.00 MB/sec execute 8 sec latency 21849.845 ms oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 709 0.02 MB/sec execute 9 sec latency 156.870 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 737 0.03 MB/sec execute 10 sec latency 156.696 ms 1 791 0.04 MB/sec execute 11 sec latency 142.866 ms 1 873 0.05 MB/sec execute 12 sec latency 79.025 ms 1 914 0.05 MB/sec execute 13 sec latency 171.732 ms 1 981 0.06 MB/sec execute 14 sec latency 146.529 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4224 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3679232 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3623936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 48128 7303168 1% /mnt/lustre 1 1042 0.12 MB/sec execute 15 sec latency 98.850 ms 1 1132 0.12 MB/sec execute 16 sec latency 67.808 ms 1 1200 0.12 MB/sec execute 17 sec latency 72.647 ms 1 1384 0.27 MB/sec execute 18 sec latency 43.926 ms 1 1481 0.26 MB/sec execute 19 sec latency 40.611 ms test_26 fail mds1 2 times Failing mds1 on oleg426-server 1 1564 0.25 MB/sec execute 20 sec latency 154.793 ms Stopping /mnt/lustre-mds1 (opts:) on oleg426-server 1 1625 0.24 MB/sec execute 21 sec latency 315.804 ms 1 1625 0.23 MB/sec execute 22 sec latency 1318.587 ms 1 1625 0.22 MB/sec execute 23 sec latency 2318.743 ms 1 1625 0.21 MB/sec execute 24 sec latency 3321.185 ms 1 1625 0.20 MB/sec execute 25 sec latency 4321.435 ms 1 1625 0.20 MB/sec execute 26 sec latency 5324.299 ms 1 1625 0.19 MB/sec execute 27 sec latency 6324.573 ms 00:34:40 (1734672880) shut down 1 1625 0.18 MB/sec execute 28 sec latency 7327.371 ms facet: mds1 facet_host: oleg426-server facet_failover_host: oleg426-server 1 1625 0.18 MB/sec execute 29 sec latency 8327.741 ms 1 1625 0.17 MB/sec execute 30 sec latency 9329.714 ms 1 1625 0.16 MB/sec execute 31 sec latency 10329.911 ms 1 1625 0.16 MB/sec execute 32 sec latency 11330.105 ms 1 1625 0.15 MB/sec execute 33 sec latency 12330.774 ms 1 1625 0.15 MB/sec execute 34 sec latency 13331.491 ms 1 1625 0.15 MB/sec execute 35 sec latency 14331.652 ms 1 1625 0.14 MB/sec execute 36 sec latency 15332.531 ms 1 1625 0.14 MB/sec execute 37 sec latency 16333.897 ms 1 1625 0.13 MB/sec execute 38 sec latency 17334.121 ms Failover mds1 to oleg426-server mount facets: mds1 1 1625 0.13 MB/sec execute 39 sec latency 18334.952 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1625 0.13 MB/sec execute 40 sec latency 19337.986 ms oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all 1 1625 0.12 MB/sec execute 41 sec latency 20338.204 ms pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 00:34:54 (1734672894) targets are mounted 00:34:54 (1734672894) facet_failover done 1 1625 0.12 MB/sec execute 42 sec latency 21338.439 ms 1 1625 0.12 MB/sec execute 43 sec latency 22339.538 ms 1 1625 0.12 MB/sec execute 44 sec latency 23339.789 ms 1 1625 0.11 MB/sec execute 45 sec latency 24340.054 ms 1 1625 0.11 MB/sec execute 46 sec latency 25340.367 ms 1 1625 0.11 MB/sec execute 47 sec latency 26340.843 ms 1 1625 0.11 MB/sec execute 48 sec latency 27341.028 ms 1 1625 0.10 MB/sec execute 49 sec latency 28341.336 ms 1 1625 0.10 MB/sec execute 50 sec latency 29341.524 ms 1 1625 0.10 MB/sec execute 51 sec latency 30343.108 ms 1 1625 0.10 MB/sec execute 52 sec latency 31343.300 ms 1 1672 0.10 MB/sec execute 53 sec latency 31728.517 ms oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1805 0.10 MB/sec execute 54 sec latency 55.081 ms 1 1888 0.10 MB/sec execute 55 sec latency 146.974 ms 1 2024 0.10 MB/sec execute 56 sec latency 86.369 ms 1 2168 0.10 MB/sec execute 57 sec latency 33.272 ms 1 2367 0.12 MB/sec execute 58 sec latency 71.166 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4480 2204032 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3660800 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 37888 3628032 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7288832 1% /mnt/lustre 1 2428 0.12 MB/sec execute 59 sec latency 94.972 ms 1 2576 0.13 MB/sec execute 60 sec latency 69.650 ms 1 2896 0.19 MB/sec execute 61 sec latency 58.614 ms test_26 fail mds1 3 times 1 3013 0.20 MB/sec execute 62 sec latency 82.653 ms Failing mds1 on oleg426-server 1 3277 0.23 MB/sec execute 63 sec latency 118.522 ms Stopping /mnt/lustre-mds1 (opts:) on oleg426-server 1 3344 0.22 MB/sec execute 64 sec latency 487.123 ms 00:35:17 (1734672917) shut down facet: mds1 facet_host: oleg426-server facet_failover_host: oleg426-server 1 3344 0.22 MB/sec execute 65 sec latency 1487.328 ms 1 3344 0.22 MB/sec execute 66 sec latency 2489.977 ms 1 3344 0.21 MB/sec execute 67 sec latency 3490.287 ms 1 3344 0.21 MB/sec execute 68 sec latency 4490.608 ms 1 3344 0.21 MB/sec execute 69 sec latency 5490.878 ms 1 3344 0.20 MB/sec execute 70 sec latency 6491.062 ms 1 3344 0.20 MB/sec execute 71 sec latency 7492.005 ms 1 3344 0.20 MB/sec execute 72 sec latency 8492.166 ms 1 3344 0.20 MB/sec execute 73 sec latency 9493.248 ms 1 3344 0.19 MB/sec execute 74 sec latency 10493.655 ms Failover mds1 to oleg426-server mount facets: mds1 1 3344 0.19 MB/sec execute 75 sec latency 11493.987 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3344 0.19 MB/sec execute 76 sec latency 12495.327 ms 1 3344 0.19 MB/sec execute 77 sec latency 13495.591 ms 1 3344 0.18 MB/sec execute 78 sec latency 14496.355 ms oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 1 3344 0.18 MB/sec execute 79 sec latency 15496.607 ms Started lustre-MDT0000 00:35:31 (1734672931) targets are mounted 00:35:31 (1734672931) facet_failover done 1 3344 0.18 MB/sec execute 80 sec latency 16497.049 ms 1 3344 0.18 MB/sec execute 81 sec latency 17497.319 ms 1 3344 0.17 MB/sec execute 82 sec latency 18497.737 ms 1 3374 0.17 MB/sec execute 83 sec latency 19113.248 ms oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3579 0.21 MB/sec execute 84 sec latency 39.413 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3697 0.21 MB/sec execute 85 sec latency 39.336 ms 1 3816 0.22 MB/sec execute 86 sec latency 35.357 ms 1 3915 0.22 MB/sec execute 87 sec latency 48.928 ms 1 4009 0.22 MB/sec execute 88 sec latency 39.094 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5248 2203264 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 19456 3569664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 41984 3587072 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 61440 7156736 1% /mnt/lustre 1 4084 0.21 MB/sec execute 89 sec latency 53.033 ms 1 4180 0.21 MB/sec execute 90 sec latency 53.733 ms 1 4281 0.21 MB/sec execute 91 sec latency 100.964 ms 1 4352 0.21 MB/sec execute 92 sec latency 100.956 ms test_26 fail mds1 4 times 1 4439 0.21 MB/sec execute 93 sec latency 106.442 ms Failing mds1 on oleg426-server Stopping /mnt/lustre-mds1 (opts:) on oleg426-server 1 4488 0.21 MB/sec execute 94 sec latency 244.505 ms 1 4488 0.21 MB/sec execute 95 sec latency 1244.728 ms 00:35:47 (1734672947) shut down facet: mds1 facet_host: oleg426-server facet_failover_host: oleg426-server 1 4488 0.21 MB/sec execute 96 sec latency 2244.888 ms 1 4488 0.20 MB/sec execute 97 sec latency 3245.368 ms 1 4488 0.20 MB/sec execute 98 sec latency 4245.555 ms 1 4488 0.20 MB/sec execute 99 sec latency 5246.143 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec Failover mds1 to oleg426-server mount facets: mds1 1 cleanup 106 sec Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 107 sec 1 cleanup 108 sec oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 1 cleanup 109 sec Started lustre-MDT0000 00:36:02 (1734672962) targets are mounted 00:36:02 (1734672962) facet_failover done 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 0 cleanup 114 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 662 132.311 31728.489 Close 492 43.940 19113.125 Rename 29 40.898 112.373 Unlink 131 10.307 20.008 Qpathinfo 599 7.406 96.171 Qfileinfo 105 0.749 10.049 Qfsinfo 113 1.498 19.908 Sfileinfo 62 28.311 55.049 Find 230 3.475 136.800 WriteX 344 4.267 32.462 ReadX 1011 0.930 98.837 LockX 2 4.000 5.391 UnlockX 2 3.851 4.195 Flush 65 78.658 171.715 Throughput 0.200152 MB/sec 1 clients 1 procs max_latency=31728.517 ms stopping dbench on /mnt/lustre at Fri Dec 20 00:36:06 EST 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Dec 20 00:36:09 EST 2024 waiting for dbench pid 19469 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 1 of 1 processes prepared for launch 0 sec releasing clients 1 112 5.24 MB/sec warmup 1 sec latency 105.847 ms 1 200 3.92 MB/sec warmup 2 sec latency 99.326 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5888 2202624 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3591168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 45056 3570688 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 7161856 1% /mnt/lustre 1 294 3.61 MB/sec warmup 3 sec latency 58.799 ms 1 373 3.32 MB/sec warmup 4 sec latency 68.809 ms 1 462 3.23 MB/sec warmup 5 sec latency 59.713 ms 1 603 3.49 MB/sec warmup 6 sec latency 61.682 ms test_26 fail mds1 5 times 1 699 3.03 MB/sec warmup 7 sec latency 115.271 ms Failing mds1 on oleg426-server Stopping /mnt/lustre-mds1 (opts:) on oleg426-server 1 740 2.66 MB/sec warmup 8 sec latency 176.928 ms 00:36:18 (1734672978) shut down 1 740 2.37 MB/sec warmup 9 sec latency 1177.218 ms facet: mds1 facet_host: oleg426-server facet_failover_host: oleg426-server 1 740 2.13 MB/sec warmup 10 sec latency 2178.169 ms 1 740 1.94 MB/sec warmup 11 sec latency 3178.510 ms 1 740 1.78 MB/sec warmup 12 sec latency 4178.725 ms 1 740 1.64 MB/sec warmup 13 sec latency 5179.292 ms 1 740 1.52 MB/sec warmup 14 sec latency 6179.716 ms 1 740 1.42 MB/sec warmup 15 sec latency 7179.913 ms 1 740 1.33 MB/sec warmup 16 sec latency 8181.022 ms 1 740 1.25 MB/sec warmup 17 sec latency 9181.482 ms 1 740 1.18 MB/sec warmup 18 sec latency 10182.240 ms 1 740 1.12 MB/sec warmup 19 sec latency 11182.764 ms Failover mds1 to oleg426-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 740 0.00 MB/sec execute 1 sec latency 13183.252 ms 1 740 0.00 MB/sec execute 2 sec latency 14184.011 ms oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all 1 740 0.00 MB/sec execute 3 sec latency 15186.071 ms pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 00:36:32 (1734672992) targets are mounted 00:36:32 (1734672992) facet_failover done 1 740 0.00 MB/sec execute 4 sec latency 16186.338 ms 1 740 0.00 MB/sec execute 5 sec latency 17187.378 ms 1 740 0.00 MB/sec execute 6 sec latency 18187.530 ms 1 794 0.02 MB/sec execute 7 sec latency 18289.721 ms oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 873 0.04 MB/sec execute 8 sec latency 67.305 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 916 0.05 MB/sec execute 9 sec latency 130.426 ms 1 990 0.06 MB/sec execute 10 sec latency 131.349 ms 1 1082 0.15 MB/sec execute 11 sec latency 108.501 ms 1 1172 0.14 MB/sec execute 12 sec latency 51.667 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6784 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3535872 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 12288 3573760 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 52224 7109632 1% /mnt/lustre 1 1242 0.14 MB/sec execute 13 sec latency 54.171 ms 1 1409 0.34 MB/sec execute 14 sec latency 58.137 ms 1 1504 0.32 MB/sec execute 15 sec latency 101.128 ms 1 1591 0.30 MB/sec execute 16 sec latency 119.881 ms test_26 fail mds1 6 times 1 1675 0.29 MB/sec execute 17 sec latency 144.015 ms Failing mds1 on oleg426-server Stopping /mnt/lustre-mds1 (opts:) on oleg426-server 1 1804 0.28 MB/sec execute 18 sec latency 58.901 ms 1 1821 0.26 MB/sec execute 19 sec latency 831.763 ms 00:36:48 (1734673008) shut down facet: mds1 facet_host: oleg426-server facet_failover_host: oleg426-server 1 1821 0.25 MB/sec execute 20 sec latency 1831.920 ms 1 1821 0.24 MB/sec execute 21 sec latency 2832.081 ms 1 1821 0.23 MB/sec execute 22 sec latency 3832.586 ms 1 1821 0.22 MB/sec execute 23 sec latency 4834.289 ms 1 1821 0.21 MB/sec execute 24 sec latency 5834.471 ms 1 1821 0.20 MB/sec execute 25 sec latency 6838.080 ms 1 1821 0.19 MB/sec execute 26 sec latency 7838.343 ms 1 1821 0.19 MB/sec execute 27 sec latency 8838.668 ms 1 1821 0.18 MB/sec execute 28 sec latency 9838.898 ms 1 1821 0.17 MB/sec execute 29 sec latency 10839.127 ms Failover mds1 to oleg426-server mount facets: mds1 1 1821 0.17 MB/sec execute 30 sec latency 11841.283 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1821 0.16 MB/sec execute 31 sec latency 12843.174 ms 1 1821 0.16 MB/sec execute 32 sec latency 13843.375 ms oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all 1 1821 0.15 MB/sec execute 33 sec latency 14843.512 ms pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 00:37:02 (1734673022) targets are mounted 00:37:02 (1734673022) facet_failover done 1 1821 0.15 MB/sec execute 34 sec latency 15843.687 ms 1 1821 0.14 MB/sec execute 35 sec latency 16843.817 ms 1 1830 0.14 MB/sec execute 36 sec latency 17666.021 ms oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1961 0.14 MB/sec execute 37 sec latency 116.652 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2063 0.14 MB/sec execute 38 sec latency 42.825 ms 1 2206 0.14 MB/sec execute 39 sec latency 37.126 ms 1 2350 0.16 MB/sec execute 40 sec latency 152.374 ms 1 2416 0.17 MB/sec execute 41 sec latency 117.213 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 7680 2200832 1% /mnt/lustre[MDT:0] 1 2479 0.18 MB/sec execute 42 sec latency 159.277 ms lustre-OST0000_UUID 3771392 49152 3588096 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3602432 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 69632 7190528 1% /mnt/lustre 1 2608 0.18 MB/sec execute 43 sec latency 58.977 ms 1 2917 0.28 MB/sec execute 44 sec latency 95.596 ms 1 3024 0.27 MB/sec execute 45 sec latency 88.617 ms test_26 fail mds1 7 times 1 3305 0.31 MB/sec execute 46 sec latency 120.988 ms Failing mds1 on oleg426-server Stopping /mnt/lustre-mds1 (opts:) on oleg426-server 1 3513 0.36 MB/sec execute 47 sec latency 138.429 ms 1 3538 0.36 MB/sec execute 48 sec latency 812.684 ms 00:37:17 (1734673037) shut down facet: mds1 facet_host: oleg426-server facet_failover_host: oleg426-server 1 3538 0.35 MB/sec execute 49 sec latency 1813.019 ms 1 3538 0.34 MB/sec execute 50 sec latency 2813.743 ms 1 3538 0.33 MB/sec execute 51 sec latency 3813.936 ms 1 3538 0.33 MB/sec execute 52 sec latency 4814.108 ms 1 3538 0.32 MB/sec execute 53 sec latency 5814.284 ms 1 3538 0.32 MB/sec execute 54 sec latency 6814.492 ms 1 3538 0.31 MB/sec execute 55 sec latency 7814.710 ms 1 3538 0.30 MB/sec execute 56 sec latency 8814.997 ms 1 3538 0.30 MB/sec execute 57 sec latency 9815.819 ms 1 3538 0.29 MB/sec execute 58 sec latency 10816.028 ms Failover mds1 to oleg426-server mount facets: mds1 1 3538 0.29 MB/sec execute 59 sec latency 11816.329 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3538 0.28 MB/sec execute 60 sec latency 12816.650 ms 1 3538 0.28 MB/sec execute 61 sec latency 13817.034 ms oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all 1 3538 0.28 MB/sec execute 62 sec latency 14817.148 ms pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 00:37:31 (1734673051) targets are mounted 00:37:31 (1734673051) facet_failover done 1 3538 0.27 MB/sec execute 63 sec latency 15817.645 ms 1 3538 0.27 MB/sec execute 64 sec latency 16817.844 ms 1 3538 0.26 MB/sec execute 65 sec latency 17817.977 ms 1 3560 0.26 MB/sec execute 66 sec latency 18593.240 ms oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3714 0.26 MB/sec execute 67 sec latency 31.845 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3839 0.27 MB/sec execute 68 sec latency 45.899 ms 1 3967 0.27 MB/sec execute 69 sec latency 32.326 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 4063 0.27 MB/sec execute 70 sec latency 40.630 ms 1 4161 0.26 MB/sec execute 71 sec latency 58.313 ms 1 4263 0.26 MB/sec execute 72 sec latency 144.789 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Dec 20 00:37:42 EST 2024 with return code 0 19469 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 19469 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished