== replay-dual test 26: dbench and tar with mds failover ========================================================== 03:53:17 (1722239597) Starting client oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Started clients oleg416-client.virtnet: 192.168.204.116@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 15697 Started dbench loop with 15698 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Jul 29 03:53:18 EDT 2024 waiting for dbench pid 15723 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 280 10.44 MB/sec warmup 1 sec latency 19.475 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3633152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3725312 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7358464 1% /mnt/lustre 1 572 9.58 MB/sec warmup 2 sec latency 18.517 ms 1 847 7.19 MB/sec warmup 3 sec latency 70.821 ms 1 1127 5.75 MB/sec warmup 4 sec latency 38.432 ms test_26 fail mds1 1 times Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 1490 5.21 MB/sec warmup 5 sec latency 49.474 ms reboot facets: mds1 1 1500 4.34 MB/sec warmup 6 sec latency 966.760 ms 1 1500 3.72 MB/sec warmup 7 sec latency 1966.943 ms 1 1500 3.26 MB/sec warmup 8 sec latency 2967.135 ms 1 1500 2.89 MB/sec warmup 9 sec latency 3967.285 ms 1 1500 2.60 MB/sec warmup 10 sec latency 4967.443 ms 1 1500 2.37 MB/sec warmup 11 sec latency 5967.613 ms 1 1500 2.17 MB/sec warmup 12 sec latency 6967.797 ms 1 1500 2.00 MB/sec warmup 13 sec latency 7967.971 ms 1 1500 1.86 MB/sec warmup 14 sec latency 8968.122 ms 1 1500 1.74 MB/sec warmup 15 sec latency 9968.308 ms Failover mds1 to oleg416-server 1 1500 1.63 MB/sec warmup 16 sec latency 10968.482 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1500 1.53 MB/sec warmup 17 sec latency 11968.642 ms 1 1500 1.45 MB/sec warmup 18 sec latency 12968.837 ms 1 1500 1.37 MB/sec warmup 19 sec latency 13969.024 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 8 1 1500 0.00 MB/sec execute 1 sec latency 15969.277 ms pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1500 0.00 MB/sec execute 2 sec latency 16969.508 ms 1 1500 0.00 MB/sec execute 3 sec latency 17969.719 ms 1 1500 0.00 MB/sec execute 4 sec latency 18969.895 ms 1 1500 0.00 MB/sec execute 5 sec latency 19970.075 ms 1 1500 0.00 MB/sec execute 6 sec latency 20970.245 ms 1 1500 0.00 MB/sec execute 7 sec latency 21970.419 ms 1 1500 0.00 MB/sec execute 8 sec latency 22970.595 ms 1 1500 0.00 MB/sec execute 9 sec latency 23970.789 ms 1 1500 0.00 MB/sec execute 10 sec latency 24971.002 ms 1 1500 0.00 MB/sec execute 11 sec latency 25971.193 ms 1 1500 0.00 MB/sec execute 12 sec latency 26971.348 ms 1 1500 0.00 MB/sec execute 13 sec latency 27971.515 ms 1 1500 0.00 MB/sec execute 14 sec latency 28971.669 ms 1 1560 0.00 MB/sec execute 15 sec latency 29814.757 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1934 0.02 MB/sec execute 16 sec latency 42.556 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2337 0.10 MB/sec execute 17 sec latency 25.938 ms 1 2765 0.33 MB/sec execute 18 sec latency 35.755 ms 1 3352 0.49 MB/sec execute 19 sec latency 43.005 ms 1 3801 0.69 MB/sec execute 20 sec latency 15.833 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4864 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3647488 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3573760 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 60416 7221248 1% /mnt/lustre 1 4041 0.67 MB/sec execute 21 sec latency 15.562 ms 1 4347 0.66 MB/sec execute 22 sec latency 57.188 ms test_26 fail mds1 2 times 1 4645 0.70 MB/sec execute 23 sec latency 49.146 ms Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 5029 0.80 MB/sec execute 24 sec latency 156.627 ms reboot facets: mds1 1 5029 0.76 MB/sec execute 25 sec latency 1156.776 ms 1 5029 0.73 MB/sec execute 26 sec latency 2156.945 ms 1 5029 0.71 MB/sec execute 27 sec latency 3157.373 ms 1 5029 0.68 MB/sec execute 28 sec latency 4157.556 ms 1 5029 0.66 MB/sec execute 29 sec latency 5157.761 ms 1 5029 0.64 MB/sec execute 30 sec latency 6157.920 ms 1 5029 0.62 MB/sec execute 31 sec latency 7158.096 ms 1 5029 0.60 MB/sec execute 32 sec latency 8158.282 ms 1 5029 0.58 MB/sec execute 33 sec latency 9158.444 ms 1 5029 0.56 MB/sec execute 34 sec latency 10158.594 ms Failover mds1 to oleg416-server mount facets: mds1 1 5029 0.55 MB/sec execute 35 sec latency 11158.826 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5029 0.53 MB/sec execute 36 sec latency 12159.003 ms 1 5029 0.52 MB/sec execute 37 sec latency 13159.173 ms 1 5029 0.50 MB/sec execute 38 sec latency 14159.321 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 8 1 5029 0.49 MB/sec execute 39 sec latency 15159.435 ms pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 1 5029 0.48 MB/sec execute 40 sec latency 16159.552 ms 1 5029 0.47 MB/sec execute 41 sec latency 17159.765 ms 1 5313 0.46 MB/sec execute 42 sec latency 17194.442 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5701 0.46 MB/sec execute 43 sec latency 46.814 ms 1 6037 0.50 MB/sec execute 44 sec latency 49.019 ms 1 6580 0.59 MB/sec execute 45 sec latency 37.368 ms 1 7230 0.69 MB/sec execute 46 sec latency 30.252 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 6912 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 46080 3683328 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 21504 3707904 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 7391232 1% /mnt/lustre 1 7512 0.70 MB/sec execute 47 sec latency 17.331 ms 1 7799 0.70 MB/sec execute 48 sec latency 51.172 ms 1 8025 0.69 MB/sec execute 49 sec latency 56.253 ms test_26 fail mds1 3 times Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 8300 0.70 MB/sec execute 50 sec latency 41.426 ms 1 8339 0.69 MB/sec execute 51 sec latency 843.255 ms reboot facets: mds1 1 8339 0.68 MB/sec execute 52 sec latency 1843.450 ms 1 8339 0.67 MB/sec execute 53 sec latency 2843.615 ms 1 8339 0.65 MB/sec execute 54 sec latency 3843.809 ms 1 8339 0.64 MB/sec execute 55 sec latency 4844.003 ms 1 8339 0.63 MB/sec execute 56 sec latency 5844.186 ms 1 8339 0.62 MB/sec execute 57 sec latency 6844.392 ms 1 8339 0.61 MB/sec execute 58 sec latency 7844.637 ms 1 8339 0.60 MB/sec execute 59 sec latency 8844.899 ms 1 8339 0.59 MB/sec execute 60 sec latency 9845.168 ms 1 8339 0.58 MB/sec execute 61 sec latency 10845.346 ms Failover mds1 to oleg416-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 8339 0.57 MB/sec execute 62 sec latency 11845.460 ms 1 8339 0.56 MB/sec execute 63 sec latency 12845.585 ms 1 8339 0.55 MB/sec execute 64 sec latency 13845.709 ms 1 8339 0.54 MB/sec execute 65 sec latency 14845.853 ms 1 8339 0.53 MB/sec execute 66 sec latency 15846.055 ms 1 8339 0.53 MB/sec execute 67 sec latency 16846.220 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 1 8339 0.52 MB/sec execute 68 sec latency 17846.384 ms 1 8339 0.51 MB/sec execute 69 sec latency 18846.551 ms 1 8339 0.50 MB/sec execute 70 sec latency 19846.703 ms 1 8339 0.50 MB/sec execute 71 sec latency 20846.882 ms 1 8339 0.49 MB/sec execute 72 sec latency 21847.068 ms 1 8339 0.48 MB/sec execute 73 sec latency 22847.267 ms 1 8339 0.48 MB/sec execute 74 sec latency 23847.424 ms 1 8339 0.47 MB/sec execute 75 sec latency 24847.600 ms 1 8339 0.46 MB/sec execute 76 sec latency 25847.772 ms 1 8339 0.46 MB/sec execute 77 sec latency 26847.943 ms 1 8339 0.45 MB/sec execute 78 sec latency 27848.087 ms 1 8339 0.45 MB/sec execute 79 sec latency 28848.250 ms 1 8339 0.44 MB/sec execute 80 sec latency 29848.371 ms 1 8634 0.47 MB/sec execute 81 sec latency 30131.530 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8948 0.47 MB/sec execute 82 sec latency 35.988 ms 1 9356 0.47 MB/sec execute 83 sec latency 45.683 ms 1 9684 0.49 MB/sec execute 84 sec latency 60.154 ms 1 10303 0.55 MB/sec execute 85 sec latency 47.229 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 8832 2199808 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 48128 3694592 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3575808 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 71680 7270400 1% /mnt/lustre 1 10779 0.59 MB/sec execute 86 sec latency 20.510 ms 1 11051 0.60 MB/sec execute 87 sec latency 15.427 ms 1 11346 0.60 MB/sec execute 88 sec latency 77.615 ms test_26 fail mds1 4 times Failing mds1 on oleg416-server 1 11607 0.60 MB/sec execute 89 sec latency 44.222 ms Stopping /mnt/lustre-mds1 (opts:) on oleg416-server reboot facets: mds1 1 11695 0.60 MB/sec execute 90 sec latency 728.402 ms 1 11695 0.60 MB/sec execute 91 sec latency 1728.610 ms 1 11695 0.59 MB/sec execute 92 sec latency 2728.837 ms 1 11695 0.58 MB/sec execute 93 sec latency 3729.052 ms 1 11695 0.58 MB/sec execute 94 sec latency 4729.268 ms 1 11695 0.57 MB/sec execute 95 sec latency 5729.453 ms 1 11695 0.56 MB/sec execute 96 sec latency 6729.615 ms 1 11695 0.56 MB/sec execute 97 sec latency 7729.775 ms 1 11695 0.55 MB/sec execute 98 sec latency 8729.935 ms 1 11695 0.55 MB/sec execute 99 sec latency 9730.102 ms Failover mds1 to oleg416-server 1 cleanup 100 sec mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 1 cleanup 106 sec 0 cleanup 107 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1769 43.903 29814.742 Close 1289 1.417 5.994 Rename 72 428.669 30131.515 Unlink 363 3.893 14.135 Qpathinfo 1610 1.969 29.760 Qfileinfo 277 0.349 1.810 Qfsinfo 295 0.376 2.104 Sfileinfo 134 5.066 9.318 Find 619 1.021 9.629 WriteX 878 1.709 6.795 ReadX 2750 0.058 25.926 LockX 6 1.320 1.558 UnlockX 6 1.279 1.554 Flush 128 24.817 77.601 Throughput 0.547308 MB/sec 1 clients 1 procs max_latency=30131.530 ms stopping dbench on /mnt/lustre at Mon Jul 29 03:55:25 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Jul 29 03:55:26 EDT 2024 waiting for dbench pid 19842 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 of 1 processes prepared for launch 0 sec releasing clients mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 289 10.70 MB/sec warmup 1 sec latency 25.706 ms 1 642 10.53 MB/sec warmup 2 sec latency 17.171 ms 1 882 7.22 MB/sec warmup 3 sec latency 77.368 ms 1 1257 5.77 MB/sec warmup 4 sec latency 26.294 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10496 2198144 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3596288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 53248 3537920 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 69632 7134208 1% /mnt/lustre 1 1689 5.24 MB/sec warmup 5 sec latency 45.339 ms 1 2086 4.44 MB/sec warmup 6 sec latency 38.551 ms test_26 fail mds1 5 times 1 2457 4.02 MB/sec warmup 7 sec latency 38.631 ms Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 2937 4.19 MB/sec warmup 8 sec latency 126.015 ms reboot facets: mds1 1 2937 3.72 MB/sec warmup 9 sec latency 1126.184 ms 1 2937 3.35 MB/sec warmup 10 sec latency 2126.354 ms 1 2937 3.04 MB/sec warmup 11 sec latency 3126.542 ms 1 2937 2.79 MB/sec warmup 12 sec latency 4126.685 ms 1 2937 2.58 MB/sec warmup 13 sec latency 5126.897 ms 1 2937 2.39 MB/sec warmup 14 sec latency 6127.029 ms 1 2937 2.23 MB/sec warmup 15 sec latency 7127.182 ms 1 2937 2.09 MB/sec warmup 16 sec latency 8127.317 ms 1 2937 1.97 MB/sec warmup 17 sec latency 9127.478 ms 1 2937 1.86 MB/sec warmup 18 sec latency 10127.635 ms Failover mds1 to oleg416-server mount facets: mds1 1 2937 1.76 MB/sec warmup 19 sec latency 11127.814 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2937 0.00 MB/sec execute 1 sec latency 13128.021 ms 1 2937 0.00 MB/sec execute 2 sec latency 14128.155 ms 1 2937 0.00 MB/sec execute 3 sec latency 15128.338 ms 1 2937 0.00 MB/sec execute 4 sec latency 16128.492 ms 1 2937 0.00 MB/sec execute 5 sec latency 17128.673 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 1 2937 0.00 MB/sec execute 6 sec latency 18128.840 ms 1 2937 0.00 MB/sec execute 7 sec latency 19128.959 ms 1 2937 0.00 MB/sec execute 8 sec latency 20129.113 ms 1 2937 0.00 MB/sec execute 9 sec latency 21129.288 ms 1 2937 0.00 MB/sec execute 10 sec latency 22129.475 ms 1 2937 0.00 MB/sec execute 11 sec latency 23129.614 ms 1 2937 0.00 MB/sec execute 12 sec latency 24129.775 ms 1 2937 0.00 MB/sec execute 13 sec latency 25129.929 ms 1 2937 0.00 MB/sec execute 14 sec latency 26130.063 ms 1 2937 0.00 MB/sec execute 15 sec latency 27130.211 ms 1 2937 0.00 MB/sec execute 16 sec latency 28130.403 ms 1 2937 0.00 MB/sec execute 17 sec latency 29130.506 ms 1 3209 0.04 MB/sec execute 18 sec latency 29674.238 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3707 0.28 MB/sec execute 19 sec latency 17.065 ms 1 3995 0.33 MB/sec execute 20 sec latency 20.161 ms 1 4283 0.33 MB/sec execute 21 sec latency 87.226 ms 1 4549 0.34 MB/sec execute 22 sec latency 49.264 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10496 2198144 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 27648 3702784 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 66560 3687424 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 94208 7390208 2% /mnt/lustre 1 5009 0.51 MB/sec execute 23 sec latency 29.703 ms 1 5284 0.50 MB/sec execute 24 sec latency 69.848 ms 1 5670 0.49 MB/sec execute 25 sec latency 44.567 ms test_26 fail mds1 6 times 1 6037 0.55 MB/sec execute 26 sec latency 58.426 ms Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 6452 0.68 MB/sec execute 27 sec latency 385.973 ms reboot facets: mds1 1 6452 0.66 MB/sec execute 28 sec latency 1386.099 ms 1 6452 0.64 MB/sec execute 29 sec latency 2386.271 ms 1 6452 0.62 MB/sec execute 30 sec latency 3386.426 ms 1 6452 0.60 MB/sec execute 31 sec latency 4386.610 ms 1 6452 0.58 MB/sec execute 32 sec latency 5386.799 ms 1 6452 0.56 MB/sec execute 33 sec latency 6386.987 ms 1 6452 0.54 MB/sec execute 34 sec latency 7387.140 ms 1 6452 0.53 MB/sec execute 35 sec latency 8387.313 ms 1 6452 0.51 MB/sec execute 36 sec latency 9387.510 ms 1 6452 0.50 MB/sec execute 37 sec latency 10387.747 ms Failover mds1 to oleg416-server mount facets: mds1 1 6452 0.49 MB/sec execute 38 sec latency 11387.932 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 6452 0.47 MB/sec execute 39 sec latency 12388.097 ms 1 6452 0.46 MB/sec execute 40 sec latency 13388.286 ms 1 6452 0.45 MB/sec execute 41 sec latency 14388.513 ms 1 6452 0.44 MB/sec execute 42 sec latency 15388.734 ms 1 6452 0.43 MB/sec execute 43 sec latency 16388.913 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 1 6452 0.42 MB/sec execute 44 sec latency 17389.012 ms 1 6537 0.43 MB/sec execute 45 sec latency 18124.702 ms 1 7136 0.52 MB/sec execute 46 sec latency 32.520 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7454 0.55 MB/sec execute 47 sec latency 16.000 ms 1 7704 0.54 MB/sec execute 48 sec latency 17.193 ms 1 7914 0.54 MB/sec execute 49 sec latency 111.344 ms 1 8226 0.55 MB/sec execute 50 sec latency 44.130 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 7296 2201344 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 61440 3692544 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 87040 7429120 2% /mnt/lustre 1 8611 0.60 MB/sec execute 51 sec latency 72.357 ms 1 8944 0.60 MB/sec execute 52 sec latency 30.906 ms 1 9390 0.61 MB/sec execute 53 sec latency 51.857 ms test_26 fail mds1 7 times Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 9757 0.64 MB/sec execute 54 sec latency 54.584 ms reboot facets: mds1 1 9884 0.67 MB/sec execute 55 sec latency 799.916 ms 1 9884 0.66 MB/sec execute 56 sec latency 1800.156 ms 1 9884 0.64 MB/sec execute 57 sec latency 2800.390 ms 1 9884 0.63 MB/sec execute 58 sec latency 3800.563 ms 1 9884 0.62 MB/sec execute 59 sec latency 4800.773 ms 1 9884 0.61 MB/sec execute 60 sec latency 5800.966 ms 1 9884 0.60 MB/sec execute 61 sec latency 6801.123 ms 1 9884 0.59 MB/sec execute 62 sec latency 7801.272 ms 1 9884 0.58 MB/sec execute 63 sec latency 8801.473 ms 1 9884 0.57 MB/sec execute 64 sec latency 9801.666 ms Failover mds1 to oleg416-server 1 9884 0.56 MB/sec execute 65 sec latency 10801.848 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 9884 0.56 MB/sec execute 66 sec latency 11801.976 ms 1 9884 0.55 MB/sec execute 67 sec latency 12802.139 ms 1 9884 0.54 MB/sec execute 68 sec latency 13802.335 ms 1 9884 0.53 MB/sec execute 69 sec latency 14802.502 ms 1 9884 0.52 MB/sec execute 70 sec latency 15802.694 ms 1 9884 0.52 MB/sec execute 71 sec latency 16802.861 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 1 9884 0.51 MB/sec execute 72 sec latency 17802.967 ms Started lustre-MDT0000 1 9884 0.50 MB/sec execute 73 sec latency 18803.161 ms 1 9884 0.50 MB/sec execute 74 sec latency 19803.402 ms 1 9884 0.49 MB/sec execute 75 sec latency 20803.669 ms 1 9884 0.48 MB/sec execute 76 sec latency 21803.861 ms 1 9884 0.48 MB/sec execute 77 sec latency 22804.046 ms 1 9884 0.47 MB/sec execute 78 sec latency 23804.268 ms 1 9884 0.46 MB/sec execute 79 sec latency 24804.484 ms 1 9884 0.46 MB/sec execute 80 sec latency 25804.726 ms 1 9884 0.45 MB/sec execute 81 sec latency 26804.962 ms 1 9884 0.45 MB/sec execute 82 sec latency 27805.219 ms 1 9884 0.44 MB/sec execute 83 sec latency 28805.446 ms 1 9884 0.44 MB/sec execute 84 sec latency 29805.705 ms 1 9884 0.43 MB/sec execute 85 sec latency 30805.876 ms 1 9989 0.43 MB/sec execute 86 sec latency 31691.966 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 10530 0.47 MB/sec execute 87 sec latency 36.327 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 10922 0.51 MB/sec execute 88 sec latency 13.868 ms 1 11197 0.50 MB/sec execute 89 sec latency 14.049 ms 1 11439 0.50 MB/sec execute 90 sec latency 105.291 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Jul 29 03:57:17 EDT 2024 with return code 0 19842 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 19842 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished