== replay-dual test 26: dbench and tar with mds failover ========================================================== 21:23:20 (1746149000) Starting client oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre Started clients oleg420-client.virtnet: 192.168.204.120@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 4327 Started dbench loop with 4328 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu May 1 21:23:22 EDT 2025 waiting for dbench pid 4378 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2136 1285552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1908 1285780 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26112 3580768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27656 7186244 1% /mnt/lustre 1 111 5.24 MB/sec warmup 1 sec latency 42.638 ms 1 189 3.79 MB/sec warmup 2 sec latency 90.075 ms 1 230 2.96 MB/sec warmup 3 sec latency 514.456 ms 1 313 2.87 MB/sec warmup 4 sec latency 627.322 ms 1 436 3.08 MB/sec warmup 5 sec latency 55.834 ms test_26 fail mds1 1 times 1 557 3.18 MB/sec warmup 6 sec latency 53.542 ms Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 1 699 3.03 MB/sec warmup 7 sec latency 45.131 ms 1 744 2.66 MB/sec warmup 8 sec latency 367.181 ms 21:23:31 (1746149011) shut down facet: mds1 facet_host: oleg420-server facet_failover_host: oleg420-server 1 744 2.37 MB/sec warmup 9 sec latency 1367.656 ms 1 744 2.13 MB/sec warmup 10 sec latency 2367.903 ms 1 744 1.94 MB/sec warmup 11 sec latency 3368.595 ms 1 744 1.78 MB/sec warmup 12 sec latency 4371.973 ms 1 744 1.64 MB/sec warmup 13 sec latency 5373.750 ms 1 744 1.52 MB/sec warmup 14 sec latency 6374.452 ms 1 744 1.42 MB/sec warmup 15 sec latency 7374.617 ms 1 744 1.33 MB/sec warmup 16 sec latency 8374.967 ms 1 744 1.25 MB/sec warmup 17 sec latency 9375.125 ms 1 744 1.18 MB/sec warmup 18 sec latency 10375.275 ms Failover mds1 to oleg420-server mount facets: mds1 1 744 1.12 MB/sec warmup 19 sec latency 11375.431 ms 1 744 0.00 MB/sec execute 1 sec latency 13375.671 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 744 0.00 MB/sec execute 2 sec latency 14376.009 ms 1 744 0.00 MB/sec execute 3 sec latency 15376.191 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 1 744 0.00 MB/sec execute 4 sec latency 16378.359 ms 21:23:47 (1746149027) targets are mounted 21:23:47 (1746149027) facet_failover done 1 744 0.00 MB/sec execute 5 sec latency 17378.457 ms 1 744 0.00 MB/sec execute 6 sec latency 18378.723 ms 1 744 0.00 MB/sec execute 7 sec latency 19379.318 ms 1 745 0.00 MB/sec execute 8 sec latency 20371.878 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 864 0.03 MB/sec execute 9 sec latency 40.598 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 955 0.05 MB/sec execute 10 sec latency 41.103 ms 1 1074 0.15 MB/sec execute 11 sec latency 72.682 ms 1 1211 0.14 MB/sec execute 12 sec latency 39.967 ms 1 1406 0.36 MB/sec execute 13 sec latency 37.785 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2712 1284976 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 7852 3590284 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 32912 3565160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 40764 7155444 1% /mnt/lustre 1 1512 0.34 MB/sec execute 14 sec latency 45.220 ms 1 1646 0.33 MB/sec execute 15 sec latency 40.353 ms 1 1736 0.31 MB/sec execute 16 sec latency 65.519 ms 1 1849 0.30 MB/sec execute 17 sec latency 46.182 ms 1 1980 0.29 MB/sec execute 18 sec latency 39.949 ms 1 2128 0.28 MB/sec execute 19 sec latency 40.368 ms test_26 fail mds2 2 times Failing mds2 on oleg420-server 1 2272 0.29 MB/sec execute 20 sec latency 40.269 ms Stopping /mnt/lustre-mds2 (opts:) on oleg420-server 1 2412 0.32 MB/sec execute 21 sec latency 39.244 ms 21:24:04 (1746149044) shut down facet: mds2 facet_host: oleg420-server facet_failover_host: oleg420-server 1 2576 0.35 MB/sec execute 22 sec latency 43.648 ms 1 2896 0.50 MB/sec execute 23 sec latency 65.170 ms 1 3026 0.51 MB/sec execute 24 sec latency 38.352 ms 1 3359 0.56 MB/sec execute 25 sec latency 34.681 ms 1 3599 0.66 MB/sec execute 26 sec latency 29.294 ms 1 3768 0.68 MB/sec execute 27 sec latency 33.222 ms 1 3889 0.66 MB/sec execute 28 sec latency 24.532 ms 1 4010 0.65 MB/sec execute 29 sec latency 29.026 ms 1 4083 0.63 MB/sec execute 30 sec latency 386.163 ms 1 4083 0.61 MB/sec execute 31 sec latency 1386.389 ms Failover mds2 to oleg420-server mount facets: mds2 1 4083 0.59 MB/sec execute 32 sec latency 2386.592 ms 1 4083 0.57 MB/sec execute 33 sec latency 3386.751 ms 1 4083 0.55 MB/sec execute 34 sec latency 4387.214 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4083 0.54 MB/sec execute 35 sec latency 5387.493 ms 1 4083 0.52 MB/sec execute 36 sec latency 6387.768 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 1 4083 0.51 MB/sec execute 37 sec latency 7387.944 ms Started lustre-MDT0001 21:24:20 (1746149060) targets are mounted 21:24:20 (1746149060) facet_failover done 1 4083 0.49 MB/sec execute 38 sec latency 8388.455 ms 1 4083 0.48 MB/sec execute 39 sec latency 9388.648 ms 1 4083 0.47 MB/sec execute 40 sec latency 10390.684 ms 1 4083 0.46 MB/sec execute 41 sec latency 11391.075 ms 1 4165 0.45 MB/sec execute 42 sec latency 11720.672 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4292 0.44 MB/sec execute 43 sec latency 47.816 ms 1 4384 0.44 MB/sec execute 44 sec latency 56.880 ms 1 4489 0.43 MB/sec execute 45 sec latency 42.066 ms 1 4627 0.45 MB/sec execute 46 sec latency 42.523 ms 1 4756 0.44 MB/sec execute 47 sec latency 33.697 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2840 1284848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2544 1285144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12440 3591648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37344 3566712 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49784 7158360 1% /mnt/lustre 1 4948 0.50 MB/sec execute 48 sec latency 36.005 ms 1 5052 0.49 MB/sec execute 49 sec latency 48.583 ms 1 5145 0.48 MB/sec execute 50 sec latency 40.583 ms 1 5240 0.47 MB/sec execute 51 sec latency 51.783 ms 1 5342 0.46 MB/sec execute 52 sec latency 54.753 ms test_26 fail mds1 3 times 1 5403 0.46 MB/sec execute 53 sec latency 81.704 ms Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 1 5519 0.45 MB/sec execute 54 sec latency 50.836 ms 1 5589 0.44 MB/sec execute 55 sec latency 62.491 ms 21:24:38 (1746149078) shut down facet: mds1 facet_host: oleg420-server facet_failover_host: oleg420-server 1 5724 0.44 MB/sec execute 56 sec latency 49.554 ms 1 5820 0.43 MB/sec execute 57 sec latency 269.506 ms 1 5820 0.42 MB/sec execute 58 sec latency 1269.979 ms 1 5820 0.42 MB/sec execute 59 sec latency 2270.342 ms 1 5820 0.41 MB/sec execute 60 sec latency 3270.885 ms 1 5820 0.40 MB/sec execute 61 sec latency 4274.464 ms 1 5820 0.40 MB/sec execute 62 sec latency 5274.615 ms 1 5820 0.39 MB/sec execute 63 sec latency 6274.763 ms 1 5820 0.38 MB/sec execute 64 sec latency 7275.047 ms 1 5820 0.38 MB/sec execute 65 sec latency 8275.207 ms Failover mds1 to oleg420-server mount facets: mds1 1 5820 0.37 MB/sec execute 66 sec latency 9275.376 ms 1 5820 0.37 MB/sec execute 67 sec latency 10275.536 ms 1 5820 0.36 MB/sec execute 68 sec latency 11276.779 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5820 0.36 MB/sec execute 69 sec latency 12279.326 ms 1 5820 0.35 MB/sec execute 70 sec latency 13279.544 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 1 5820 0.35 MB/sec execute 71 sec latency 14280.721 ms Started lustre-MDT0000 21:24:54 (1746149094) targets are mounted 21:24:54 (1746149094) facet_failover done 1 5820 0.34 MB/sec execute 72 sec latency 15282.613 ms 1 5820 0.34 MB/sec execute 73 sec latency 16289.720 ms 1 5820 0.33 MB/sec execute 74 sec latency 17289.908 ms 1 5820 0.33 MB/sec execute 75 sec latency 18290.239 ms 1 5820 0.32 MB/sec execute 76 sec latency 19290.380 ms 1 5896 0.33 MB/sec execute 77 sec latency 19934.822 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5986 0.33 MB/sec execute 78 sec latency 39.316 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6117 0.34 MB/sec execute 79 sec latency 45.535 ms 1 6326 0.37 MB/sec execute 80 sec latency 54.150 ms 1 6545 0.39 MB/sec execute 81 sec latency 35.102 ms 1 6852 0.40 MB/sec execute 82 sec latency 35.378 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3036 1284652 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12548 3585712 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37808 3561172 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50356 7146884 1% /mnt/lustre 1 7077 0.44 MB/sec execute 83 sec latency 48.935 ms 1 7187 0.43 MB/sec execute 84 sec latency 53.899 ms 1 7304 0.44 MB/sec execute 85 sec latency 52.216 ms 1 7380 0.44 MB/sec execute 86 sec latency 63.681 ms 1 7483 0.43 MB/sec execute 87 sec latency 40.546 ms test_26 fail mds2 4 times 1 7566 0.43 MB/sec execute 88 sec latency 40.278 ms Failing mds2 on oleg420-server 1 7642 0.43 MB/sec execute 89 sec latency 74.439 ms Stopping /mnt/lustre-mds2 (opts:) on oleg420-server 1 7721 0.42 MB/sec execute 90 sec latency 283.436 ms 21:25:14 (1746149114) shut down facet: mds2 facet_host: oleg420-server facet_failover_host: oleg420-server 1 7721 0.42 MB/sec execute 91 sec latency 1284.641 ms 1 7721 0.41 MB/sec execute 92 sec latency 2287.102 ms 1 7721 0.41 MB/sec execute 93 sec latency 3287.253 ms 1 7721 0.40 MB/sec execute 94 sec latency 4287.664 ms 1 7721 0.40 MB/sec execute 95 sec latency 5287.808 ms 1 7721 0.39 MB/sec execute 96 sec latency 6287.915 ms 1 7721 0.39 MB/sec execute 97 sec latency 7288.482 ms 1 7721 0.39 MB/sec execute 98 sec latency 8288.629 ms 1 7721 0.38 MB/sec execute 99 sec latency 9288.937 ms 1 cleanup 100 sec 1 cleanup 101 sec Failover mds2 to oleg420-server mount facets: mds2 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 cleanup 105 sec 1 cleanup 106 sec oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 1 cleanup 107 sec pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 21:25:30 (1746149130) targets are mounted 21:25:30 (1746149130) facet_failover done 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 0 cleanup 112 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1204 38.180 19934.812 Close 881 4.628 18.342 Rename 51 31.793 65.496 Unlink 245 9.794 37.121 Qpathinfo 1129 34.616 20952.080 Qfileinfo 190 0.725 6.213 Qfsinfo 199 1.739 11.021 Sfileinfo 94 21.788 47.806 Find 423 52.963 20371.869 WriteX 586 4.387 34.732 ReadX 1894 0.327 62.959 LockX 4 3.403 4.454 UnlockX 4 3.075 4.120 Flush 74 22.628 50.822 Throughput 0.382448 MB/sec 1 clients 1 procs max_latency=20371.878 ms stopping dbench on /mnt/lustre at Thu May 1 21:25:35 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu May 1 21:25:38 EDT 2025 waiting for dbench pid 8853 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 79 3.46 MB/sec warmup 1 sec latency 62.421 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3848 1283840 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9112 3590592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30624 3545288 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39736 7135880 1% /mnt/lustre 1 173 3.53 MB/sec warmup 2 sec latency 81.997 ms 1 274 3.39 MB/sec warmup 3 sec latency 57.549 ms 1 381 3.39 MB/sec warmup 4 sec latency 57.410 ms 1 486 3.39 MB/sec warmup 5 sec latency 64.047 ms 1 666 3.52 MB/sec warmup 6 sec latency 63.977 ms test_26 fail mds1 5 times 1 761 3.05 MB/sec warmup 7 sec latency 54.223 ms Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 1 815 2.68 MB/sec warmup 8 sec latency 69.738 ms 1 869 2.40 MB/sec warmup 9 sec latency 611.018 ms 21:25:47 (1746149147) shut down facet: mds1 facet_host: oleg420-server facet_failover_host: oleg420-server 1 869 2.16 MB/sec warmup 10 sec latency 1611.649 ms 1 869 1.97 MB/sec warmup 11 sec latency 2611.795 ms 1 869 1.80 MB/sec warmup 12 sec latency 3612.133 ms 1 869 1.66 MB/sec warmup 13 sec latency 4612.305 ms 1 869 1.55 MB/sec warmup 14 sec latency 5615.497 ms 1 869 1.44 MB/sec warmup 15 sec latency 6615.707 ms 1 869 1.35 MB/sec warmup 16 sec latency 7615.957 ms 1 869 1.27 MB/sec warmup 17 sec latency 8617.237 ms 1 869 1.20 MB/sec warmup 18 sec latency 9618.896 ms 1 869 1.14 MB/sec warmup 19 sec latency 10619.346 ms Failover mds1 to oleg420-server mount facets: mds1 1 869 0.00 MB/sec execute 1 sec latency 12620.114 ms 1 869 0.00 MB/sec execute 2 sec latency 13620.306 ms 1 869 0.00 MB/sec execute 3 sec latency 14621.353 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 869 0.00 MB/sec execute 4 sec latency 15622.557 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 1 869 0.00 MB/sec execute 5 sec latency 16622.712 ms pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 21:26:04 (1746149164) targets are mounted 21:26:04 (1746149164) facet_failover done 1 869 0.00 MB/sec execute 6 sec latency 17622.905 ms 1 869 0.00 MB/sec execute 7 sec latency 18623.717 ms 1 869 0.00 MB/sec execute 8 sec latency 19624.334 ms 1 869 0.00 MB/sec execute 9 sec latency 20624.480 ms 1 869 0.00 MB/sec execute 10 sec latency 21624.798 ms 1 869 0.00 MB/sec execute 11 sec latency 22624.944 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 940 0.01 MB/sec execute 12 sec latency 22855.454 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1086 0.10 MB/sec execute 13 sec latency 62.070 ms 1 1233 0.10 MB/sec execute 14 sec latency 31.115 ms 1 1421 0.29 MB/sec execute 15 sec latency 43.572 ms 1 1559 0.28 MB/sec execute 16 sec latency 58.854 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4168 1283520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15248 3576884 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38996 3559648 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 54244 7136532 1% /mnt/lustre 1 1674 0.27 MB/sec execute 17 sec latency 37.483 ms 1 1837 0.26 MB/sec execute 18 sec latency 34.086 ms 1 1984 0.25 MB/sec execute 19 sec latency 31.437 ms 1 2116 0.25 MB/sec execute 20 sec latency 35.522 ms 1 2325 0.29 MB/sec execute 21 sec latency 28.374 ms test_26 fail mds2 6 times 1 2434 0.29 MB/sec execute 22 sec latency 46.818 ms Failing mds2 on oleg420-server Stopping /mnt/lustre-mds2 (opts:) on oleg420-server 1 2577 0.32 MB/sec execute 23 sec latency 52.770 ms 1 2582 0.30 MB/sec execute 24 sec latency 974.925 ms 21:26:22 (1746149182) shut down facet: mds2 facet_host: oleg420-server facet_failover_host: oleg420-server 1 2582 0.29 MB/sec execute 25 sec latency 1975.844 ms 1 2582 0.28 MB/sec execute 26 sec latency 2976.062 ms 1 2582 0.27 MB/sec execute 27 sec latency 3976.245 ms 1 2582 0.26 MB/sec execute 28 sec latency 4976.465 ms 1 2582 0.25 MB/sec execute 29 sec latency 5976.960 ms 1 2582 0.24 MB/sec execute 30 sec latency 6977.123 ms 1 2582 0.24 MB/sec execute 31 sec latency 7977.290 ms 1 2582 0.23 MB/sec execute 32 sec latency 8977.454 ms 1 2582 0.22 MB/sec execute 33 sec latency 9977.635 ms 1 2582 0.21 MB/sec execute 34 sec latency 10978.161 ms Failover mds2 to oleg420-server mount facets: mds2 1 2582 0.21 MB/sec execute 35 sec latency 11978.485 ms 1 2582 0.20 MB/sec execute 36 sec latency 12979.476 ms 1 2582 0.20 MB/sec execute 37 sec latency 13979.832 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2582 0.19 MB/sec execute 38 sec latency 14981.088 ms 1 2582 0.19 MB/sec execute 39 sec latency 15981.361 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 1 2582 0.18 MB/sec execute 40 sec latency 16981.562 ms 21:26:38 (1746149198) targets are mounted 21:26:38 (1746149198) facet_failover done 1 2582 0.18 MB/sec execute 41 sec latency 17982.030 ms 1 2582 0.17 MB/sec execute 42 sec latency 18982.743 ms 1 2582 0.17 MB/sec execute 43 sec latency 19983.133 ms 1 2761 0.23 MB/sec execute 44 sec latency 20333.460 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2979 0.26 MB/sec execute 45 sec latency 46.955 ms 1 3235 0.28 MB/sec execute 46 sec latency 54.800 ms 1 3365 0.29 MB/sec execute 47 sec latency 37.592 ms 1 3576 0.35 MB/sec execute 48 sec latency 38.391 ms 1 3707 0.35 MB/sec execute 49 sec latency 30.445 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 3830 0.36 MB/sec execute 50 sec latency 35.751 ms 1 3942 0.36 MB/sec execute 51 sec latency 32.615 ms 1 4043 0.35 MB/sec execute 52 sec latency 33.538 ms 1 4158 0.35 MB/sec execute 53 sec latency 40.409 ms tar: Error is not recoverable: exiting now 1 4299 0.35 MB/sec execute 54 sec latency 32.652 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu May 1 21:26:52 EDT 2025 with return code 0 8853 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 8853 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished