== replay-dual test 26: dbench and tar with mds failover ========================================================== 15:56:03 (1746129363) Starting client oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre Started clients oleg130-client.virtnet: 192.168.201.130@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 4832 Started dbench loop with 4834 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2092 1285596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3588880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3080 7194364 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu May 1 15:56:08 EDT 2025 waiting for dbench pid 4913 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 44 1.70 MB/sec warmup 1 sec latency 79.421 ms 1 116 2.62 MB/sec warmup 2 sec latency 114.585 ms 1 137 2.00 MB/sec warmup 3 sec latency 698.047 ms 1 158 1.63 MB/sec warmup 4 sec latency 1092.611 ms 1 213 1.65 MB/sec warmup 5 sec latency 104.950 ms 1 275 1.70 MB/sec warmup 6 sec latency 97.665 ms test_26 fail mds1 1 times 1 321 1.66 MB/sec warmup 7 sec latency 114.535 ms Failing mds1 on oleg130-server 1 361 1.60 MB/sec warmup 8 sec latency 125.656 ms Stopping /mnt/lustre-mds1 (opts:) on oleg130-server 1 406 1.59 MB/sec warmup 9 sec latency 142.571 ms 1 467 1.63 MB/sec warmup 10 sec latency 85.922 ms 15:56:19 (1746129379) shut down facet: mds1 facet_host: oleg130-server facet_failover_host: oleg130-server 1 501 1.58 MB/sec warmup 11 sec latency 317.056 ms 1 501 1.45 MB/sec warmup 12 sec latency 1318.857 ms 1 501 1.34 MB/sec warmup 13 sec latency 2320.698 ms 1 501 1.25 MB/sec warmup 14 sec latency 3320.911 ms 1 501 1.16 MB/sec warmup 15 sec latency 4321.464 ms 1 501 1.09 MB/sec warmup 16 sec latency 5321.766 ms 1 501 1.03 MB/sec warmup 17 sec latency 6321.922 ms 1 501 0.97 MB/sec warmup 18 sec latency 7326.571 ms 1 501 0.92 MB/sec warmup 19 sec latency 8328.461 ms 1 501 0.00 MB/sec execute 1 sec latency 10330.579 ms Failover mds1 to oleg130-server mount facets: mds1 1 501 0.00 MB/sec execute 2 sec latency 11331.742 ms 1 501 0.00 MB/sec execute 3 sec latency 12334.181 ms 1 501 0.00 MB/sec execute 4 sec latency 13334.411 ms 1 501 0.00 MB/sec execute 5 sec latency 14337.726 ms 1 501 0.00 MB/sec execute 6 sec latency 15338.191 ms 1 501 0.00 MB/sec execute 7 sec latency 16340.367 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 501 0.00 MB/sec execute 8 sec latency 17340.606 ms 1 501 0.00 MB/sec execute 9 sec latency 18340.819 ms 1 501 0.00 MB/sec execute 10 sec latency 19341.946 ms 1 501 0.00 MB/sec execute 11 sec latency 20344.101 ms oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 1 501 0.00 MB/sec execute 12 sec latency 21344.287 ms Started lustre-MDT0000 15:56:40 (1746129400) targets are mounted 15:56:40 (1746129400) facet_failover done 1 501 0.00 MB/sec execute 13 sec latency 22344.553 ms 1 501 0.00 MB/sec execute 14 sec latency 23346.146 ms 1 502 0.00 MB/sec execute 15 sec latency 24338.556 ms 1 589 0.18 MB/sec execute 16 sec latency 125.047 ms oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 661 0.22 MB/sec execute 17 sec latency 80.062 ms 1 691 0.21 MB/sec execute 18 sec latency 169.084 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 726 0.20 MB/sec execute 19 sec latency 100.769 ms 1 769 0.19 MB/sec execute 20 sec latency 100.285 ms 1 799 0.19 MB/sec execute 21 sec latency 186.475 ms 1 861 0.19 MB/sec execute 22 sec latency 98.196 ms 1 900 0.18 MB/sec execute 23 sec latency 90.758 ms 1 933 0.18 MB/sec execute 24 sec latency 111.073 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2248 1285440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 34532 3569256 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11216 3594112 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 45748 7163368 1% /mnt/lustre 1 981 0.18 MB/sec execute 25 sec latency 110.264 ms 1 1058 0.21 MB/sec execute 26 sec latency 72.915 ms 1 1124 0.20 MB/sec execute 27 sec latency 96.617 ms 1 1155 0.20 MB/sec execute 28 sec latency 216.901 ms 1 1170 0.19 MB/sec execute 29 sec latency 1027.850 ms 1 1224 0.19 MB/sec execute 30 sec latency 84.829 ms 1 1277 0.18 MB/sec execute 31 sec latency 67.641 ms test_26 fail mds2 2 times 1 1442 0.27 MB/sec execute 32 sec latency 62.535 ms Failing mds2 on oleg130-server 1 1487 0.26 MB/sec execute 33 sec latency 92.009 ms Stopping /mnt/lustre-mds2 (opts:) on oleg130-server 1 1538 0.25 MB/sec execute 34 sec latency 68.528 ms 1 1566 0.25 MB/sec execute 35 sec latency 690.247 ms 1 1566 0.24 MB/sec execute 36 sec latency 1690.621 ms 1 1566 0.23 MB/sec execute 37 sec latency 2690.834 ms 1 1566 0.23 MB/sec execute 38 sec latency 3691.092 ms 1 1566 0.22 MB/sec execute 39 sec latency 4693.010 ms 1 1566 0.22 MB/sec execute 40 sec latency 5693.166 ms 1 1566 0.21 MB/sec execute 41 sec latency 6693.567 ms 15:57:09 (1746129429) shut down facet: mds2 facet_host: oleg130-server facet_failover_host: oleg130-server 1 1566 0.21 MB/sec execute 42 sec latency 7693.779 ms 1 1566 0.20 MB/sec execute 43 sec latency 8698.974 ms 1 1566 0.20 MB/sec execute 44 sec latency 9699.176 ms 1 1566 0.19 MB/sec execute 45 sec latency 10701.263 ms 1 1566 0.19 MB/sec execute 46 sec latency 11704.107 ms 1 1566 0.18 MB/sec execute 47 sec latency 12704.359 ms 1 1566 0.18 MB/sec execute 48 sec latency 13705.509 ms 1 1566 0.18 MB/sec execute 49 sec latency 14705.704 ms 1 1566 0.17 MB/sec execute 50 sec latency 15705.998 ms 1 1566 0.17 MB/sec execute 51 sec latency 16706.226 ms Failover mds2 to oleg130-server mount facets: mds2 1 1566 0.17 MB/sec execute 52 sec latency 17710.830 ms 1 1566 0.16 MB/sec execute 53 sec latency 18711.096 ms 1 1566 0.16 MB/sec execute 54 sec latency 19718.705 ms 1 1566 0.16 MB/sec execute 55 sec latency 20720.011 ms 1 1566 0.15 MB/sec execute 56 sec latency 21720.163 ms 1 1566 0.15 MB/sec execute 57 sec latency 22725.291 ms 1 1566 0.15 MB/sec execute 58 sec latency 23726.503 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1566 0.15 MB/sec execute 59 sec latency 24727.522 ms 1 1566 0.14 MB/sec execute 60 sec latency 25728.256 ms 1 1566 0.14 MB/sec execute 61 sec latency 26728.632 ms oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all 1 1566 0.14 MB/sec execute 62 sec latency 27729.397 ms pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 1 1566 0.14 MB/sec execute 63 sec latency 28729.525 ms Started lustre-MDT0001 15:57:31 (1746129451) targets are mounted 15:57:31 (1746129451) facet_failover done 1 1566 0.13 MB/sec execute 64 sec latency 29737.543 ms 1 1566 0.13 MB/sec execute 65 sec latency 30742.616 ms 1 1568 0.13 MB/sec execute 66 sec latency 31680.382 ms 1 1615 0.13 MB/sec execute 67 sec latency 137.463 ms oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1682 0.13 MB/sec execute 68 sec latency 76.920 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1731 0.13 MB/sec execute 69 sec latency 106.998 ms 1 1797 0.13 MB/sec execute 70 sec latency 85.748 ms 1 1857 0.13 MB/sec execute 71 sec latency 68.477 ms 1 1920 0.12 MB/sec execute 72 sec latency 84.263 ms 1 1988 0.12 MB/sec execute 73 sec latency 69.199 ms 1 2060 0.12 MB/sec execute 74 sec latency 60.442 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2692 1284996 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2500 1285188 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37440 3569388 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12572 3594132 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50012 7163520 1% /mnt/lustre 1 2139 0.12 MB/sec execute 75 sec latency 67.690 ms 1 2191 0.12 MB/sec execute 76 sec latency 64.799 ms 1 2300 0.13 MB/sec execute 77 sec latency 69.202 ms 1 2367 0.13 MB/sec execute 78 sec latency 121.672 ms 1 2376 0.13 MB/sec execute 79 sec latency 878.909 ms 1 2412 0.13 MB/sec execute 80 sec latency 918.181 ms 1 2466 0.14 MB/sec execute 81 sec latency 88.666 ms test_26 fail mds1 3 times 1 2549 0.14 MB/sec execute 82 sec latency 82.723 ms 1 2594 0.14 MB/sec execute 83 sec latency 128.123 ms Failing mds1 on oleg130-server 1 2752 0.17 MB/sec execute 84 sec latency 129.600 ms Stopping /mnt/lustre-mds1 (opts:) on oleg130-server 1 2797 0.17 MB/sec execute 85 sec latency 256.149 ms 1 2797 0.17 MB/sec execute 86 sec latency 1256.568 ms 15:57:54 (1746129474) shut down facet: mds1 facet_host: oleg130-server facet_failover_host: oleg130-server 1 2797 0.17 MB/sec execute 87 sec latency 2257.645 ms 1 2797 0.16 MB/sec execute 88 sec latency 3260.736 ms 1 2797 0.16 MB/sec execute 89 sec latency 4262.649 ms 1 2797 0.16 MB/sec execute 90 sec latency 5262.825 ms 1 2797 0.16 MB/sec execute 91 sec latency 6263.158 ms 1 2797 0.16 MB/sec execute 92 sec latency 7265.699 ms 1 2797 0.16 MB/sec execute 93 sec latency 8266.113 ms 1 2797 0.15 MB/sec execute 94 sec latency 9266.558 ms 1 2797 0.15 MB/sec execute 95 sec latency 10266.785 ms 1 2797 0.15 MB/sec execute 96 sec latency 11268.656 ms Failover mds1 to oleg130-server mount facets: mds1 1 2797 0.15 MB/sec execute 97 sec latency 12268.798 ms 1 2797 0.15 MB/sec execute 98 sec latency 13270.700 ms 1 2797 0.15 MB/sec execute 99 sec latency 14273.551 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all 1 cleanup 107 sec pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 1 cleanup 108 sec Started lustre-MDT0000 15:58:16 (1746129496) targets are mounted 15:58:16 (1746129496) facet_failover done 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 cleanup 112 sec 1 cleanup 113 sec mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 0 cleanup 114 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 377 193.594 31680.367 Close 319 10.137 39.251 Rename 18 82.816 186.451 Unlink 41 18.465 42.375 Qpathinfo 362 12.482 72.030 Qfileinfo 91 1.282 7.272 Qfsinfo 52 4.926 34.750 Sfileinfo 50 70.243 1027.458 Find 127 13.216 53.221 WriteX 277 9.003 59.580 ReadX 545 0.658 82.531 LockX 2 12827.202 25647.054 UnlockX 1 8.917 8.917 Flush 35 76.391 918.167 Throughput 0.146293 MB/sec 1 clients 1 procs max_latency=31680.382 ms stopping dbench on /mnt/lustre at Thu May 1 15:58:22 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2740 1284948 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2468 1285220 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22696 3577196 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2168 3604200 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 24864 7181396 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu May 1 15:58:27 EDT 2025 waiting for dbench pid 8319 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 38 1.30 MB/sec warmup 1 sec latency 90.575 ms 1 121 2.74 MB/sec warmup 2 sec latency 107.181 ms 1 160 2.17 MB/sec warmup 3 sec latency 132.038 ms 1 215 2.09 MB/sec warmup 4 sec latency 105.286 ms 1 279 2.08 MB/sec warmup 5 sec latency 93.134 ms 1 341 2.04 MB/sec warmup 6 sec latency 97.829 ms test_26 fail mds2 4 times 1 404 2.05 MB/sec warmup 7 sec latency 83.778 ms 1 461 2.02 MB/sec warmup 8 sec latency 85.600 ms Failing mds2 on oleg130-server Stopping /mnt/lustre-mds2 (opts:) on oleg130-server 1 511 1.97 MB/sec warmup 9 sec latency 91.489 ms 1 606 2.09 MB/sec warmup 10 sec latency 188.440 ms 1 606 1.90 MB/sec warmup 11 sec latency 1188.551 ms 15:58:39 (1746129519) shut down facet: mds2 facet_host: oleg130-server facet_failover_host: oleg130-server 1 606 1.75 MB/sec warmup 12 sec latency 2189.711 ms 1 606 1.61 MB/sec warmup 13 sec latency 3189.863 ms 1 606 1.50 MB/sec warmup 14 sec latency 4192.260 ms 1 606 1.40 MB/sec warmup 15 sec latency 5192.488 ms 1 606 1.31 MB/sec warmup 16 sec latency 6195.355 ms 1 606 1.23 MB/sec warmup 17 sec latency 7195.483 ms 1 606 1.16 MB/sec warmup 18 sec latency 8196.162 ms 1 606 1.10 MB/sec warmup 19 sec latency 9196.525 ms 1 606 0.00 MB/sec execute 1 sec latency 11196.870 ms Failover mds2 to oleg130-server mount facets: mds2 1 606 0.00 MB/sec execute 2 sec latency 12197.185 ms 1 606 0.00 MB/sec execute 3 sec latency 13197.369 ms 1 606 0.00 MB/sec execute 4 sec latency 14199.152 ms 1 606 0.00 MB/sec execute 5 sec latency 15201.762 ms 1 606 0.00 MB/sec execute 6 sec latency 16201.885 ms 1 606 0.00 MB/sec execute 7 sec latency 17202.020 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 606 0.00 MB/sec execute 8 sec latency 18202.375 ms 1 606 0.00 MB/sec execute 9 sec latency 19206.166 ms 1 606 0.00 MB/sec execute 10 sec latency 20206.329 ms 1 606 0.00 MB/sec execute 11 sec latency 21206.529 ms oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 1 606 0.00 MB/sec execute 12 sec latency 22209.005 ms Started lustre-MDT0001 15:59:00 (1746129540) targets are mounted 15:59:00 (1746129540) facet_failover done 1 606 0.00 MB/sec execute 13 sec latency 23209.984 ms 1 606 0.00 MB/sec execute 14 sec latency 24212.324 ms 1 616 0.00 MB/sec execute 15 sec latency 24923.522 ms 1 684 0.01 MB/sec execute 16 sec latency 96.274 ms oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 718 0.02 MB/sec execute 17 sec latency 146.411 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 767 0.02 MB/sec execute 18 sec latency 76.290 ms 1 803 0.03 MB/sec execute 19 sec latency 107.450 ms 1 873 0.03 MB/sec execute 20 sec latency 61.158 ms 1 916 0.04 MB/sec execute 21 sec latency 64.350 ms 1 974 0.04 MB/sec execute 22 sec latency 100.846 ms 1 1036 0.08 MB/sec execute 23 sec latency 90.341 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3392 1284296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 42348 3554252 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11144 3591996 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 53492 7146248 1% /mnt/lustre 1 1082 0.08 MB/sec execute 24 sec latency 85.617 ms 1 1142 0.08 MB/sec execute 25 sec latency 73.069 ms 1 1190 0.08 MB/sec execute 26 sec latency 101.099 ms 1 1231 0.08 MB/sec execute 27 sec latency 109.457 ms 1 1274 0.08 MB/sec execute 28 sec latency 84.803 ms 1 1399 0.17 MB/sec execute 29 sec latency 85.874 ms 1 1455 0.17 MB/sec execute 30 sec latency 92.054 ms 1 1511 0.16 MB/sec execute 31 sec latency 74.160 ms test_26 fail mds1 5 times 1 1562 0.16 MB/sec execute 32 sec latency 77.913 ms 1 1596 0.16 MB/sec execute 33 sec latency 78.764 ms Failing mds1 on oleg130-server 1 1632 0.15 MB/sec execute 34 sec latency 122.033 ms Stopping /mnt/lustre-mds1 (opts:) on oleg130-server 1 1675 0.15 MB/sec execute 35 sec latency 92.683 ms 1 1701 0.15 MB/sec execute 36 sec latency 267.916 ms 1 1701 0.14 MB/sec execute 37 sec latency 1268.169 ms 1 1701 0.14 MB/sec execute 38 sec latency 2269.334 ms 15:59:26 (1746129566) shut down facet: mds1 facet_host: oleg130-server facet_failover_host: oleg130-server 1 1701 0.13 MB/sec execute 39 sec latency 3269.785 ms 1 1701 0.13 MB/sec execute 40 sec latency 4272.769 ms 1 1701 0.13 MB/sec execute 41 sec latency 5272.936 ms 1 1701 0.13 MB/sec execute 42 sec latency 6274.149 ms 1 1701 0.12 MB/sec execute 43 sec latency 7274.335 ms 1 1701 0.12 MB/sec execute 44 sec latency 8277.107 ms 1 1701 0.12 MB/sec execute 45 sec latency 9277.836 ms 1 1701 0.11 MB/sec execute 46 sec latency 10278.047 ms 1 1701 0.11 MB/sec execute 47 sec latency 11278.354 ms 1 1701 0.11 MB/sec execute 48 sec latency 12279.333 ms 1 1701 0.11 MB/sec execute 49 sec latency 13288.259 ms Failover mds1 to oleg130-server mount facets: mds1 1 1701 0.11 MB/sec execute 50 sec latency 14288.484 ms 1 1701 0.10 MB/sec execute 51 sec latency 15289.849 ms 1 1701 0.10 MB/sec execute 52 sec latency 16290.034 ms 1 1701 0.10 MB/sec execute 53 sec latency 17290.400 ms 1 1701 0.10 MB/sec execute 54 sec latency 18291.690 ms 1 1701 0.10 MB/sec execute 55 sec latency 19291.878 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1701 0.09 MB/sec execute 56 sec latency 20292.054 ms 1 1701 0.09 MB/sec execute 57 sec latency 21292.899 ms 1 1701 0.09 MB/sec execute 58 sec latency 22293.639 ms 1 1701 0.09 MB/sec execute 59 sec latency 23293.949 ms 1 1701 0.09 MB/sec execute 60 sec latency 24294.163 ms 1 1701 0.09 MB/sec execute 61 sec latency 25294.363 ms 1 1701 0.08 MB/sec execute 62 sec latency 26294.903 ms 1 1701 0.08 MB/sec execute 63 sec latency 27297.493 ms 1 1701 0.08 MB/sec execute 64 sec latency 28301.434 ms 1 1701 0.08 MB/sec execute 65 sec latency 29301.885 ms 1 1701 0.08 MB/sec execute 66 sec latency 30302.063 ms 1 1701 0.08 MB/sec execute 67 sec latency 31308.362 ms 1 1701 0.08 MB/sec execute 68 sec latency 32308.938 ms oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all 1 1701 0.08 MB/sec execute 69 sec latency 33309.119 ms pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 1 1701 0.08 MB/sec execute 70 sec latency 34309.746 ms Started lustre-MDT0000 15:59:58 (1746129598) targets are mounted 15:59:58 (1746129598) facet_failover done 1 1701 0.07 MB/sec execute 71 sec latency 35310.710 ms 1 1701 0.07 MB/sec execute 72 sec latency 36310.855 ms 1 1701 0.07 MB/sec execute 73 sec latency 37313.451 ms 1 1701 0.07 MB/sec execute 74 sec latency 38314.027 ms 1 1762 0.07 MB/sec execute 75 sec latency 38667.254 ms oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1824 0.07 MB/sec execute 76 sec latency 65.084 ms 1 1867 0.07 MB/sec execute 77 sec latency 112.098 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1928 0.07 MB/sec execute 78 sec latency 91.464 ms 1 1986 0.07 MB/sec execute 79 sec latency 85.962 ms 1 2036 0.07 MB/sec execute 80 sec latency 111.521 ms 1 2095 0.07 MB/sec execute 81 sec latency 63.003 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 2158 0.07 MB/sec execute 82 sec latency 62.938 ms 1 2234 0.07 MB/sec execute 83 sec latency 59.028 ms 1 2347 0.08 MB/sec execute 84 sec latency 95.231 ms 1 2405 0.08 MB/sec execute 85 sec latency 101.324 ms 1 2466 0.09 MB/sec execute 86 sec latency 75.915 ms 1 2568 0.09 MB/sec execute 87 sec latency 58.807 ms 1 2655 0.10 MB/sec execute 88 sec latency 86.985 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu May 1 16:00:16 EDT 2025 with return code 0 8319 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 8319 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished