== replay-dual test 26: dbench and tar with mds failover ========================================================== 00:21:59 (1744172519) Starting client oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre Started clients oleg314-client.virtnet: 192.168.203.114@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 4095 Started dbench loop with 4096 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Apr 9 00:22:01 EDT 2025 waiting for dbench pid 4137 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2004 1285684 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26120 3555952 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27656 7161436 1% /mnt/lustre 1 155 6.54 MB/sec warmup 1 sec latency 42.686 ms 1 329 5.97 MB/sec warmup 2 sec latency 30.896 ms 1 485 5.65 MB/sec warmup 3 sec latency 32.403 ms 1 692 5.30 MB/sec warmup 4 sec latency 35.490 ms 1 825 4.30 MB/sec warmup 5 sec latency 32.629 ms test_26 fail mds1 1 times Failing mds1 on oleg314-server 1 976 3.65 MB/sec warmup 6 sec latency 29.731 ms Stopping /mnt/lustre-mds1 (opts:) on oleg314-server 1 1088 3.28 MB/sec warmup 7 sec latency 422.000 ms 1 1088 2.87 MB/sec warmup 8 sec latency 1422.233 ms 1 1088 2.55 MB/sec warmup 9 sec latency 2422.542 ms 1 1088 2.29 MB/sec warmup 10 sec latency 3422.799 ms 1 1088 2.08 MB/sec warmup 11 sec latency 4423.048 ms 1 1088 1.91 MB/sec warmup 12 sec latency 5423.264 ms 00:22:14 (1744172534) shut down facet: mds1 facet_host: oleg314-server facet_failover_host: oleg314-server 1 1088 1.76 MB/sec warmup 13 sec latency 6423.514 ms 1 1088 1.64 MB/sec warmup 14 sec latency 7423.737 ms 1 1088 1.53 MB/sec warmup 15 sec latency 8423.979 ms 1 1088 1.43 MB/sec warmup 16 sec latency 9424.229 ms 1 1088 1.35 MB/sec warmup 17 sec latency 10424.527 ms 1 1088 1.27 MB/sec warmup 18 sec latency 11424.812 ms 1 1088 1.21 MB/sec warmup 19 sec latency 12425.063 ms 1 1088 0.00 MB/sec execute 1 sec latency 14425.441 ms 1 1088 0.00 MB/sec execute 2 sec latency 15425.683 ms 1 1088 0.00 MB/sec execute 3 sec latency 16425.851 ms Failover mds1 to oleg314-server mount facets: mds1 1 1088 0.00 MB/sec execute 4 sec latency 17426.024 ms 1 1088 0.00 MB/sec execute 5 sec latency 18426.184 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1088 0.00 MB/sec execute 6 sec latency 19426.368 ms 1 1088 0.00 MB/sec execute 7 sec latency 20426.546 ms oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all 1 1088 0.00 MB/sec execute 8 sec latency 21426.729 ms pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 00:22:30 (1744172550) targets are mounted 00:22:30 (1744172550) facet_failover done 1 1088 0.00 MB/sec execute 9 sec latency 22426.956 ms 1 1088 0.00 MB/sec execute 10 sec latency 23427.218 ms 1 1088 0.00 MB/sec execute 11 sec latency 24427.530 ms 1 1088 0.00 MB/sec execute 12 sec latency 25427.700 ms oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1236 0.01 MB/sec execute 13 sec latency 25806.376 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1534 0.23 MB/sec execute 14 sec latency 55.829 ms 1 1857 0.23 MB/sec execute 15 sec latency 16.901 ms 1 2181 0.24 MB/sec execute 16 sec latency 16.834 ms 1 2490 0.34 MB/sec execute 17 sec latency 23.919 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2948 1284740 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2620 1285068 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38988 3563188 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 10764 3591392 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49752 7154580 1% /mnt/lustre 1 3017 0.59 MB/sec execute 18 sec latency 27.276 ms 1 3528 0.81 MB/sec execute 19 sec latency 28.211 ms 1 3785 0.84 MB/sec execute 20 sec latency 18.176 ms 1 3942 0.82 MB/sec execute 21 sec latency 27.164 ms test_26 fail mds2 2 times Failing mds2 on oleg314-server Stopping /mnt/lustre-mds2 (opts:) on oleg314-server 1 4079 0.78 MB/sec execute 22 sec latency 26.923 ms 00:22:44 (1744172564) shut down facet: mds2 facet_host: oleg314-server facet_failover_host: oleg314-server 1 4114 0.75 MB/sec execute 23 sec latency 819.654 ms 1 4114 0.71 MB/sec execute 24 sec latency 1819.753 ms 1 4114 0.69 MB/sec execute 25 sec latency 2820.081 ms 1 4114 0.66 MB/sec execute 26 sec latency 3820.406 ms 1 4114 0.64 MB/sec execute 27 sec latency 4820.546 ms 1 4114 0.61 MB/sec execute 28 sec latency 5822.917 ms 1 4114 0.59 MB/sec execute 29 sec latency 6823.169 ms 1 4114 0.57 MB/sec execute 30 sec latency 7823.434 ms 1 4114 0.55 MB/sec execute 31 sec latency 8823.639 ms 1 4114 0.54 MB/sec execute 32 sec latency 9823.762 ms Failover mds2 to oleg314-server mount facets: mds2 1 4114 0.52 MB/sec execute 33 sec latency 10823.885 ms 1 4114 0.50 MB/sec execute 34 sec latency 11824.079 ms 1 4114 0.49 MB/sec execute 35 sec latency 12824.197 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4114 0.48 MB/sec execute 36 sec latency 13824.294 ms oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0001 00:22:58 (1744172578) targets are mounted 00:22:58 (1744172578) facet_failover done 1 4114 0.46 MB/sec execute 37 sec latency 14824.395 ms 1 4114 0.45 MB/sec execute 38 sec latency 15824.490 ms 1 4114 0.44 MB/sec execute 39 sec latency 16824.611 ms 1 4114 0.43 MB/sec execute 40 sec latency 17824.715 ms 1 4214 0.42 MB/sec execute 41 sec latency 18482.771 ms oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4455 0.43 MB/sec execute 42 sec latency 21.523 ms 1 4722 0.45 MB/sec execute 43 sec latency 19.363 ms 1 5055 0.51 MB/sec execute 44 sec latency 19.869 ms 1 5292 0.50 MB/sec execute 45 sec latency 19.534 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4316 1283372 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2460 1285228 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 39132 3564212 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 10832 3592160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49964 7156372 1% /mnt/lustre 1 5561 0.49 MB/sec execute 46 sec latency 26.090 ms 1 5899 0.51 MB/sec execute 47 sec latency 13.964 ms 1 6162 0.52 MB/sec execute 48 sec latency 115.699 ms 1 6786 0.62 MB/sec execute 49 sec latency 24.354 ms test_26 fail mds1 3 times Failing mds1 on oleg314-server 1 7238 0.70 MB/sec execute 50 sec latency 12.713 ms Stopping /mnt/lustre-mds1 (opts:) on oleg314-server 1 7325 0.70 MB/sec execute 51 sec latency 713.180 ms 1 7325 0.69 MB/sec execute 52 sec latency 1713.383 ms 1 7325 0.68 MB/sec execute 53 sec latency 2713.545 ms 1 7325 0.67 MB/sec execute 54 sec latency 3713.681 ms 1 7325 0.65 MB/sec execute 55 sec latency 4713.825 ms 1 7325 0.64 MB/sec execute 56 sec latency 5713.991 ms 00:23:18 (1744172598) shut down facet: mds1 facet_host: oleg314-server facet_failover_host: oleg314-server 1 7325 0.63 MB/sec execute 57 sec latency 6714.156 ms 1 7325 0.62 MB/sec execute 58 sec latency 7714.338 ms 1 7325 0.61 MB/sec execute 59 sec latency 8714.497 ms 1 7325 0.60 MB/sec execute 60 sec latency 9714.676 ms 1 7325 0.59 MB/sec execute 61 sec latency 10714.839 ms 1 7325 0.58 MB/sec execute 62 sec latency 11714.990 ms 1 7325 0.57 MB/sec execute 63 sec latency 12715.129 ms 1 7325 0.56 MB/sec execute 64 sec latency 13715.299 ms 1 7325 0.55 MB/sec execute 65 sec latency 14715.467 ms 1 7325 0.54 MB/sec execute 66 sec latency 15715.642 ms Failover mds1 to oleg314-server mount facets: mds1 1 7325 0.54 MB/sec execute 67 sec latency 16715.793 ms 1 7325 0.53 MB/sec execute 68 sec latency 17715.945 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7325 0.52 MB/sec execute 69 sec latency 18716.075 ms oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all 1 7325 0.51 MB/sec execute 70 sec latency 19716.191 ms pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 00:23:32 (1744172612) targets are mounted 00:23:32 (1744172612) facet_failover done 1 7325 0.51 MB/sec execute 71 sec latency 20716.358 ms 1 7325 0.50 MB/sec execute 72 sec latency 21716.462 ms 1 7325 0.49 MB/sec execute 73 sec latency 22716.599 ms 1 7325 0.49 MB/sec execute 74 sec latency 23716.793 ms 1 7325 0.48 MB/sec execute 75 sec latency 24716.905 ms 1 7325 0.47 MB/sec execute 76 sec latency 25717.015 ms oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 7495 0.47 MB/sec execute 77 sec latency 25887.901 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7691 0.46 MB/sec execute 78 sec latency 21.810 ms 1 7935 0.47 MB/sec execute 79 sec latency 20.309 ms 1 8227 0.48 MB/sec execute 80 sec latency 16.277 ms 1 8570 0.51 MB/sec execute 81 sec latency 15.634 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4832 1282856 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 57616 3542376 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14944 3590152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 72560 7132528 2% /mnt/lustre 1 8819 0.51 MB/sec execute 82 sec latency 20.437 ms 1 8948 0.50 MB/sec execute 83 sec latency 440.955 ms 1 9274 0.50 MB/sec execute 84 sec latency 13.918 ms test_26 fail mds2 4 times 1 9571 0.52 MB/sec execute 85 sec latency 18.132 ms Failing mds2 on oleg314-server Stopping /mnt/lustre-mds2 (opts:) on oleg314-server 1 10068 0.57 MB/sec execute 86 sec latency 19.541 ms 1 10590 0.62 MB/sec execute 87 sec latency 18.079 ms 1 10897 0.63 MB/sec execute 88 sec latency 14.994 ms 1 11129 0.62 MB/sec execute 89 sec latency 13.717 ms 1 11173 0.62 MB/sec execute 90 sec latency 732.637 ms 1 11173 0.61 MB/sec execute 91 sec latency 1732.802 ms 00:23:53 (1744172633) shut down facet: mds2 facet_host: oleg314-server facet_failover_host: oleg314-server 1 11173 0.60 MB/sec execute 92 sec latency 2732.946 ms 1 11173 0.60 MB/sec execute 93 sec latency 3733.200 ms 1 11173 0.59 MB/sec execute 94 sec latency 4733.422 ms 1 11173 0.58 MB/sec execute 95 sec latency 5733.569 ms 1 11173 0.58 MB/sec execute 96 sec latency 6733.720 ms 1 11173 0.57 MB/sec execute 97 sec latency 7733.888 ms 1 11173 0.57 MB/sec execute 98 sec latency 8734.082 ms 1 11173 0.56 MB/sec execute 99 sec latency 9734.237 ms 1 cleanup 100 sec 1 cleanup 101 sec Failover mds2 to oleg314-server mount facets: mds2 1 cleanup 102 sec 1 cleanup 103 sec Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 cleanup 104 sec 1 cleanup 105 sec oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0001 00:24:07 (1744172647) targets are mounted 00:24:07 (1744172647) facet_failover done 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 0 cleanup 111 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1760 24.388 25887.890 Close 1278 1.697 5.011 Rename 73 14.715 28.039 Unlink 333 60.219 18482.704 Qpathinfo 1602 15.641 20667.049 Qfileinfo 280 0.463 2.580 Qfsinfo 283 0.913 7.229 Sfileinfo 130 8.700 15.416 Find 606 45.258 25806.354 WriteX 842 2.083 10.950 ReadX 2782 0.124 36.059 LockX 6 1.458 1.635 UnlockX 6 1.391 1.661 Flush 105 8.956 115.685 Throughput 0.559428 MB/sec 1 clients 1 procs max_latency=25887.901 ms stopping dbench on /mnt/lustre at Wed Apr 9 00:24:12 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Apr 9 00:24:14 EDT 2025 waiting for dbench pid 8580 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 232 8.88 MB/sec warmup 1 sec latency 27.366 ms 1 498 8.67 MB/sec warmup 2 sec latency 17.289 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5448 1282240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3548 1284140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 49128 3523292 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6320 3595472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 55448 7118764 1% /mnt/lustre 1 791 7.15 MB/sec warmup 3 sec latency 20.234 ms 1 915 5.44 MB/sec warmup 4 sec latency 480.695 ms 1 1204 4.61 MB/sec warmup 5 sec latency 16.186 ms 1 1562 4.35 MB/sec warmup 6 sec latency 19.522 ms test_26 fail mds1 5 times Failing mds1 on oleg314-server Stopping /mnt/lustre-mds1 (opts:) on oleg314-server 1 1849 3.77 MB/sec warmup 7 sec latency 25.260 ms 00:24:22 (1744172662) shut down facet: mds1 facet_host: oleg314-server facet_failover_host: oleg314-server 1 1849 3.29 MB/sec warmup 8 sec latency 1025.392 ms 1 1849 2.93 MB/sec warmup 9 sec latency 2025.585 ms 1 1849 2.64 MB/sec warmup 10 sec latency 3025.779 ms 1 1849 2.40 MB/sec warmup 11 sec latency 4025.951 ms 1 1849 2.20 MB/sec warmup 12 sec latency 5026.124 ms 1 1849 2.03 MB/sec warmup 13 sec latency 6026.269 ms 1 1849 1.88 MB/sec warmup 14 sec latency 7026.393 ms 1 1849 1.76 MB/sec warmup 15 sec latency 8026.597 ms 1 1849 1.65 MB/sec warmup 16 sec latency 9026.766 ms 1 1849 1.55 MB/sec warmup 17 sec latency 10026.896 ms Failover mds1 to oleg314-server mount facets: mds1 1 1849 1.46 MB/sec warmup 18 sec latency 11027.018 ms 1 1849 1.39 MB/sec warmup 19 sec latency 12027.177 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1849 0.00 MB/sec execute 1 sec latency 14027.420 ms oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 1 1849 0.00 MB/sec execute 2 sec latency 15027.611 ms Started lustre-MDT0000 00:24:36 (1744172676) targets are mounted 00:24:36 (1744172676) facet_failover done 1 1849 0.00 MB/sec execute 3 sec latency 16027.827 ms 1 1849 0.00 MB/sec execute 4 sec latency 17027.984 ms 1 1849 0.00 MB/sec execute 5 sec latency 18028.144 ms 1 1849 0.00 MB/sec execute 6 sec latency 19028.322 ms 1 1972 0.01 MB/sec execute 7 sec latency 19611.761 ms oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2145 0.04 MB/sec execute 8 sec latency 27.050 ms 1 2388 0.18 MB/sec execute 9 sec latency 45.197 ms 1 2602 0.27 MB/sec execute 10 sec latency 30.987 ms 1 2987 0.65 MB/sec execute 11 sec latency 43.596 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4628 1283060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2676 1285012 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 42388 3548412 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 7796 3588676 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50184 7137088 1% /mnt/lustre 1 3339 0.75 MB/sec execute 12 sec latency 38.881 ms 1 3621 0.94 MB/sec execute 13 sec latency 57.823 ms 1 3806 0.96 MB/sec execute 14 sec latency 22.116 ms 1 3973 0.91 MB/sec execute 15 sec latency 23.598 ms 1 4120 0.86 MB/sec execute 16 sec latency 23.524 ms test_26 fail mds2 6 times Failing mds2 on oleg314-server Stopping /mnt/lustre-mds2 (opts:) on oleg314-server 1 4350 0.83 MB/sec execute 17 sec latency 31.852 ms 1 4554 0.81 MB/sec execute 18 sec latency 33.692 ms 1 4564 0.77 MB/sec execute 19 sec latency 956.167 ms 1 4564 0.73 MB/sec execute 20 sec latency 1956.291 ms 1 4564 0.70 MB/sec execute 21 sec latency 2956.421 ms 1 4564 0.67 MB/sec execute 22 sec latency 3956.568 ms 1 4564 0.64 MB/sec execute 23 sec latency 4956.669 ms 00:24:57 (1744172697) shut down facet: mds2 facet_host: oleg314-server facet_failover_host: oleg314-server 1 4564 0.61 MB/sec execute 24 sec latency 5956.796 ms 1 4564 0.59 MB/sec execute 25 sec latency 6956.964 ms 1 4564 0.56 MB/sec execute 26 sec latency 7957.106 ms 1 4564 0.54 MB/sec execute 27 sec latency 8957.259 ms 1 4564 0.52 MB/sec execute 28 sec latency 9957.449 ms 1 4564 0.51 MB/sec execute 29 sec latency 10957.655 ms 1 4564 0.49 MB/sec execute 30 sec latency 11957.893 ms 1 4564 0.47 MB/sec execute 31 sec latency 12958.121 ms 1 4564 0.46 MB/sec execute 32 sec latency 13958.297 ms 1 4564 0.44 MB/sec execute 33 sec latency 14958.472 ms Failover mds2 to oleg314-server mount facets: mds2 1 4564 0.43 MB/sec execute 34 sec latency 15958.666 ms 1 4564 0.42 MB/sec execute 35 sec latency 16958.913 ms 1 4564 0.41 MB/sec execute 36 sec latency 17959.106 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4564 0.40 MB/sec execute 37 sec latency 18959.235 ms 1 4564 0.39 MB/sec execute 38 sec latency 19959.411 ms oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0001 00:25:13 (1744172713) targets are mounted 00:25:13 (1744172713) facet_failover done 1 4564 0.38 MB/sec execute 39 sec latency 20959.577 ms 1 4564 0.37 MB/sec execute 40 sec latency 21959.767 ms 1 4564 0.36 MB/sec execute 41 sec latency 22960.111 ms 1 4564 0.35 MB/sec execute 42 sec latency 23960.442 ms 1 4564 0.34 MB/sec execute 43 sec latency 24960.658 ms oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4714 0.36 MB/sec execute 44 sec latency 25397.930 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4987 0.42 MB/sec execute 45 sec latency 21.594 ms 1 5213 0.41 MB/sec execute 46 sec latency 39.436 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Apr 9 00:25:20 EDT 2025 with return code 0 8580 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 8580 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished