== replay-dual test 26: dbench and tar with mds failover ========================================================== 19:44:30 (1755042270) Starting client oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre Started clients oleg405-client.virtnet: 192.168.204.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 64960 Started dbench loop with 64961 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22024 3584800 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 23560 7190284 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Aug 12 19:44:34 EDT 2025 waiting for dbench pid 65024 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 88 4.06 MB/sec warmup 1 sec latency 56.185 ms 1 174 3.53 MB/sec warmup 2 sec latency 63.723 ms 1 271 3.39 MB/sec warmup 3 sec latency 60.163 ms 1 368 3.26 MB/sec warmup 4 sec latency 57.973 ms test_26 fail mds1 1 times 1 469 3.28 MB/sec warmup 5 sec latency 54.886 ms Failing mds1 on oleg405-server 1 575 3.23 MB/sec warmup 6 sec latency 93.800 ms Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 1 674 3.02 MB/sec warmup 7 sec latency 203.206 ms 19:44:42 (1755042282) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server 1 767 2.67 MB/sec warmup 8 sec latency 56.653 ms 1 874 2.41 MB/sec warmup 9 sec latency 51.610 ms 1 955 2.18 MB/sec warmup 10 sec latency 50.688 ms 1 995 1.99 MB/sec warmup 11 sec latency 483.293 ms 1 995 1.83 MB/sec warmup 12 sec latency 1483.527 ms 1 995 1.68 MB/sec warmup 13 sec latency 2483.734 ms 1 995 1.56 MB/sec warmup 14 sec latency 3483.941 ms 1 995 1.46 MB/sec warmup 15 sec latency 4484.325 ms 1 995 1.37 MB/sec warmup 16 sec latency 5484.512 ms 1 995 1.29 MB/sec warmup 17 sec latency 6484.708 ms 1 995 1.22 MB/sec warmup 18 sec latency 7484.966 ms Failover mds1 to oleg405-server mount facets: mds1 1 995 1.15 MB/sec warmup 19 sec latency 8485.176 ms 1 995 0.00 MB/sec execute 1 sec latency 10487.003 ms 1 995 0.00 MB/sec execute 2 sec latency 11487.806 ms 1 995 0.00 MB/sec execute 3 sec latency 12488.431 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.00 MB/sec execute 4 sec latency 13489.070 ms 1 995 0.00 MB/sec execute 5 sec latency 14489.272 ms 1 995 0.00 MB/sec execute 6 sec latency 15489.562 ms 1 995 0.00 MB/sec execute 7 sec latency 16489.766 ms 1 995 0.00 MB/sec execute 8 sec latency 17490.047 ms 1 995 0.00 MB/sec execute 9 sec latency 18493.989 ms 1 995 0.00 MB/sec execute 10 sec latency 19498.372 ms 1 995 0.00 MB/sec execute 11 sec latency 20498.581 ms 1 995 0.00 MB/sec execute 12 sec latency 21498.838 ms 1 995 0.00 MB/sec execute 13 sec latency 22500.209 ms 1 995 0.00 MB/sec execute 14 sec latency 23500.423 ms 1 995 0.00 MB/sec execute 15 sec latency 24500.631 ms 1 995 0.00 MB/sec execute 16 sec latency 25501.464 ms oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 1 995 0.00 MB/sec execute 17 sec latency 26504.345 ms pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 19:45:11 (1755042311) targets are mounted 19:45:11 (1755042311) facet_failover done 1 995 0.00 MB/sec execute 18 sec latency 27505.244 ms 1 995 0.00 MB/sec execute 19 sec latency 28506.243 ms 1 995 0.00 MB/sec execute 20 sec latency 29508.028 ms 1 995 0.00 MB/sec execute 21 sec latency 30508.182 ms 1 1076 0.05 MB/sec execute 22 sec latency 30514.985 ms oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1154 0.05 MB/sec execute 23 sec latency 57.986 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1223 0.05 MB/sec execute 24 sec latency 64.583 ms 1 1285 0.05 MB/sec execute 25 sec latency 95.513 ms 1 1447 0.16 MB/sec execute 26 sec latency 65.957 ms 1 1537 0.16 MB/sec execute 27 sec latency 50.812 ms 1 1646 0.15 MB/sec execute 28 sec latency 50.699 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2324 1285364 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38448 3568108 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19780 3584848 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 58228 7152956 1% /mnt/lustre 1 1721 0.15 MB/sec execute 29 sec latency 68.646 ms 1 1839 0.15 MB/sec execute 30 sec latency 52.543 ms 1 1902 0.15 MB/sec execute 31 sec latency 430.011 ms 1 1957 0.14 MB/sec execute 32 sec latency 915.049 ms 1 2033 0.14 MB/sec execute 33 sec latency 59.206 ms 1 2152 0.14 MB/sec execute 34 sec latency 51.429 ms test_26 fail mds2 2 times 1 2240 0.14 MB/sec execute 35 sec latency 52.621 ms Failing mds2 on oleg405-server Stopping /mnt/lustre-mds2 (opts:) on oleg405-server 1 2356 0.17 MB/sec execute 36 sec latency 43.038 ms 1 2426 0.17 MB/sec execute 37 sec latency 64.361 ms 19:45:31 (1755042331) shut down facet: mds2 facet_host: oleg405-server facet_failover_host: oleg405-server 1 2559 0.18 MB/sec execute 38 sec latency 55.506 ms 1 2776 0.26 MB/sec execute 39 sec latency 45.511 ms 1 2987 0.29 MB/sec execute 40 sec latency 58.588 ms 1 3246 0.31 MB/sec execute 41 sec latency 51.167 ms 1 3432 0.35 MB/sec execute 42 sec latency 36.616 ms 1 3624 0.39 MB/sec execute 43 sec latency 40.190 ms 1 3761 0.41 MB/sec execute 44 sec latency 35.540 ms 1 3848 0.40 MB/sec execute 45 sec latency 45.015 ms 1 3928 0.39 MB/sec execute 46 sec latency 107.535 ms 1 4010 0.39 MB/sec execute 47 sec latency 51.465 ms Failover mds2 to oleg405-server mount facets: mds2 1 4083 0.38 MB/sec execute 48 sec latency 157.257 ms 1 4083 0.37 MB/sec execute 49 sec latency 1157.479 ms 1 4083 0.36 MB/sec execute 50 sec latency 2157.730 ms 1 4083 0.36 MB/sec execute 51 sec latency 3158.915 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4083 0.35 MB/sec execute 52 sec latency 4160.627 ms 1 4083 0.34 MB/sec execute 53 sec latency 5160.958 ms 1 4083 0.34 MB/sec execute 54 sec latency 6162.337 ms oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 1 4083 0.33 MB/sec execute 55 sec latency 7162.498 ms Started lustre-MDT0001 19:45:49 (1755042349) targets are mounted 19:45:49 (1755042349) facet_failover done 1 4083 0.32 MB/sec execute 56 sec latency 8162.701 ms 1 4083 0.32 MB/sec execute 57 sec latency 9164.340 ms 1 4083 0.31 MB/sec execute 58 sec latency 10165.543 ms 1 4151 0.31 MB/sec execute 59 sec latency 10538.781 ms oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4266 0.31 MB/sec execute 60 sec latency 50.578 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4349 0.30 MB/sec execute 61 sec latency 73.172 ms 1 4446 0.30 MB/sec execute 62 sec latency 96.684 ms 1 4509 0.30 MB/sec execute 63 sec latency 66.848 ms 1 4627 0.31 MB/sec execute 64 sec latency 49.871 ms 1 4724 0.31 MB/sec execute 65 sec latency 61.962 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2364 1285324 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38668 3567452 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 29728 3575556 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 68396 7143008 1% /mnt/lustre 1 4818 0.31 MB/sec execute 66 sec latency 47.481 ms 1 4997 0.35 MB/sec execute 67 sec latency 36.587 ms 1 5029 0.34 MB/sec execute 68 sec latency 674.475 ms 1 5137 0.34 MB/sec execute 69 sec latency 37.550 ms 1 5230 0.33 MB/sec execute 70 sec latency 65.278 ms test_26 fail mds1 3 times 1 5294 0.33 MB/sec execute 71 sec latency 65.353 ms Failing mds1 on oleg405-server Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 1 5364 0.33 MB/sec execute 72 sec latency 97.878 ms 1 5467 0.32 MB/sec execute 73 sec latency 135.524 ms 1 5589 0.32 MB/sec execute 74 sec latency 45.223 ms 1 5724 0.32 MB/sec execute 75 sec latency 40.004 ms 1 5820 0.32 MB/sec execute 76 sec latency 295.972 ms 1 5820 0.31 MB/sec execute 77 sec latency 1298.797 ms 1 5820 0.31 MB/sec execute 78 sec latency 2299.005 ms 19:46:12 (1755042372) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server 1 5820 0.30 MB/sec execute 79 sec latency 3300.796 ms 1 5820 0.30 MB/sec execute 80 sec latency 4302.679 ms 1 5820 0.30 MB/sec execute 81 sec latency 5304.241 ms 1 5820 0.29 MB/sec execute 82 sec latency 6304.454 ms 1 5820 0.29 MB/sec execute 83 sec latency 7304.677 ms 1 5820 0.29 MB/sec execute 84 sec latency 8305.671 ms 1 5820 0.28 MB/sec execute 85 sec latency 9306.095 ms 1 5820 0.28 MB/sec execute 86 sec latency 10306.437 ms 1 5820 0.28 MB/sec execute 87 sec latency 11307.352 ms 1 5820 0.27 MB/sec execute 88 sec latency 12310.141 ms Failover mds1 to oleg405-server mount facets: mds1 1 5820 0.27 MB/sec execute 89 sec latency 13311.123 ms 1 5820 0.27 MB/sec execute 90 sec latency 14312.857 ms 1 5820 0.26 MB/sec execute 91 sec latency 15314.908 ms 1 5820 0.26 MB/sec execute 92 sec latency 16315.687 ms 1 5820 0.26 MB/sec execute 93 sec latency 17316.047 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5820 0.26 MB/sec execute 94 sec latency 18316.254 ms 1 5820 0.25 MB/sec execute 95 sec latency 19316.572 ms 1 5820 0.25 MB/sec execute 96 sec latency 20316.827 ms 1 5820 0.25 MB/sec execute 97 sec latency 21317.161 ms 1 5820 0.24 MB/sec execute 98 sec latency 22317.373 ms 1 5820 0.24 MB/sec execute 99 sec latency 23317.695 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 1 cleanup 106 sec pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 19:46:41 (1755042401) targets are mounted 19:46:41 (1755042401) facet_failover done 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 0 cleanup 112 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 858 101.759 34588.982 Close 660 6.942 674.457 Rename 33 42.715 95.489 Unlink 144 11.623 26.545 Qpathinfo 838 19.739 10538.759 Qfileinfo 140 1.038 10.444 Qfsinfo 118 1.115 11.384 Sfileinfo 66 30.130 96.656 Find 283 5.777 46.593 WriteX 405 5.537 35.298 ReadX 1228 0.640 101.548 LockX 2 7.304 7.332 UnlockX 2 5.078 5.607 Flush 49 26.125 66.832 Throughput 0.242168 MB/sec 1 clients 1 procs max_latency=30514.985 ms stopping dbench on /mnt/lustre at Tue Aug 12 19:46:46 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Aug 12 19:46:50 EDT 2025 waiting for dbench pid 68166 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3048 1284640 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2764 1284924 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 27152 3577588 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21820 3585100 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 48972 7162688 1% /mnt/lustre 1 111 5.24 MB/sec warmup 1 sec latency 43.344 ms 1 207 4.05 MB/sec warmup 2 sec latency 60.312 ms 1 263 3.31 MB/sec warmup 3 sec latency 551.747 ms 1 365 3.26 MB/sec warmup 4 sec latency 50.147 ms 1 462 3.23 MB/sec warmup 5 sec latency 57.878 ms test_26 fail mds2 4 times 1 522 3.01 MB/sec warmup 6 sec latency 107.474 ms Failing mds2 on oleg405-server Stopping /mnt/lustre-mds2 (opts:) on oleg405-server 1 658 3.01 MB/sec warmup 7 sec latency 83.980 ms 1 731 2.66 MB/sec warmup 8 sec latency 66.433 ms 1 798 2.39 MB/sec warmup 9 sec latency 75.960 ms 1 878 2.17 MB/sec warmup 10 sec latency 57.329 ms 1 932 1.98 MB/sec warmup 11 sec latency 77.592 ms 1 988 1.82 MB/sec warmup 12 sec latency 79.279 ms 1 995 1.68 MB/sec warmup 13 sec latency 849.326 ms 1 995 1.56 MB/sec warmup 14 sec latency 1849.555 ms 1 995 1.46 MB/sec warmup 15 sec latency 2854.252 ms 19:47:06 (1755042426) shut down facet: mds2 facet_host: oleg405-server facet_failover_host: oleg405-server 1 995 1.37 MB/sec warmup 16 sec latency 3857.612 ms 1 995 1.29 MB/sec warmup 17 sec latency 4857.830 ms 1 995 1.22 MB/sec warmup 18 sec latency 5858.062 ms 1 995 1.15 MB/sec warmup 19 sec latency 6858.254 ms 1 995 0.00 MB/sec execute 1 sec latency 8859.013 ms 1 995 0.00 MB/sec execute 2 sec latency 9861.128 ms 1 995 0.00 MB/sec execute 3 sec latency 10864.218 ms 1 995 0.00 MB/sec execute 4 sec latency 11864.662 ms 1 995 0.00 MB/sec execute 5 sec latency 12870.328 ms Failover mds2 to oleg405-server mount facets: mds2 1 995 0.00 MB/sec execute 6 sec latency 13877.934 ms 1 995 0.00 MB/sec execute 7 sec latency 14878.265 ms 1 995 0.00 MB/sec execute 8 sec latency 15878.453 ms 1 995 0.00 MB/sec execute 9 sec latency 16880.758 ms 1 995 0.00 MB/sec execute 10 sec latency 17880.917 ms 1 995 0.00 MB/sec execute 11 sec latency 18882.037 ms 1 995 0.00 MB/sec execute 12 sec latency 19883.059 ms 1 995 0.00 MB/sec execute 13 sec latency 20883.288 ms 1 995 0.00 MB/sec execute 14 sec latency 21885.886 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 995 0.00 MB/sec execute 15 sec latency 22888.933 ms 1 995 0.00 MB/sec execute 16 sec latency 23889.433 ms 1 995 0.00 MB/sec execute 17 sec latency 24892.010 ms 1 995 0.00 MB/sec execute 18 sec latency 25892.397 ms oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 1 995 0.00 MB/sec execute 19 sec latency 26892.576 ms pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0001 19:47:30 (1755042450) targets are mounted 19:47:30 (1755042450) facet_failover done 1 995 0.00 MB/sec execute 20 sec latency 27892.810 ms 1 995 0.00 MB/sec execute 21 sec latency 28892.934 ms 1 995 0.00 MB/sec execute 22 sec latency 29896.738 ms 1 1011 0.02 MB/sec execute 23 sec latency 30568.866 ms 1 1086 0.04 MB/sec execute 24 sec latency 92.064 ms oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1156 0.04 MB/sec execute 25 sec latency 66.598 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1225 0.04 MB/sec execute 26 sec latency 90.074 ms 1 1387 0.15 MB/sec execute 27 sec latency 83.726 ms 1 1474 0.15 MB/sec execute 28 sec latency 63.699 ms 1 1580 0.15 MB/sec execute 29 sec latency 52.058 ms 1 1652 0.14 MB/sec execute 30 sec latency 83.281 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3392 1284296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3220 1284468 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 41292 3564972 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 31848 3574480 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 73140 7139452 2% /mnt/lustre 1 1729 0.14 MB/sec execute 31 sec latency 73.121 ms 1 1825 0.14 MB/sec execute 32 sec latency 57.162 ms 1 1909 0.14 MB/sec execute 33 sec latency 80.427 ms 1 1936 0.13 MB/sec execute 34 sec latency 850.274 ms 1 2018 0.13 MB/sec execute 35 sec latency 50.194 ms 1 2136 0.13 MB/sec execute 36 sec latency 64.397 ms test_26 fail mds1 5 times 1 2191 0.13 MB/sec execute 37 sec latency 73.194 ms Failing mds1 on oleg405-server 1 2329 0.16 MB/sec execute 38 sec latency 71.776 ms Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 1 2389 0.16 MB/sec execute 39 sec latency 67.092 ms 1 2467 0.17 MB/sec execute 40 sec latency 71.181 ms 19:47:50 (1755042470) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server 1 2596 0.17 MB/sec execute 41 sec latency 52.751 ms 1 2913 0.27 MB/sec execute 42 sec latency 48.680 ms 1 3176 0.28 MB/sec execute 43 sec latency 42.425 ms 1 3365 0.31 MB/sec execute 44 sec latency 43.657 ms 1 3558 0.37 MB/sec execute 45 sec latency 39.904 ms 1 3667 0.36 MB/sec execute 46 sec latency 43.224 ms 1 3776 0.38 MB/sec execute 47 sec latency 53.101 ms 1 3863 0.37 MB/sec execute 48 sec latency 53.586 ms 1 3952 0.37 MB/sec execute 49 sec latency 41.780 ms 1 4028 0.36 MB/sec execute 50 sec latency 69.855 ms Failover mds1 to oleg405-server mount facets: mds1 1 4083 0.36 MB/sec execute 51 sec latency 100.427 ms 1 4083 0.35 MB/sec execute 52 sec latency 1104.420 ms 1 4083 0.34 MB/sec execute 53 sec latency 2104.625 ms 1 4083 0.34 MB/sec execute 54 sec latency 3105.307 ms 1 4083 0.33 MB/sec execute 55 sec latency 4108.902 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.32 MB/sec execute 56 sec latency 5109.066 ms 1 4083 0.32 MB/sec execute 57 sec latency 6109.350 ms 1 4083 0.31 MB/sec execute 58 sec latency 7109.535 ms 1 4083 0.31 MB/sec execute 59 sec latency 8109.991 ms oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 1 4083 0.30 MB/sec execute 60 sec latency 9112.761 ms Started lustre-MDT0000 19:48:11 (1755042491) targets are mounted 19:48:11 (1755042491) facet_failover done 1 4083 0.30 MB/sec execute 61 sec latency 10113.000 ms 1 4083 0.29 MB/sec execute 62 sec latency 11114.922 ms 1 4083 0.29 MB/sec execute 63 sec latency 12115.400 ms 1 4148 0.28 MB/sec execute 64 sec latency 12351.954 ms oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4236 0.28 MB/sec execute 65 sec latency 66.000 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4297 0.28 MB/sec execute 66 sec latency 76.685 ms 1 4360 0.28 MB/sec execute 67 sec latency 64.487 ms 1 4448 0.28 MB/sec execute 68 sec latency 71.147 ms kill: sending signal to 64990 failed: No such process 1 4521 0.28 MB/sec execute 69 sec latency 66.316 ms 1 4597 0.29 MB/sec execute 70 sec latency 58.118 ms 1 4714 0.28 MB/sec execute 71 sec latency 45.286 ms 1 4799 0.28 MB/sec execute 72 sec latency 58.268 ms 1 4962 0.32 MB/sec execute 73 sec latency 60.692 ms 1 5052 0.31 MB/sec execute 74 sec latency 82.465 ms 1 5143 0.31 MB/sec execute 75 sec latency 46.545 ms 1 5247 0.31 MB/sec execute 76 sec latency 66.973 ms 1 5301 0.30 MB/sec execute 77 sec latency 129.529 ms 1 5390 0.30 MB/sec execute 78 sec latency 53.203 ms 1 5501 0.30 MB/sec execute 79 sec latency 91.301 ms 1 5581 0.30 MB/sec execute 80 sec latency 49.716 ms 1 5712 0.29 MB/sec execute 81 sec latency 79.405 ms 1 5832 0.30 MB/sec execute 82 sec latency 50.419 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Aug 12 19:48:33 EDT 2025 with return code 0 68166 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 68166 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished