== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 19:37:01 (1736987821) Starting client oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre Started clients oleg459-client.virtnet: 192.168.204.159@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg459-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=19479 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1285396 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1808 1285880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1444 3592364 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25996 3556304 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27440 7148668 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server reboot facets: mds1 Failover mds1 to oleg459-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2248 1285440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1764 1285924 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1448 3593488 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26412 3569776 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27860 7163264 1% /mnt/lustre oleg459-client.virtnet: looking for dbench program oleg459-client.virtnet: /usr/bin/dbench oleg459-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg459-client.virtnet oleg459-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg459-client.virtnet' oleg459-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg459-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg459-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg459-client.virtnet at Wed Jan 15 19:37:02 EST 2025 oleg459-client.virtnet: waiting for dbench pid 19503 oleg459-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg459-client.virtnet: oleg459-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg459-client.virtnet: failed to create barrier semaphore oleg459-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg459-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg459-client.virtnet: releasing clients oleg459-client.virtnet: 1 456 15.90 MB/sec warmup 1 sec latency 20.243 ms oleg459-client.virtnet: 1 1072 11.46 MB/sec warmup 2 sec latency 13.304 ms oleg459-client.virtnet: 1 1785 8.77 MB/sec warmup 3 sec latency 12.153 ms oleg459-client.virtnet: 1 2321 6.96 MB/sec warmup 4 sec latency 375.995 ms oleg459-client.virtnet: 1 3359 7.09 MB/sec warmup 5 sec latency 10.647 ms oleg459-client.virtnet: 1 3849 6.63 MB/sec warmup 6 sec latency 395.492 ms oleg459-client.virtnet: 1 3849 5.68 MB/sec warmup 7 sec latency 1395.651 ms oleg459-client.virtnet: 1 3849 4.97 MB/sec warmup 8 sec latency 2395.816 ms oleg459-client.virtnet: 1 3849 4.42 MB/sec warmup 9 sec latency 3395.982 ms oleg459-client.virtnet: 1 3849 3.98 MB/sec warmup 10 sec latency 4396.144 ms oleg459-client.virtnet: 1 3849 3.61 MB/sec warmup 11 sec latency 5396.332 ms oleg459-client.virtnet: 1 3849 3.31 MB/sec warmup 12 sec latency 6396.519 ms oleg459-client.virtnet: 1 3849 3.06 MB/sec warmup 13 sec latency 7396.706 ms oleg459-client.virtnet: 1 3849 2.84 MB/sec warmup 14 sec latency 8396.936 ms oleg459-client.virtnet: 1 3849 2.65 MB/sec warmup 15 sec latency 9397.174 ms oleg459-client.virtnet: 1 3849 2.48 MB/sec warmup 16 sec latency 10397.395 ms oleg459-client.virtnet: 1 3849 2.34 MB/sec warmup 17 sec latency 11397.542 ms oleg459-client.virtnet: 1 3849 2.21 MB/sec warmup 18 sec latency 12397.659 ms oleg459-client.virtnet: 1 3849 2.09 MB/sec warmup 19 sec latency 13397.823 ms oleg459-client.virtnet: 1 3849 1.99 MB/sec warmup 20 sec latency 14397.998 ms oleg459-client.virtnet: 1 3849 1.89 MB/sec warmup 21 sec latency 15398.255 ms oleg459-client.virtnet: 1 3849 1.81 MB/sec warmup 22 sec latency 16398.408 ms oleg459-client.virtnet: 1 3849 1.73 MB/sec warmup 23 sec latency 17398.586 ms oleg459-client.virtnet: 1 3849 1.66 MB/sec warmup 24 sec latency 18398.838 ms oleg459-client.virtnet: 1 3849 1.59 MB/sec warmup 25 sec latency 19399.108 ms oleg459-client.virtnet: 1 3849 1.53 MB/sec warmup 26 sec latency 20399.376 ms oleg459-client.virtnet: 1 3849 1.47 MB/sec warmup 27 sec latency 21399.537 ms oleg459-client.virtnet: 1 4100 1.43 MB/sec warmup 28 sec latency 21479.178 ms oleg459-client.virtnet: 1 4552 1.41 MB/sec warmup 29 sec latency 15.275 ms oleg459-client.virtnet: 1 5040 1.50 MB/sec warmup 30 sec latency 15.568 ms oleg459-client.virtnet: 1 5622 1.48 MB/sec warmup 31 sec latency 10.824 ms oleg459-client.virtnet: 1 6013 1.48 MB/sec warmup 32 sec latency 19.738 ms oleg459-client.virtnet: 1 6570 1.60 MB/sec warmup 33 sec test_70b fail mds2 2 times Failing mds2 on oleg459-server Stopping /mnt/lustre-mds2 (opts:) on oleg459-server reboot facets: mds2 Failover mds2 to oleg459-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12804 3593224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36568 3569436 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49372 7162660 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server reboot facets: mds1 latency 18.466 ms oleg459-client.virtnet: 1 7185 1.70 MB/sec warmup 34 sec latency 16.877 ms oleg459-client.virtnet: 1 7454 1.69 MB/sec warmup 35 sec latency 17.380 ms oleg459-client.virtnet: 1 7763 1.65 MB/sec warmup 36 sec latency 18.341 ms oleg459-client.virtnet: 1 8209 1.65 MB/sec warmup 37 sec latency 23.916 ms oleg459-client.virtnet: 1 8733 1.69 MB/sec warmup 38 sec latency 12.424 ms oleg459-client.virtnet: 1 9086 1.66 MB/sec warmup 39 sec latency 19.159 ms oleg459-client.virtnet: 1 9532 1.66 MB/sec warmup 40 sec latency 10.797 ms oleg459-client.virtnet: 1 10289 1.77 MB/sec warmup 41 sec latency 17.825 ms oleg459-client.virtnet: 1 10773 1.83 MB/sec warmup 42 sec latency 16.740 ms oleg459-client.virtnet: 1 11038 1.82 MB/sec warmup 43 sec latency 18.303 ms oleg459-client.virtnet: 1 11378 1.79 MB/sec warmup 44 sec latency 16.876 ms oleg459-client.virtnet: 1 11863 1.79 MB/sec warmup 45 sec latency 18.256 ms oleg459-client.virtnet: 1 12520 1.82 MB/sec warmup 46 sec latency 12.111 ms oleg459-client.virtnet: 1 13071 1.82 MB/sec warmup 47 sec latency 10.977 ms oleg459-client.virtnet: 1 13885 1.91 MB/sec warmup 48 sec latency 14.963 ms oleg459-client.virtnet: 1 14419 1.98 MB/sec warmup 49 sec latency 16.477 ms oleg459-client.virtnet: 1 14678 1.95 MB/sec warmup 50 sec latency 18.216 ms oleg459-client.virtnet: 1 15075 1.92 MB/sec warmup 51 sec latency 20.401 ms oleg459-client.virtnet: 1 15601 1.97 MB/sec warmup 52 sec latency 15.383 ms oleg459-client.virtnet: 1 15994 1.94 MB/sec warmup 53 sec latency 359.984 ms oleg459-client.virtnet: 1 15994 1.90 MB/sec warmup 54 sec latency 1360.329 ms oleg459-client.virtnet: 1 15994 1.87 MB/sec warmup 55 sec latency 2360.604 ms oleg459-client.virtnet: 1 15994 1.84 MB/sec warmup 56 sec latency 3360.838 ms oleg459-client.virtnet: 1 16461 1.81 MB/sec warmup 57 sec latency 3430.867 ms oleg459-client.virtnet: 1 17363 1.91 MB/sec warmup 58 sec latency 14.053 ms oleg459-client.virtnet: 1 17984 1.97 MB/sec warmup 59 sec latency 11.429 ms oleg459-client.virtnet: 1 18836 1.74 MB/sec execute 1 sec latency 21.901 ms oleg459-client.virtnet: 1 19322 2.47 MB/sec execute 2 sec latency 19.593 ms oleg459-client.virtnet: 1 19591 1.72 MB/sec execute 3 sec latency 334.847 ms oleg459-client.virtnet: 1 19989 1.40 MB/sec execute 4 sec latency 10.656 ms oleg459-client.virtnet: 1 20499 2.16 MB/sec execute 5 sec latency 25.568 ms oleg459-client.virtnet: 1 20499 1.80 MB/sec execute 6 sec latency 1025.781 ms oleg459-client.virtnet: 1 20499 1.54 MB/sec execute 7 sec latency 2025.951 ms oleg459-client.virtnet: 1 20499 1.35 MB/sec execute 8 sec latency 3026.084 ms oleg459-client.virtnet: 1 20499 1.20 MB/sec execute 9 sec latency 4026.253 ms oleg459-client.virtnet: 1 20499 1.08 MB/sec execute 10 sec latency 5026.387 ms oleg459-client.virtnet: 1 20499 0.98 MB/sec execute 11 sec latency 6026.557 ms oleg459-client.virtnet: 1 20499 0.90 MB/sec execute 12 sec latency 7026.709 ms oleg459-client.virtnet: 1 20499 0.83 MB/sec execute 13 sec latency 8026.905 ms oleg459-client.virtnet: 1 20499 0.77 MB/sec execute 14 sec latency 9027.150 ms oleg459-client.virtnet: 1 20499 0.72 MB/sec execute 15 sec latency 10027.380 ms oleg459-client.virtnet: 1 20499 0.67 MB/sec execute 16 sec latency 11027.587 ms oleg459-client.virtnet: 1 20499 0.63 MB/sec execute 17 sec latency 12027.735 ms oleg459-client.virtnet: 1 20499 0.60 MB/sec execute 18 sec latency 13027.890 ms oleg459-client.virtnet: 1 20499 0.57 MB/sec exeFailover mds1 to oleg459-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12808 3593412 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36400 3570028 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49208 7163440 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg459-server Stopping /mnt/lustre-mds2 (opts:) on oleg459-server reboot facets: mds2 Failover mds2 to oleg459-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 cute 19 sec latency 14028.027 ms oleg459-client.virtnet: 1 20499 0.54 MB/sec execute 20 sec latency 15028.180 ms oleg459-client.virtnet: 1 20499 0.51 MB/sec execute 21 sec latency 16028.333 ms oleg459-client.virtnet: 1 20499 0.49 MB/sec execute 22 sec latency 17028.501 ms oleg459-client.virtnet: 1 20499 0.47 MB/sec execute 23 sec latency 18028.622 ms oleg459-client.virtnet: 1 20499 0.45 MB/sec execute 24 sec latency 19028.771 ms oleg459-client.virtnet: 1 20499 0.43 MB/sec execute 25 sec latency 20028.934 ms oleg459-client.virtnet: 1 20499 0.41 MB/sec execute 26 sec latency 21029.083 ms oleg459-client.virtnet: 1 20499 0.40 MB/sec execute 27 sec latency 22029.327 ms oleg459-client.virtnet: 1 20499 0.39 MB/sec execute 28 sec latency 23029.587 ms oleg459-client.virtnet: 1 20499 0.37 MB/sec execute 29 sec latency 24029.826 ms oleg459-client.virtnet: 1 20499 0.36 MB/sec execute 30 sec latency 25029.996 ms oleg459-client.virtnet: 1 20499 0.35 MB/sec execute 31 sec latency 26030.225 ms oleg459-client.virtnet: 1 20499 0.34 MB/sec execute 32 sec latency 27030.423 ms oleg459-client.virtnet: 1 20499 0.33 MB/sec execute 33 sec latency 28030.623 ms oleg459-client.virtnet: 1 20499 0.32 MB/sec execute 34 sec latency 29030.816 ms oleg459-client.virtnet: 1 20499 0.31 MB/sec execute 35 sec latency 30031.036 ms oleg459-client.virtnet: 1 20499 0.30 MB/sec execute 36 sec latency 31031.313 ms oleg459-client.virtnet: 1 20499 0.29 MB/sec execute 37 sec latency 32031.546 ms oleg459-client.virtnet: 1 20499 0.28 MB/sec execute 38 sec latency 33031.715 ms oleg459-client.virtnet: 1 20499 0.28 MB/sec execute 39 sec latency 34031.859 ms oleg459-client.virtnet: 1 20499 0.27 MB/sec execute 40 sec latency 35031.993 ms oleg459-client.virtnet: 1 21108 0.35 MB/sec execute 41 sec latency 35482.534 ms oleg459-client.virtnet: 1 21733 0.45 MB/sec execute 42 sec latency 15.616 ms oleg459-client.virtnet: 1 22303 0.46 MB/sec execute 43 sec latency 13.988 ms oleg459-client.virtnet: 1 22759 0.55 MB/sec execute 44 sec latency 17.797 ms oleg459-client.virtnet: 1 23131 0.54 MB/sec execute 45 sec latency 17.394 ms oleg459-client.virtnet: 1 23673 0.56 MB/sec execute 46 sec latency 11.404 ms oleg459-client.virtnet: 1 24226 0.66 MB/sec execute 47 sec latency 14.238 ms oleg459-client.virtnet: 1 24933 0.76 MB/sec execute 48 sec latency 18.306 ms oleg459-client.virtnet: 1 25381 0.78 MB/sec execute 49 sec latency 15.296 ms oleg459-client.virtnet: 1 25960 0.80 MB/sec execute 50 sec latency 11.203 ms oleg459-client.virtnet: 1 26381 0.85 MB/sec execute 51 sec latency 17.442 ms oleg459-client.virtnet: 1 27028 0.84 MB/sec execute 52 sec latency 11.847 ms oleg459-client.virtnet: 1 27808 0.95 MB/sec execute 53 sec latency 14.501 ms oleg459-client.virtnet: 1 28429 1.03 MB/sec execute 54 sec latency 17.375 ms oleg459-client.virtnet: 1 28726 1.03 MB/sec execute 55 sec latency 17.200 ms oleg459-client.virtnet: 1 29037 1.02 MB/sec execute 56 sec latency 17.330 ms oleg459-client.virtnet: 1 29410 1.02 MB/sec execute 57 sec latency 17.058 ms oleg459-client.virtnet: 1 29999 1.08 MB/sec execute 58 sec latency 13.641 ms oleg459-client.virtnet: 1 30646 1.07 MB/sec execute 59 sec latency 9.448 ms oleg459-client.virtnet: 1 31212 1.14 MB/sec execute 60 sec latency 15.684 ms oleg459-client.virtnet: 1 32032 1.23 MB/sec execute 61 sec latency 16.463 ms oleg459-client.virtnet: 1 32485 1.23 MB/sec execute 62 sec latency 11.770 ms oleg459-client.virtnet: 1 33096 1.24 MB/sec execute 63 sec latency 10oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12928 3593100 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36492 3567468 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49420 7160568 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server reboot facets: mds1 Failover mds1 to oleg459-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 .863 ms oleg459-client.virtnet: 1 33445 1.27 MB/sec execute 64 sec latency 206.970 ms oleg459-client.virtnet: 1 33445 1.25 MB/sec execute 65 sec latency 1207.150 ms oleg459-client.virtnet: 1 33445 1.23 MB/sec execute 66 sec latency 2207.418 ms oleg459-client.virtnet: 1 33445 1.21 MB/sec execute 67 sec latency 3207.669 ms oleg459-client.virtnet: 1 33616 1.20 MB/sec execute 68 sec latency 3860.616 ms oleg459-client.virtnet: 1 34271 1.20 MB/sec execute 69 sec latency 18.177 ms oleg459-client.virtnet: 1 34992 1.27 MB/sec execute 70 sec latency 16.383 ms oleg459-client.virtnet: 1 35630 1.32 MB/sec execute 71 sec latency 15.951 ms oleg459-client.virtnet: 1 35909 1.32 MB/sec execute 72 sec latency 17.594 ms oleg459-client.virtnet: 1 36230 1.31 MB/sec execute 73 sec latency 18.669 ms oleg459-client.virtnet: 1 36498 1.30 MB/sec execute 74 sec latency 405.992 ms oleg459-client.virtnet: 1 37210 1.34 MB/sec execute 75 sec latency 423.336 ms oleg459-client.virtnet: 1 37798 1.34 MB/sec execute 76 sec latency 10.008 ms oleg459-client.virtnet: 1 37927 1.33 MB/sec execute 77 sec latency 707.147 ms oleg459-client.virtnet: 1 37927 1.31 MB/sec execute 78 sec latency 1707.392 ms oleg459-client.virtnet: 1 37927 1.30 MB/sec execute 79 sec latency 2707.619 ms oleg459-client.virtnet: 1 37927 1.28 MB/sec execute 80 sec latency 3707.865 ms oleg459-client.virtnet: 1 37927 1.27 MB/sec execute 81 sec latency 4708.095 ms oleg459-client.virtnet: 1 37927 1.25 MB/sec execute 82 sec latency 5708.338 ms oleg459-client.virtnet: 1 37927 1.24 MB/sec execute 83 sec latency 6708.504 ms oleg459-client.virtnet: 1 37927 1.22 MB/sec execute 84 sec latency 7708.719 ms oleg459-client.virtnet: 1 37927 1.21 MB/sec execute 85 sec latency 8708.898 ms oleg459-client.virtnet: 1 37927 1.19 MB/sec execute 86 sec latency 9709.152 ms oleg459-client.virtnet: 1 37927 1.18 MB/sec execute 87 sec latency 10709.334 ms oleg459-client.virtnet: 1 37927 1.17 MB/sec execute 88 sec latency 11709.559 ms oleg459-client.virtnet: 1 37927 1.15 MB/sec execute 89 sec latency 12709.793 ms oleg459-client.virtnet: 1 37927 1.14 MB/sec execute 90 sec latency 13710.082 ms oleg459-client.virtnet: 1 37927 1.13 MB/sec execute 91 sec latency 14710.332 ms oleg459-client.virtnet: 1 37927 1.11 MB/sec execute 92 sec latency 15710.489 ms oleg459-client.virtnet: 1 37927 1.10 MB/sec execute 93 sec latency 16710.733 ms oleg459-client.virtnet: 1 37927 1.09 MB/sec execute 94 sec latency 17710.920 ms oleg459-client.virtnet: 1 37927 1.08 MB/sec execute 95 sec latency 18711.099 ms oleg459-client.virtnet: 1 37927 1.07 MB/sec execute 96 sec latency 19711.361 ms oleg459-client.virtnet: 1 37927 1.06 MB/sec execute 97 sec latency 20711.624 ms oleg459-client.virtnet: 1 37927 1.05 MB/sec execute 98 sec latency 21711.895 ms oleg459-client.virtnet: 1 37927 1.04 MB/sec execute 99 sec latency 22712.206 ms oleg459-client.virtnet: 1 37927 1.03 MB/sec execute 100 sec latency 23712.465 ms oleg459-client.virtnet: 1 37927 1.02 MB/sec execute 101 sec latency 24712.629 ms oleg459-client.virtnet: 1 37927 1.01 MB/sec execute 102 sec latency 25712.726 ms oleg459-client.virtnet: 1 37927 1.00 MB/sec execute 103 sec latency 26712.897 ms oleg459-client.virtnet: 1 37927 0.99 MB/sec execute 104 sec latency 27713.101 ms oleg459-client.virtnet: 1 37927 0.98 MB/sec execute 105 sec latency 28713.423 ms oleg459-client.virtnet: 1 37927 0.97 MB/sec execute 106 sec latency 29713.639 ms oleg459-client.virtnet: 1 38767 1.03 MB/sec execute 107 sec latency 2975oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2204 1285484 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12984 3593976 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36556 3569576 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49540 7163552 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg459-server Stopping /mnt/lustre-mds2 (opts:) on oleg459-server reboot facets: mds2 Failover mds2 to oleg459-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1285444 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12940 3593088 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36412 3567676 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49352 7160764 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server reboot facets: mds1 Failover mds1 to oleg459-server mount facets: mds1 7.673 ms oleg459-client.virtnet: 1 39434 1.06 MB/sec execute 108 sec latency 11.022 ms oleg459-client.virtnet: 1 39785 1.05 MB/sec execute 109 sec latency 24.761 ms oleg459-client.virtnet: 1 40550 1.09 MB/sec execute 110 sec latency 8.401 ms oleg459-client.virtnet: 1 41318 1.08 MB/sec execute 111 sec latency 8.669 ms oleg459-client.virtnet: 1 42418 1.15 MB/sec execute 112 sec latency 8.600 ms oleg459-client.virtnet: 1 43083 1.18 MB/sec execute 113 sec latency 9.827 ms oleg459-client.virtnet: 1 43691 1.19 MB/sec execute 114 sec latency 12.748 ms oleg459-client.virtnet: 1 44287 1.21 MB/sec execute 115 sec latency 13.414 ms oleg459-client.virtnet: 1 45017 1.21 MB/sec execute 116 sec latency 9.486 ms oleg459-client.virtnet: 1 46163 1.29 MB/sec execute 117 sec latency 8.897 ms oleg459-client.virtnet: 1 46741 1.29 MB/sec execute 118 sec latency 8.911 ms oleg459-client.virtnet: 1 47369 1.30 MB/sec execute 119 sec latency 11.589 ms oleg459-client.virtnet: 1 48114 1.32 MB/sec execute 120 sec latency 9.221 ms oleg459-client.virtnet: 1 49045 1.36 MB/sec execute 121 sec latency 7.729 ms oleg459-client.virtnet: 1 49975 1.41 MB/sec execute 122 sec latency 8.697 ms oleg459-client.virtnet: 1 50509 1.40 MB/sec execute 123 sec latency 11.794 ms oleg459-client.virtnet: 1 50965 1.40 MB/sec execute 124 sec latency 15.126 ms oleg459-client.virtnet: 1 51707 1.42 MB/sec execute 125 sec latency 9.496 ms oleg459-client.virtnet: 1 52619 1.46 MB/sec execute 126 sec latency 8.490 ms oleg459-client.virtnet: 1 53522 1.50 MB/sec execute 127 sec latency 13.057 ms oleg459-client.virtnet: 1 53660 1.49 MB/sec execute 128 sec latency 684.833 ms oleg459-client.virtnet: 1 53660 1.48 MB/sec execute 129 sec latency 1685.035 ms oleg459-client.virtnet: 1 53660 1.47 MB/sec execute 130 sec latency 2685.229 ms oleg459-client.virtnet: 1 53660 1.46 MB/sec execute 131 sec latency 3685.460 ms oleg459-client.virtnet: 1 53660 1.45 MB/sec execute 132 sec latency 4685.626 ms oleg459-client.virtnet: 1 54109 1.44 MB/sec execute 133 sec latency 4984.850 ms oleg459-client.virtnet: 1 54839 1.46 MB/sec execute 134 sec latency 9.139 ms oleg459-client.virtnet: 1 55606 1.47 MB/sec execute 135 sec latency 9.312 ms oleg459-client.virtnet: 1 56587 1.51 MB/sec execute 136 sec latency 13.668 ms oleg459-client.virtnet: 1 57234 1.53 MB/sec execute 137 sec latency 10.301 ms oleg459-client.virtnet: 1 57607 1.53 MB/sec execute 138 sec latency 390.821 ms oleg459-client.virtnet: 1 58069 1.53 MB/sec execute 139 sec latency 17.885 ms oleg459-client.virtnet: 1 58652 1.54 MB/sec execute 140 sec latency 16.045 ms oleg459-client.virtnet: 1 58796 1.53 MB/sec execute 141 sec latency 850.036 ms oleg459-client.virtnet: 1 58796 1.52 MB/sec execute 142 sec latency 1850.210 ms oleg459-client.virtnet: 1 58796 1.51 MB/sec execute 143 sec latency 2850.468 ms oleg459-client.virtnet: 1 58796 1.50 MB/sec execute 144 sec latency 3850.678 ms oleg459-client.virtnet: 1 58796 1.49 MB/sec execute 145 sec latency 4850.847 ms oleg459-client.virtnet: 1 58796 1.48 MB/sec execute 146 sec latency 5850.989 ms oleg459-client.virtnet: 1 58796 1.47 MB/sec execute 147 sec latency 6851.199 ms oleg459-client.virtnet: 1 58796 1.46 MB/sec execute 148 sec latency 7851.366 ms oleg459-client.virtnet: 1 58796 1.45 MB/sec execute 149 sec latency 8851.537 ms oleg459-client.virtnet: 1 58796 1.44 MB/sec execute 150 sec latency 9851.740 ms oleg459-client.virtnet: 1 58796 1.43 MB/sec execute 151 sec latency 10851.883 ms oleg459-client.virtnet: 1 58796 1.42 MB/sec executeStarting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2204 1285484 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12832 3592148 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36600 3570036 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49432 7162184 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg459-server Stopping /mnt/lustre-mds2 (opts:) on oleg459-server reboot facets: mds2 Failover mds2 to oleg459-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1285448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13444 3592548 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36276 3570204 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49720 7162752 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server reboot facets: mds1 152 sec latency 11852.010 ms oleg459-client.virtnet: 1 58796 1.41 MB/sec execute 153 sec latency 12852.227 ms oleg459-client.virtnet: 1 58796 1.40 MB/sec execute 154 sec latency 13852.504 ms oleg459-client.virtnet: 1 58796 1.39 MB/sec execute 155 sec latency 14852.726 ms oleg459-client.virtnet: 1 58796 1.38 MB/sec execute 156 sec latency 15852.962 ms oleg459-client.virtnet: 1 58796 1.37 MB/sec execute 157 sec latency 16853.213 ms oleg459-client.virtnet: 1 58796 1.36 MB/sec execute 158 sec latency 17853.402 ms oleg459-client.virtnet: 1 58796 1.36 MB/sec execute 159 sec latency 18853.625 ms oleg459-client.virtnet: 1 58796 1.35 MB/sec execute 160 sec latency 19853.829 ms oleg459-client.virtnet: 1 59046 1.34 MB/sec execute 161 sec latency 20489.307 ms oleg459-client.virtnet: 1 59973 1.38 MB/sec execute 162 sec latency 13.352 ms oleg459-client.virtnet: 1 60653 1.40 MB/sec execute 163 sec latency 11.615 ms oleg459-client.virtnet: 1 61109 1.40 MB/sec execute 164 sec latency 14.198 ms oleg459-client.virtnet: 1 61781 1.42 MB/sec execute 165 sec latency 14.949 ms oleg459-client.virtnet: 1 62499 1.42 MB/sec execute 166 sec latency 10.012 ms oleg459-client.virtnet: 1 63498 1.45 MB/sec execute 167 sec latency 10.107 ms oleg459-client.virtnet: 1 64211 1.48 MB/sec execute 168 sec latency 16.185 ms oleg459-client.virtnet: 1 64794 1.47 MB/sec execute 169 sec latency 13.285 ms oleg459-client.virtnet: 1 65317 1.49 MB/sec execute 170 sec latency 15.693 ms oleg459-client.virtnet: 1 65721 1.48 MB/sec execute 171 sec latency 16.489 ms oleg459-client.virtnet: 1 66436 1.49 MB/sec execute 172 sec latency 10.690 ms oleg459-client.virtnet: 1 67437 1.54 MB/sec execute 173 sec latency 16.587 ms oleg459-client.virtnet: 1 68083 1.54 MB/sec execute 174 sec latency 9.316 ms oleg459-client.virtnet: 1 68582 1.54 MB/sec execute 175 sec latency 16.783 ms oleg459-client.virtnet: 1 69219 1.55 MB/sec execute 176 sec latency 10.287 ms oleg459-client.virtnet: 1 69911 1.55 MB/sec execute 177 sec latency 9.888 ms oleg459-client.virtnet: 1 71071 1.60 MB/sec execute 178 sec latency 8.814 ms oleg459-client.virtnet: 1 71669 1.60 MB/sec execute 179 sec latency 10.432 ms oleg459-client.virtnet: 1 72396 1.62 MB/sec execute 180 sec latency 9.122 ms oleg459-client.virtnet: 1 73137 1.62 MB/sec execute 181 sec latency 10.575 ms oleg459-client.virtnet: 1 74013 1.64 MB/sec execute 182 sec latency 9.561 ms oleg459-client.virtnet: 1 74484 1.66 MB/sec execute 183 sec latency 605.243 ms oleg459-client.virtnet: 1 74484 1.65 MB/sec execute 184 sec latency 1605.456 ms oleg459-client.virtnet: 1 74484 1.64 MB/sec execute 185 sec latency 2605.691 ms oleg459-client.virtnet: 1 74484 1.63 MB/sec execute 186 sec latency 3605.993 ms oleg459-client.virtnet: 1 74629 1.63 MB/sec execute 187 sec latency 4377.273 ms oleg459-client.virtnet: 1 75205 1.63 MB/sec execute 188 sec latency 9.732 ms oleg459-client.virtnet: 1 75945 1.64 MB/sec execute 189 sec latency 10.844 ms oleg459-client.virtnet: 1 76702 1.64 MB/sec execute 190 sec latency 10.460 ms oleg459-client.virtnet: 1 77774 1.67 MB/sec execute 191 sec latency 9.741 ms oleg459-client.virtnet: 1 78497 1.69 MB/sec execute 192 sec latency 9.902 ms oleg459-client.virtnet: 1 79077 1.69 MB/sec execute 193 sec latency 17.592 ms oleg459-client.virtnet: 1 79781 1.70 MB/sec execute 194 sec latency 11.897 ms oleg459-client.virtnet: 1 80153 1.70 MB/sec execute 195 sec latency 516.163 ms oleg459-client.virtnet: 1 80153 1.69 MB/sec execute 196 sec latency 1516.318 ms oleg459-client.virtnet: 1 Failover mds1 to oleg459-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2212 1285476 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13076 3591044 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36428 3570148 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49504 7161192 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg459-server Stopping /mnt/lustre-mds2 (opts:) on oleg459-server reboot facets: mds2 Failover mds2 to oleg459-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0001 80153 1.68 MB/sec execute 197 sec latency 2516.464 ms oleg459-client.virtnet: 1 80153 1.67 MB/sec execute 198 sec latency 3516.675 ms oleg459-client.virtnet: 1 80153 1.66 MB/sec execute 199 sec latency 4516.792 ms oleg459-client.virtnet: 1 80153 1.65 MB/sec execute 200 sec latency 5516.986 ms oleg459-client.virtnet: 1 80153 1.64 MB/sec execute 201 sec latency 6517.141 ms oleg459-client.virtnet: 1 80153 1.64 MB/sec execute 202 sec latency 7517.297 ms oleg459-client.virtnet: 1 80153 1.63 MB/sec execute 203 sec latency 8517.481 ms oleg459-client.virtnet: 1 80153 1.62 MB/sec execute 204 sec latency 9517.648 ms oleg459-client.virtnet: 1 80153 1.61 MB/sec execute 205 sec latency 10517.861 ms oleg459-client.virtnet: 1 80153 1.60 MB/sec execute 206 sec latency 11518.082 ms oleg459-client.virtnet: 1 80153 1.60 MB/sec execute 207 sec latency 12518.217 ms oleg459-client.virtnet: 1 80153 1.59 MB/sec execute 208 sec latency 13518.454 ms oleg459-client.virtnet: 1 80153 1.58 MB/sec execute 209 sec latency 14518.707 ms oleg459-client.virtnet: 1 80153 1.57 MB/sec execute 210 sec latency 15518.882 ms oleg459-client.virtnet: 1 80153 1.57 MB/sec execute 211 sec latency 16519.176 ms oleg459-client.virtnet: 1 80153 1.56 MB/sec execute 212 sec latency 17519.413 ms oleg459-client.virtnet: 1 80153 1.55 MB/sec execute 213 sec latency 18519.546 ms oleg459-client.virtnet: 1 80153 1.54 MB/sec execute 214 sec latency 19519.791 ms oleg459-client.virtnet: 1 80153 1.54 MB/sec execute 215 sec latency 20519.982 ms oleg459-client.virtnet: 1 80421 1.54 MB/sec execute 216 sec latency 21106.781 ms oleg459-client.virtnet: 1 81280 1.56 MB/sec execute 217 sec latency 12.464 ms oleg459-client.virtnet: 1 81914 1.58 MB/sec execute 218 sec latency 16.089 ms oleg459-client.virtnet: 1 82312 1.57 MB/sec execute 219 sec latency 13.643 ms oleg459-client.virtnet: 1 82938 1.58 MB/sec execute 220 sec latency 14.246 ms oleg459-client.virtnet: 1 83666 1.58 MB/sec execute 221 sec latency 9.341 ms oleg459-client.virtnet: 1 84619 1.61 MB/sec execute 222 sec latency 8.791 ms oleg459-client.virtnet: 1 85505 1.63 MB/sec execute 223 sec latency 10.925 ms oleg459-client.virtnet: 1 86100 1.62 MB/sec execute 224 sec latency 14.703 ms oleg459-client.virtnet: 1 86828 1.64 MB/sec execute 225 sec latency 11.518 ms oleg459-client.virtnet: 1 87573 1.64 MB/sec execute 226 sec latency 9.055 ms oleg459-client.virtnet: 1 88554 1.66 MB/sec execute 227 sec latency 10.023 ms oleg459-client.virtnet: 1 89181 1.68 MB/sec execute 228 sec latency 11.398 ms oleg459-client.virtnet: 1 89852 1.68 MB/sec execute 229 sec latency 11.524 ms oleg459-client.virtnet: 1 90489 1.69 MB/sec execute 230 sec latency 12.282 ms oleg459-client.virtnet: 1 91129 1.69 MB/sec execute 231 sec latency 9.397 ms oleg459-client.virtnet: 1 92000 1.71 MB/sec execute 232 sec latency 12.032 ms oleg459-client.virtnet: 1 92608 1.72 MB/sec execute 233 sec latency 12.138 ms oleg459-client.virtnet: 1 93165 1.72 MB/sec execute 234 sec latency 16.916 ms oleg459-client.virtnet: 1 93837 1.73 MB/sec execute 235 sec latency 9.118 ms oleg459-client.virtnet: 1 94451 1.73 MB/sec execute 236 sec latency 14.679 ms oleg459-client.virtnet: 1 95447 1.75 MB/sec execute 237 sec latency 9.370 ms oleg459-client.virtnet: 1 96067 1.77 MB/sec execute 238 sec latency 17.123 ms oleg459-client.virtnet: 1 96503 1.76 MB/sec execute 239 sec latency 68.192 ms oleg459-client.virtnet: 1 96503 1.75 MB/sec execute 240 sec latency 1068.450 ms oleg459-client.virtnet: 1 96503 1.75 MB/sec execute 241 sec loleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec atency 2068.698 ms oleg459-client.virtnet: 1 96503 1.74 MB/sec execute 242 sec latency 3068.830 ms oleg459-client.virtnet: 1 96911 1.74 MB/sec execute 243 sec latency 3071.190 ms oleg459-client.virtnet: 1 97492 1.74 MB/sec execute 244 sec latency 14.161 ms oleg459-client.virtnet: 1 98219 1.74 MB/sec execute 245 sec latency 14.966 ms oleg459-client.virtnet: 1 99059 1.76 MB/sec execute 246 sec latency 17.762 ms oleg459-client.virtnet: 1 99535 1.78 MB/sec execute 247 sec latency 13.372 ms oleg459-client.virtnet: 1 99981 1.77 MB/sec execute 248 sec latency 17.511 ms oleg459-client.virtnet: 1 100430 1.77 MB/sec execute 249 sec latency 15.976 ms oleg459-client.virtnet: 1 100854 1.78 MB/sec execute 250 sec latency 17.095 ms oleg459-client.virtnet: 1 101205 1.77 MB/sec execute 251 sec latency 17.808 ms oleg459-client.virtnet: 1 101607 1.77 MB/sec execute 252 sec latency 11.522 ms oleg459-client.virtnet: 1 102341 1.79 MB/sec execute 253 sec latency 10.440 ms oleg459-client.virtnet: 1 103188 1.80 MB/sec execute 254 sec latency 10.127 ms oleg459-client.virtnet: 1 103701 1.80 MB/sec execute 255 sec latency 13.606 ms oleg459-client.virtnet: 1 104080 1.80 MB/sec execute 256 sec latency 14.831 ms oleg459-client.virtnet: 1 104758 1.81 MB/sec execute 257 sec latency 12.006 ms oleg459-client.virtnet: 1 105318 1.81 MB/sec execute 258 sec latency 10.843 ms oleg459-client.virtnet: 1 106465 1.84 MB/sec execute 259 sec latency 11.271 ms oleg459-client.virtnet: 1 107040 1.84 MB/sec execute 260 sec latency 9.040 ms oleg459-client.virtnet: 1 107605 1.84 MB/sec execute 261 sec latency 10.963 ms oleg459-client.virtnet: 1 108350 1.84 MB/sec execute 262 sec latency 13.674 ms oleg459-client.virtnet: 1 108894 1.84 MB/sec execute 263 sec latency 15.411 ms oleg459-client.virtnet: 1 109474 1.86 MB/sec execute 264 sec latency 18.210 ms oleg459-client.virtnet: 1 110264 1.87 MB/sec execute 265 sec latency 17.589 ms oleg459-client.virtnet: 1 110677 1.87 MB/sec execute 266 sec latency 16.756 ms oleg459-client.virtnet: 1 110990 1.86 MB/sec execute 267 sec latency 18.827 ms oleg459-client.virtnet: 1 111454 1.87 MB/sec execute 268 sec latency 16.539 ms oleg459-client.virtnet: 1 111911 1.87 MB/sec execute 269 sec latency 17.236 ms oleg459-client.virtnet: 1 112443 1.87 MB/sec execute 270 sec latency 13.292 ms oleg459-client.virtnet: 1 113193 1.88 MB/sec execute 271 sec latency 17.214 ms oleg459-client.virtnet: 1 113824 1.90 MB/sec execute 272 sec latency 16.462 ms oleg459-client.virtnet: 1 114225 1.89 MB/sec execute 273 sec latency 15.872 ms oleg459-client.virtnet: 1 114954 1.90 MB/sec execute 274 sec latency 13.058 ms oleg459-client.virtnet: 1 115327 1.90 MB/sec execute 275 sec latency 18.178 ms oleg459-client.virtnet: 1 116046 1.90 MB/sec execute 276 sec latency 8.334 ms oleg459-client.virtnet: 1 117118 1.93 MB/sec execute 277 sec latency 11.264 ms oleg459-client.virtnet: 1 117530 1.93 MB/sec execute 278 sec latency 12.916 ms oleg459-client.virtnet: 1 117993 1.92 MB/sec execute 279 sec latency 25.773 ms oleg459-client.virtnet: 1 118500 1.93 MB/sec execute 280 sec latency 17.651 ms oleg459-client.virtnet: 1 119122 1.92 MB/sec execute 281 sec latency 11.613 ms oleg459-client.virtnet: 1 120065 1.94 MB/sec execute 282 sec latency 8.059 ms oleg459-client.virtnet: 1 120909 1.96 MB/sec execute 283 sec latency 19.214 ms oleg459-client.virtnet: 1 121434 1.95 MB/sec execute 284 sec latency 13.022 ms oleg459-client.virtnet: 1 122099 1.96 MB/sec execute 285 sec latency 12.953 ms oleg459-client.virtnet: 1 122487 1.96 MB/sec execute 286 sec latency 17.559 ms oleg459-client.virtnet: 1 123010 1.96 MB/sec execute 287 sec latency 11.450 ms oleg459-client.virtnet: 1 123903 1.97 MB/sec execute 288 sec latency 17.342 ms oleg459-client.virtnet: 1 124389 1.98 MB/sec execute 289 sec latency 17.022 ms oleg459-client.virtnet: 1 124665 1.98 MB/sec execute 290 sec latency 17.938 ms oleg459-client.virtnet: 1 125290 1.98 MB/sec execute 291 sec latency 15.328 ms oleg459-client.virtnet: 1 125918 1.98 MB/sec execute 292 sec latency 16.531 ms oleg459-client.virtnet: 1 126677 1.98 MB/sec execute 293 sec latency 8.130 ms oleg459-client.virtnet: 1 127802 2.01 MB/sec execute 294 sec latency 12.196 ms oleg459-client.virtnet: 1 128179 2.01 MB/sec execute 295 sec latency 16.199 ms oleg459-client.virtnet: 1 128853 2.01 MB/sec execute 296 sec latency 14.867 ms oleg459-client.virtnet: 1 129495 2.01 MB/sec execute 297 sec latency 11.633 ms oleg459-client.virtnet: 1 130131 2.01 MB/sec execute 298 sec latency 11.373 ms oleg459-client.virtnet: 1 131277 2.04 MB/sec execute 299 sec latency 9.002 ms oleg459-client.virtnet: 1 cleanup 300 sec oleg459-client.virtnet: 0 cleanup 300 sec oleg459-client.virtnet: oleg459-client.virtnet: Operation Count AvgLat MaxLat oleg459-client.virtnet: ---------------------------------------- oleg459-client.virtnet: NTCreateX 19627 9.088 35482.498 oleg459-client.virtnet: Close 14413 1.041 4.206 oleg459-client.virtnet: Rename 832 7.673 18.264 oleg459-client.virtnet: Unlink 3939 3.060 4984.811 oleg459-client.virtnet: Qpathinfo 17750 1.303 11.556 oleg459-client.virtnet: Qfileinfo 3102 0.119 2.379 oleg459-client.virtnet: Qfsinfo 3254 1.524 4377.251 oleg459-client.virtnet: Sfileinfo 1597 23.920 29757.617 oleg459-client.virtnet: Find 6861 0.355 11.846 oleg459-client.virtnet: WriteX 9690 0.972 7.232 oleg459-client.virtnet: ReadX 30783 0.011 2.582 oleg459-client.virtnet: LockX 64 1.039 1.971 oleg459-client.virtnet: UnlockX 64 1.072 2.085 oleg459-client.virtnet: Flush 1370 6.229 423.324 oleg459-client.virtnet: oleg459-client.virtnet: Throughput 2.04019 MB/sec 1 clients 1 procs max_latency=35482.534 ms oleg459-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg459-client.virtnet at Wed Jan 15 19:43:02 EST 2025 with return code 0 oleg459-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg459-client.virtnet oleg459-client.virtnet: /mnt/lustre/d70b.replay-single/oleg459-client.virtnet /mnt/lustre/d70b.replay-single/oleg459-client.virtnet oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg459-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg459-client.virtnet: removed directory: 'clients/client0' oleg459-client.virtnet: removed directory: 'clients' oleg459-client.virtnet: removed 'client.txt' oleg459-client.virtnet: /mnt/lustre/d70b.replay-single/oleg459-client.virtnet oleg459-client.virtnet: dbench successfully finished