== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 20:43:04 (1718930584) Starting client oleg213-client.virtnet: -o user_xattr,flock oleg213-server@tcp:/lustre /mnt/lustre Started clients oleg213-client.virtnet: 192.168.202.113@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg213-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=19339 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26020 3556280 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1420 3593408 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27440 7149688 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg213-server Stopping /mnt/lustre-mds1 (opts:) on oleg213-server reboot facets: mds1 Failover mds1 to oleg213-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2196 1285492 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26332 3568760 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5544 3591924 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 31876 7160684 1% /mnt/lustre oleg213-client.virtnet: looking for dbench program oleg213-client.virtnet: /usr/bin/dbench oleg213-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg213-client.virtnet oleg213-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg213-client.virtnet' oleg213-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg213-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg213-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg213-client.virtnet at Thu Jun 20 20:43:05 EDT 2024 oleg213-client.virtnet: waiting for dbench pid 19363 oleg213-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg213-client.virtnet: oleg213-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg213-client.virtnet: failed to create barrier semaphore oleg213-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg213-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg213-client.virtnet: releasing clients oleg213-client.virtnet: 1 482 16.82 MB/sec warmup 1 sec latency 14.367 ms oleg213-client.virtnet: 1 1119 11.48 MB/sec warmup 2 sec latency 13.651 ms oleg213-client.virtnet: 1 1793 8.77 MB/sec warmup 3 sec latency 12.770 ms oleg213-client.virtnet: 1 2553 7.23 MB/sec warmup 4 sec latency 9.687 ms oleg213-client.virtnet: 1 3641 7.73 MB/sec warmup 5 sec latency 10.067 ms oleg213-client.virtnet: 1 4083 6.68 MB/sec warmup 6 sec latency 115.370 ms oleg213-client.virtnet: 1 4083 5.73 MB/sec warmup 7 sec latency 1115.565 ms oleg213-client.virtnet: 1 4083 5.01 MB/sec warmup 8 sec latency 2115.743 ms oleg213-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 3115.936 ms oleg213-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 4116.173 ms oleg213-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 5116.382 ms oleg213-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 6116.532 ms oleg213-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 7116.728 ms oleg213-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 8116.939 ms oleg213-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 9117.116 ms oleg213-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 10117.276 ms oleg213-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 11117.429 ms oleg213-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 12117.572 ms oleg213-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 13117.825 ms oleg213-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 14118.060 ms oleg213-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 15118.286 ms oleg213-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 16118.479 ms oleg213-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 17118.736 ms oleg213-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 18118.920 ms oleg213-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 19119.124 ms oleg213-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 20119.328 ms oleg213-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 21119.533 ms oleg213-client.virtnet: 1 4083 1.43 MB/sec warmup 28 sec latency 22119.716 ms oleg213-client.virtnet: 1 4480 1.41 MB/sec warmup 29 sec latency 22469.030 ms oleg213-client.virtnet: 1 5174 1.51 MB/sec warmup 30 sec latency 9.157 ms oleg213-client.virtnet: 1 5943 1.52 MB/sec warmup 31 sec latency 9.831 ms oleg213-client.virtnet: 1 7060 1.80 MB/sec warmup 32 sec latency 10.860 ms oleg213-client.virtnet: 1 7607 1.79 MB/sec warmup 33 sec test_70b fail mds2 2 times Failing mds2 on oleg213-server Stopping /mnt/lustre-mds2 (opts:) on oleg213-server reboot facets: mds2 Failover mds2 to oleg213-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1796 1285892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37684 3568824 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11756 3593668 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49440 7162492 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg213-server Stopping /mnt/lustre-mds1 (opts:) on oleg213-server reboot facets: mds1 Failover mds1 to oleg213-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 latency 10.349 ms oleg213-client.virtnet: 1 8091 1.77 MB/sec warmup 34 sec latency 229.002 ms oleg213-client.virtnet: 1 8742 1.84 MB/sec warmup 35 sec latency 377.567 ms oleg213-client.virtnet: 1 9520 1.84 MB/sec warmup 36 sec latency 8.572 ms oleg213-client.virtnet: 1 9699 1.82 MB/sec warmup 37 sec latency 743.296 ms oleg213-client.virtnet: 1 9699 1.77 MB/sec warmup 38 sec latency 1743.500 ms oleg213-client.virtnet: 1 9699 1.72 MB/sec warmup 39 sec latency 2743.708 ms oleg213-client.virtnet: 1 9699 1.68 MB/sec warmup 40 sec latency 3743.940 ms oleg213-client.virtnet: 1 9699 1.64 MB/sec warmup 41 sec latency 4744.140 ms oleg213-client.virtnet: 1 9699 1.60 MB/sec warmup 42 sec latency 5744.356 ms oleg213-client.virtnet: 1 9699 1.56 MB/sec warmup 43 sec latency 6744.572 ms oleg213-client.virtnet: 1 9699 1.53 MB/sec warmup 44 sec latency 7744.753 ms oleg213-client.virtnet: 1 9699 1.49 MB/sec warmup 45 sec latency 8744.951 ms oleg213-client.virtnet: 1 9699 1.46 MB/sec warmup 46 sec latency 9745.160 ms oleg213-client.virtnet: 1 9699 1.43 MB/sec warmup 47 sec latency 10745.349 ms oleg213-client.virtnet: 1 9699 1.40 MB/sec warmup 48 sec latency 11745.473 ms oleg213-client.virtnet: 1 9699 1.37 MB/sec warmup 49 sec latency 12745.610 ms oleg213-client.virtnet: 1 9699 1.34 MB/sec warmup 50 sec latency 13745.809 ms oleg213-client.virtnet: 1 9699 1.32 MB/sec warmup 51 sec latency 14745.933 ms oleg213-client.virtnet: 1 9699 1.29 MB/sec warmup 52 sec latency 15746.140 ms oleg213-client.virtnet: 1 9699 1.27 MB/sec warmup 53 sec latency 16746.327 ms oleg213-client.virtnet: 1 9699 1.25 MB/sec warmup 54 sec latency 17746.498 ms oleg213-client.virtnet: 1 10001 1.29 MB/sec warmup 55 sec latency 18546.983 ms oleg213-client.virtnet: 1 10898 1.39 MB/sec warmup 56 sec latency 10.597 ms oleg213-client.virtnet: 1 11451 1.38 MB/sec warmup 57 sec latency 11.831 ms oleg213-client.virtnet: 1 12215 1.44 MB/sec warmup 58 sec latency 8.533 ms oleg213-client.virtnet: 1 13002 1.44 MB/sec warmup 59 sec latency 9.125 ms oleg213-client.virtnet: 1 14654 4.58 MB/sec execute 1 sec latency 9.178 ms oleg213-client.virtnet: 1 15322 3.30 MB/sec execute 2 sec latency 12.430 ms oleg213-client.virtnet: 1 16044 3.33 MB/sec execute 3 sec latency 12.324 ms oleg213-client.virtnet: 1 16948 3.90 MB/sec execute 4 sec latency 9.055 ms oleg213-client.virtnet: 1 17963 4.68 MB/sec execute 5 sec latency 11.054 ms oleg213-client.virtnet: 1 18536 4.02 MB/sec execute 6 sec latency 9.419 ms oleg213-client.virtnet: 1 19289 4.12 MB/sec execute 7 sec latency 8.731 ms oleg213-client.virtnet: 1 20059 3.75 MB/sec execute 8 sec latency 9.149 ms oleg213-client.virtnet: 1 21122 4.24 MB/sec execute 9 sec latency 9.509 ms oleg213-client.virtnet: 1 21768 4.29 MB/sec execute 10 sec latency 8.910 ms oleg213-client.virtnet: 1 22410 4.08 MB/sec execute 11 sec latency 9.736 ms oleg213-client.virtnet: 1 23123 4.02 MB/sec execute 12 sec latency 9.523 ms oleg213-client.virtnet: 1 23890 3.91 MB/sec execute 13 sec latency 8.975 ms oleg213-client.virtnet: 1 25004 4.33 MB/sec execute 14 sec latency 10.524 ms oleg213-client.virtnet: 1 25551 4.15 MB/sec execute 15 sec latency 9.633 ms oleg213-client.virtnet: 1 26097 4.01 MB/sec execute 16 sec latency 14.288 ms oleg213-client.virtnet: 1 26678 3.96 MB/sec execute 17 sec latency 11.998 ms oleg213-client.virtnet: 1 27317 3.84 MB/sec execute 18 sec latency 11.295 ms oleg213-client.virtnet: 1 28244 4.03 MB/sec execute 1Started lustre-MDT0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37572 3568032 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11812 3594064 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49384 7162096 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg213-server Stopping /mnt/lustre-mds2 (opts:) on oleg213-server reboot facets: mds2 Failover mds2 to oleg213-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37288 3569068 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12240 3593808 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49528 7162876 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg213-server Stopping /mnt/lustre-mds1 (opts:) on oleg213-server reboot facets: mds1 9 sec latency 10.995 ms oleg213-client.virtnet: 1 28756 4.05 MB/sec execute 20 sec latency 17.182 ms oleg213-client.virtnet: 1 29041 3.86 MB/sec execute 21 sec latency 18.490 ms oleg213-client.virtnet: 1 29344 3.72 MB/sec execute 22 sec latency 23.020 ms oleg213-client.virtnet: 1 29823 3.75 MB/sec execute 23 sec latency 16.224 ms oleg213-client.virtnet: 1 30251 3.60 MB/sec execute 24 sec latency 16.099 ms oleg213-client.virtnet: 1 30767 3.52 MB/sec execute 25 sec latency 12.365 ms oleg213-client.virtnet: 1 31190 3.54 MB/sec execute 26 sec latency 16.654 ms oleg213-client.virtnet: 1 32058 3.66 MB/sec execute 27 sec latency 15.891 ms oleg213-client.virtnet: 1 32558 3.58 MB/sec execute 28 sec latency 10.160 ms oleg213-client.virtnet: 1 32888 3.48 MB/sec execute 29 sec latency 434.983 ms oleg213-client.virtnet: 1 33444 3.51 MB/sec execute 30 sec latency 496.290 ms oleg213-client.virtnet: 1 34030 3.42 MB/sec execute 31 sec latency 12.982 ms oleg213-client.virtnet: 1 34304 3.35 MB/sec execute 32 sec latency 674.566 ms oleg213-client.virtnet: 1 34304 3.25 MB/sec execute 33 sec latency 1674.776 ms oleg213-client.virtnet: 1 34304 3.15 MB/sec execute 34 sec latency 2675.007 ms oleg213-client.virtnet: 1 34304 3.06 MB/sec execute 35 sec latency 3675.338 ms oleg213-client.virtnet: 1 34304 2.98 MB/sec execute 36 sec latency 4675.607 ms oleg213-client.virtnet: 1 34304 2.89 MB/sec execute 37 sec latency 5675.796 ms oleg213-client.virtnet: 1 34304 2.82 MB/sec execute 38 sec latency 6676.074 ms oleg213-client.virtnet: 1 34304 2.75 MB/sec execute 39 sec latency 7676.360 ms oleg213-client.virtnet: 1 34304 2.68 MB/sec execute 40 sec latency 8676.693 ms oleg213-client.virtnet: 1 34304 2.61 MB/sec execute 41 sec latency 9676.978 ms oleg213-client.virtnet: 1 34304 2.55 MB/sec execute 42 sec latency 10677.202 ms oleg213-client.virtnet: 1 34304 2.49 MB/sec execute 43 sec latency 11677.459 ms oleg213-client.virtnet: 1 34304 2.43 MB/sec execute 44 sec latency 12677.696 ms oleg213-client.virtnet: 1 34304 2.38 MB/sec execute 45 sec latency 13677.889 ms oleg213-client.virtnet: 1 34304 2.33 MB/sec execute 46 sec latency 14678.152 ms oleg213-client.virtnet: 1 34304 2.28 MB/sec execute 47 sec latency 15678.328 ms oleg213-client.virtnet: 1 34304 2.23 MB/sec execute 48 sec latency 16678.519 ms oleg213-client.virtnet: 1 34304 2.19 MB/sec execute 49 sec latency 17678.754 ms oleg213-client.virtnet: 1 34304 2.14 MB/sec execute 50 sec latency 18678.944 ms oleg213-client.virtnet: 1 34470 2.12 MB/sec execute 51 sec latency 19338.573 ms oleg213-client.virtnet: 1 35592 2.27 MB/sec execute 52 sec latency 9.224 ms oleg213-client.virtnet: 1 36120 2.25 MB/sec execute 53 sec latency 10.780 ms oleg213-client.virtnet: 1 36754 2.25 MB/sec execute 54 sec latency 15.443 ms oleg213-client.virtnet: 1 37494 2.27 MB/sec execute 55 sec latency 10.074 ms oleg213-client.virtnet: 1 38286 2.33 MB/sec execute 56 sec latency 8.619 ms oleg213-client.virtnet: 1 39263 2.42 MB/sec execute 57 sec latency 12.236 ms oleg213-client.virtnet: 1 39645 2.39 MB/sec execute 58 sec latency 12.133 ms oleg213-client.virtnet: 1 40212 2.38 MB/sec execute 59 sec latency 14.224 ms oleg213-client.virtnet: 1 40878 2.40 MB/sec execute 60 sec latency 10.038 ms oleg213-client.virtnet: 1 41625 2.40 MB/sec execute 61 sec latency 8.133 ms oleg213-client.virtnet: 1 42717 2.52 MB/sec execute 62 sec latency 9.019 ms oleg213-client.virtnet: 1 43276 2.50 MB/sec execute 63 sec latency 10.013 ms oleg213-client.virtnFailover mds1 to oleg213-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1832 1285856 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569008 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11928 3593488 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49352 7162496 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg213-server Stopping /mnt/lustre-mds2 (opts:) on oleg213-server reboot facets: mds2 Failover mds2 to oleg213-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec et: 1 43982 2.54 MB/sec execute 64 sec latency 9.288 ms oleg213-client.virtnet: 1 44627 2.51 MB/sec execute 65 sec latency 9.836 ms oleg213-client.virtnet: 1 45572 2.58 MB/sec execute 66 sec latency 8.751 ms oleg213-client.virtnet: 1 46444 2.63 MB/sec execute 67 sec latency 9.812 ms oleg213-client.virtnet: 1 46926 2.60 MB/sec execute 68 sec latency 14.162 ms oleg213-client.virtnet: 1 47281 2.59 MB/sec execute 69 sec latency 17.610 ms oleg213-client.virtnet: 1 47769 2.60 MB/sec execute 70 sec latency 15.104 ms oleg213-client.virtnet: 1 48175 2.57 MB/sec execute 71 sec latency 17.690 ms oleg213-client.virtnet: 1 48708 2.56 MB/sec execute 72 sec latency 14.796 ms oleg213-client.virtnet: 1 49484 2.62 MB/sec execute 73 sec latency 17.186 ms oleg213-client.virtnet: 1 50011 2.64 MB/sec execute 74 sec latency 13.045 ms oleg213-client.virtnet: 1 50515 2.62 MB/sec execute 75 sec latency 14.758 ms oleg213-client.virtnet: 1 50940 2.60 MB/sec execute 76 sec latency 17.698 ms oleg213-client.virtnet: 1 51377 2.61 MB/sec execute 77 sec latency 17.485 ms oleg213-client.virtnet: 1 52069 2.60 MB/sec execute 78 sec latency 10.721 ms oleg213-client.virtnet: 1 53048 2.66 MB/sec execute 79 sec latency 12.742 ms oleg213-client.virtnet: 1 53626 2.69 MB/sec execute 80 sec latency 16.033 ms oleg213-client.virtnet: 1 54063 2.66 MB/sec execute 81 sec latency 20.172 ms oleg213-client.virtnet: 1 54671 2.68 MB/sec execute 82 sec latency 11.945 ms oleg213-client.virtnet: 1 55362 2.66 MB/sec execute 83 sec latency 9.700 ms oleg213-client.virtnet: 1 55644 2.64 MB/sec execute 84 sec latency 504.457 ms oleg213-client.virtnet: 1 56254 2.68 MB/sec execute 85 sec latency 17.288 ms oleg213-client.virtnet: 1 56908 2.71 MB/sec execute 86 sec latency 17.501 ms oleg213-client.virtnet: 1 57025 2.69 MB/sec execute 87 sec latency 662.280 ms oleg213-client.virtnet: 1 57025 2.66 MB/sec execute 88 sec latency 1662.623 ms oleg213-client.virtnet: 1 57025 2.63 MB/sec execute 89 sec latency 2662.911 ms oleg213-client.virtnet: 1 57025 2.60 MB/sec execute 90 sec latency 3663.181 ms oleg213-client.virtnet: 1 57025 2.57 MB/sec execute 91 sec latency 4663.410 ms oleg213-client.virtnet: 1 57025 2.54 MB/sec execute 92 sec latency 5663.720 ms oleg213-client.virtnet: 1 57025 2.51 MB/sec execute 93 sec latency 6664.011 ms oleg213-client.virtnet: 1 57025 2.49 MB/sec execute 94 sec latency 7664.281 ms oleg213-client.virtnet: 1 57025 2.46 MB/sec execute 95 sec latency 8664.617 ms oleg213-client.virtnet: 1 57025 2.43 MB/sec execute 96 sec latency 9664.802 ms oleg213-client.virtnet: 1 57025 2.41 MB/sec execute 97 sec latency 10665.048 ms oleg213-client.virtnet: 1 57025 2.39 MB/sec execute 98 sec latency 11665.236 ms oleg213-client.virtnet: 1 57025 2.36 MB/sec execute 99 sec latency 12665.363 ms oleg213-client.virtnet: 1 57025 2.34 MB/sec execute 100 sec latency 13665.539 ms oleg213-client.virtnet: 1 57025 2.31 MB/sec execute 101 sec latency 14665.712 ms oleg213-client.virtnet: 1 57025 2.29 MB/sec execute 102 sec latency 15665.918 ms oleg213-client.virtnet: 1 57025 2.27 MB/sec execute 103 sec latency 16666.166 ms oleg213-client.virtnet: 1 57025 2.25 MB/sec execute 104 sec latency 17666.521 ms oleg213-client.virtnet: 1 57025 2.23 MB/sec execute 105 sec latency 18666.763 ms oleg213-client.virtnet: 1 57292 2.21 MB/sec execute 106 sec latency 19034.108 ms oleg213-client.virtnet: 1 57850 2.21 MB/sec execute 107 sec latency 9.871 ms oleg213-client.virtnet: 1 58405 2.22 MB/sec execute 108 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37288 3569276 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11984 3594020 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49272 7163296 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg213-server Stopping /mnt/lustre-mds1 (opts:) on oleg213-server reboot facets: mds1 Failover mds1 to oleg213-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37412 3569048 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12000 3593884 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49412 7162932 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg213-server Stopping /mnt/lustre-mds2 (opts:) on oleg213-server reboot facets: mds2 sec latency 16.598 ms oleg213-client.virtnet: 1 58984 2.20 MB/sec execute 109 sec latency 10.698 ms oleg213-client.virtnet: 1 59375 2.20 MB/sec execute 110 sec latency 15.209 ms oleg213-client.virtnet: 1 60385 2.27 MB/sec execute 111 sec latency 15.164 ms oleg213-client.virtnet: 1 60930 2.26 MB/sec execute 112 sec latency 10.647 ms oleg213-client.virtnet: 1 61512 2.26 MB/sec execute 113 sec latency 12.585 ms oleg213-client.virtnet: 1 62219 2.27 MB/sec execute 114 sec latency 9.157 ms oleg213-client.virtnet: 1 63116 2.30 MB/sec execute 115 sec latency 9.885 ms oleg213-client.virtnet: 1 64119 2.34 MB/sec execute 116 sec latency 9.998 ms oleg213-client.virtnet: 1 64672 2.33 MB/sec execute 117 sec latency 10.960 ms oleg213-client.virtnet: 1 65391 2.35 MB/sec execute 118 sec latency 9.295 ms oleg213-client.virtnet: 1 66167 2.35 MB/sec execute 119 sec latency 9.362 ms oleg213-client.virtnet: 1 67227 2.39 MB/sec execute 120 sec latency 8.245 ms oleg213-client.virtnet: 1 67832 2.41 MB/sec execute 121 sec latency 10.380 ms oleg213-client.virtnet: 1 68282 2.39 MB/sec execute 122 sec latency 17.683 ms oleg213-client.virtnet: 1 68919 2.41 MB/sec execute 123 sec latency 11.814 ms oleg213-client.virtnet: 1 69357 2.39 MB/sec execute 124 sec latency 16.249 ms oleg213-client.virtnet: 1 69982 2.39 MB/sec execute 125 sec latency 11.179 ms oleg213-client.virtnet: 1 71027 2.45 MB/sec execute 126 sec latency 13.538 ms oleg213-client.virtnet: 1 71388 2.45 MB/sec execute 127 sec latency 18.987 ms oleg213-client.virtnet: 1 71846 2.43 MB/sec execute 128 sec latency 13.222 ms oleg213-client.virtnet: 1 72423 2.45 MB/sec execute 129 sec latency 14.813 ms oleg213-client.virtnet: 1 72841 2.43 MB/sec execute 130 sec latency 17.912 ms oleg213-client.virtnet: 1 73365 2.42 MB/sec execute 131 sec latency 14.219 ms oleg213-client.virtnet: 1 73890 2.44 MB/sec execute 132 sec latency 15.278 ms oleg213-client.virtnet: 1 74659 2.47 MB/sec execute 133 sec latency 15.403 ms oleg213-client.virtnet: 1 75205 2.46 MB/sec execute 134 sec latency 9.322 ms oleg213-client.virtnet: 1 75543 2.45 MB/sec execute 135 sec latency 16.099 ms oleg213-client.virtnet: 1 76152 2.46 MB/sec execute 136 sec latency 11.091 ms oleg213-client.virtnet: 1 76676 2.45 MB/sec execute 137 sec latency 13.360 ms oleg213-client.virtnet: 1 77151 2.45 MB/sec execute 138 sec latency 15.294 ms oleg213-client.virtnet: 1 78173 2.50 MB/sec execute 139 sec latency 16.213 ms oleg213-client.virtnet: 1 78598 2.49 MB/sec execute 140 sec latency 17.267 ms oleg213-client.virtnet: 1 79044 2.48 MB/sec execute 141 sec latency 17.578 ms oleg213-client.virtnet: 1 79701 2.49 MB/sec execute 142 sec latency 10.528 ms oleg213-client.virtnet: 1 80107 2.48 MB/sec execute 143 sec latency 17.220 ms oleg213-client.virtnet: 1 80433 2.47 MB/sec execute 144 sec latency 356.020 ms oleg213-client.virtnet: 1 80980 2.49 MB/sec execute 145 sec latency 393.944 ms oleg213-client.virtnet: 1 81773 2.51 MB/sec execute 146 sec latency 12.162 ms oleg213-client.virtnet: 1 81959 2.50 MB/sec execute 147 sec latency 558.766 ms oleg213-client.virtnet: 1 81959 2.48 MB/sec execute 148 sec latency 1559.028 ms oleg213-client.virtnet: 1 81959 2.47 MB/sec execute 149 sec latency 2559.233 ms oleg213-client.virtnet: 1 81959 2.45 MB/sec execute 150 sec latency 3559.467 ms oleg213-client.virtnet: 1 81959 2.43 MB/sec execute 151 sec latency 4559.774 ms oleg213-client.virtnet: 1 81959 2.42 MB/sec execute 152 sec latency 5560.084 ms oleg213-client.virtnet: 1 81959 2.40 MB/sec executeFailover mds2 to oleg213-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37296 3569272 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11916 3594164 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49212 7163436 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg213-server Stopping /mnt/lustre-mds1 (opts:) on oleg213-server reboot facets: mds1 Failover mds1 to oleg213-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 153 sec latency 6560.287 ms oleg213-client.virtnet: 1 81959 2.39 MB/sec execute 154 sec latency 7560.452 ms oleg213-client.virtnet: 1 81959 2.37 MB/sec execute 155 sec latency 8560.712 ms oleg213-client.virtnet: 1 81959 2.36 MB/sec execute 156 sec latency 9561.054 ms oleg213-client.virtnet: 1 81959 2.34 MB/sec execute 157 sec latency 10561.201 ms oleg213-client.virtnet: 1 81959 2.33 MB/sec execute 158 sec latency 11561.342 ms oleg213-client.virtnet: 1 81959 2.31 MB/sec execute 159 sec latency 12561.579 ms oleg213-client.virtnet: 1 81959 2.30 MB/sec execute 160 sec latency 13561.863 ms oleg213-client.virtnet: 1 81959 2.28 MB/sec execute 161 sec latency 14562.062 ms oleg213-client.virtnet: 1 81959 2.27 MB/sec execute 162 sec latency 15562.333 ms oleg213-client.virtnet: 1 81959 2.26 MB/sec execute 163 sec latency 16562.568 ms oleg213-client.virtnet: 1 81959 2.24 MB/sec execute 164 sec latency 17562.797 ms oleg213-client.virtnet: 1 81959 2.23 MB/sec execute 165 sec latency 18563.006 ms oleg213-client.virtnet: 1 81959 2.21 MB/sec execute 166 sec latency 19563.256 ms oleg213-client.virtnet: 1 82385 2.21 MB/sec execute 167 sec latency 19744.664 ms oleg213-client.virtnet: 1 82866 2.20 MB/sec execute 168 sec latency 13.467 ms oleg213-client.virtnet: 1 83250 2.21 MB/sec execute 169 sec latency 17.040 ms oleg213-client.virtnet: 1 83615 2.20 MB/sec execute 170 sec latency 17.591 ms oleg213-client.virtnet: 1 84058 2.19 MB/sec execute 171 sec latency 11.238 ms oleg213-client.virtnet: 1 84824 2.22 MB/sec execute 172 sec latency 15.413 ms oleg213-client.virtnet: 1 85560 2.24 MB/sec execute 173 sec latency 9.683 ms oleg213-client.virtnet: 1 86035 2.23 MB/sec execute 174 sec latency 18.236 ms oleg213-client.virtnet: 1 86446 2.22 MB/sec execute 175 sec latency 22.020 ms oleg213-client.virtnet: 1 87087 2.23 MB/sec execute 176 sec latency 10.663 ms oleg213-client.virtnet: 1 87574 2.23 MB/sec execute 177 sec latency 13.374 ms oleg213-client.virtnet: 1 88144 2.24 MB/sec execute 178 sec latency 15.511 ms oleg213-client.virtnet: 1 88747 2.26 MB/sec execute 179 sec latency 17.860 ms oleg213-client.virtnet: 1 89098 2.26 MB/sec execute 180 sec latency 17.802 ms oleg213-client.virtnet: 1 89423 2.25 MB/sec execute 181 sec latency 18.561 ms oleg213-client.virtnet: 1 89890 2.24 MB/sec execute 182 sec latency 12.741 ms oleg213-client.virtnet: 1 90298 2.25 MB/sec execute 183 sec latency 18.382 ms oleg213-client.virtnet: 1 90597 2.24 MB/sec execute 184 sec latency 18.110 ms oleg213-client.virtnet: 1 91170 2.23 MB/sec execute 185 sec latency 14.755 ms oleg213-client.virtnet: 1 91735 2.25 MB/sec execute 186 sec latency 17.580 ms oleg213-client.virtnet: 1 92354 2.27 MB/sec execute 187 sec latency 18.501 ms oleg213-client.virtnet: 1 92634 2.26 MB/sec execute 188 sec latency 17.451 ms oleg213-client.virtnet: 1 93046 2.25 MB/sec execute 189 sec latency 18.898 ms oleg213-client.virtnet: 1 93665 2.26 MB/sec execute 190 sec latency 11.186 ms oleg213-client.virtnet: 1 93992 2.25 MB/sec execute 191 sec latency 17.438 ms oleg213-client.virtnet: 1 94518 2.25 MB/sec execute 192 sec latency 10.509 ms oleg213-client.virtnet: 1 95021 2.26 MB/sec execute 193 sec latency 14.722 ms oleg213-client.virtnet: 1 95960 2.28 MB/sec execute 194 sec latency 9.702 ms oleg213-client.virtnet: 1 96226 2.28 MB/sec execute 195 sec latency 17.710 ms oleg213-client.virtnet: 1 96606 2.27 MB/sec execute 196 sec latency 22.394 ms oleg213-client.virtnet: 1 96952 2.27 MB/sec execute 197 sec latency 18.674 ms oleg213-client.virtnet: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1832 1285856 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569008 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11928 3592040 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49352 7161048 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg213-server Stopping /mnt/lustre-mds2 (opts:) on oleg213-server reboot facets: mds2 Failover mds2 to oleg213-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1285896 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37532 3568848 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11756 3592324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49288 7161172 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg213-server Stopping /mnt/lustre-mds1 (opts:) on oleg213-server reboot facets: mds1 1 97365 2.27 MB/sec execute 198 sec latency 16.917 ms oleg213-client.virtnet: 1 97882 2.26 MB/sec execute 199 sec latency 10.823 ms oleg213-client.virtnet: 1 98740 2.28 MB/sec execute 200 sec latency 10.198 ms oleg213-client.virtnet: 1 99644 2.30 MB/sec execute 201 sec latency 9.738 ms oleg213-client.virtnet: 1 100072 2.30 MB/sec execute 202 sec latency 16.899 ms oleg213-client.virtnet: 1 100763 2.31 MB/sec execute 203 sec latency 16.916 ms oleg213-client.virtnet: 1 101010 2.30 MB/sec execute 204 sec latency 543.828 ms oleg213-client.virtnet: 1 101416 2.29 MB/sec execute 205 sec latency 15.880 ms oleg213-client.virtnet: 1 101835 2.29 MB/sec execute 206 sec latency 12.113 ms oleg213-client.virtnet: 1 102289 2.30 MB/sec execute 207 sec latency 546.404 ms oleg213-client.virtnet: 1 102289 2.29 MB/sec execute 208 sec latency 1546.545 ms oleg213-client.virtnet: 1 102289 2.28 MB/sec execute 209 sec latency 2546.849 ms oleg213-client.virtnet: 1 102289 2.27 MB/sec execute 210 sec latency 3547.177 ms oleg213-client.virtnet: 1 102289 2.26 MB/sec execute 211 sec latency 4547.404 ms oleg213-client.virtnet: 1 102289 2.25 MB/sec execute 212 sec latency 5547.686 ms oleg213-client.virtnet: 1 102289 2.24 MB/sec execute 213 sec latency 6547.937 ms oleg213-client.virtnet: 1 102289 2.22 MB/sec execute 214 sec latency 7548.201 ms oleg213-client.virtnet: 1 102289 2.21 MB/sec execute 215 sec latency 8548.555 ms oleg213-client.virtnet: 1 102289 2.20 MB/sec execute 216 sec latency 9548.805 ms oleg213-client.virtnet: 1 102289 2.19 MB/sec execute 217 sec latency 10549.078 ms oleg213-client.virtnet: 1 102289 2.18 MB/sec execute 218 sec latency 11549.338 ms oleg213-client.virtnet: 1 102289 2.17 MB/sec execute 219 sec latency 12549.544 ms oleg213-client.virtnet: 1 102289 2.16 MB/sec execute 220 sec latency 13549.749 ms oleg213-client.virtnet: 1 102289 2.15 MB/sec execute 221 sec latency 14549.872 ms oleg213-client.virtnet: 1 102289 2.14 MB/sec execute 222 sec latency 15550.073 ms oleg213-client.virtnet: 1 102289 2.13 MB/sec execute 223 sec latency 16550.343 ms oleg213-client.virtnet: 1 102289 2.13 MB/sec execute 224 sec latency 17550.500 ms oleg213-client.virtnet: 1 102289 2.12 MB/sec execute 225 sec latency 18550.668 ms oleg213-client.virtnet: 1 102289 2.11 MB/sec execute 226 sec latency 19550.878 ms oleg213-client.virtnet: 1 102905 2.12 MB/sec execute 227 sec latency 19683.082 ms oleg213-client.virtnet: 1 103249 2.12 MB/sec execute 228 sec latency 17.746 ms oleg213-client.virtnet: 1 103654 2.11 MB/sec execute 229 sec latency 17.351 ms oleg213-client.virtnet: 1 104002 2.11 MB/sec execute 230 sec latency 17.939 ms oleg213-client.virtnet: 1 104637 2.11 MB/sec execute 231 sec latency 17.729 ms oleg213-client.virtnet: 1 105090 2.11 MB/sec execute 232 sec latency 12.083 ms oleg213-client.virtnet: 1 105487 2.11 MB/sec execute 233 sec latency 14.664 ms oleg213-client.virtnet: 1 106271 2.12 MB/sec execute 234 sec latency 17.280 ms oleg213-client.virtnet: 1 106773 2.13 MB/sec execute 235 sec latency 15.820 ms oleg213-client.virtnet: 1 107231 2.13 MB/sec execute 236 sec latency 13.716 ms oleg213-client.virtnet: 1 107708 2.13 MB/sec execute 237 sec latency 17.566 ms oleg213-client.virtnet: 1 108129 2.13 MB/sec execute 238 sec latency 17.138 ms oleg213-client.virtnet: 1 108686 2.12 MB/sec execute 239 sec latency 10.270 ms oleg213-client.virtnet: 1 109046 2.12 MB/sec execute 240 sec latency 16.162 ms oleg213-client.virtnet: 1 109755 2.14 MB/sec execute 241 sec latency 17.197 ms oleg213-client.virtnet: 1 110297 2.15 MB/sec execute 2Failover mds1 to oleg213-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 42 sec latency 12.739 ms oleg213-client.virtnet: 1 110585 2.14 MB/sec execute 243 sec latency 18.237 ms oleg213-client.virtnet: 1 110982 2.14 MB/sec execute 244 sec latency 18.368 ms oleg213-client.virtnet: 1 111472 2.15 MB/sec execute 245 sec latency 17.703 ms oleg213-client.virtnet: 1 111813 2.14 MB/sec execute 246 sec latency 17.464 ms oleg213-client.virtnet: 1 112248 2.13 MB/sec execute 247 sec latency 10.468 ms oleg213-client.virtnet: 1 112974 2.15 MB/sec execute 248 sec latency 18.660 ms oleg213-client.virtnet: 1 113579 2.16 MB/sec execute 249 sec latency 17.992 ms oleg213-client.virtnet: 1 113868 2.16 MB/sec execute 250 sec latency 17.788 ms oleg213-client.virtnet: 1 114153 2.15 MB/sec execute 251 sec latency 18.440 ms oleg213-client.virtnet: 1 114475 2.15 MB/sec execute 252 sec latency 19.729 ms oleg213-client.virtnet: 1 115037 2.15 MB/sec execute 253 sec latency 15.672 ms oleg213-client.virtnet: 1 115645 2.15 MB/sec execute 254 sec latency 9.701 ms oleg213-client.virtnet: 1 116012 2.15 MB/sec execute 255 sec latency 14.668 ms oleg213-client.virtnet: 1 116565 2.16 MB/sec execute 256 sec latency 17.376 ms oleg213-client.virtnet: 1 117278 2.17 MB/sec execute 257 sec latency 12.559 ms oleg213-client.virtnet: 1 117506 2.17 MB/sec execute 258 sec latency 18.365 ms oleg213-client.virtnet: 1 117822 2.16 MB/sec execute 259 sec latency 23.513 ms oleg213-client.virtnet: 1 118140 2.16 MB/sec execute 260 sec latency 17.280 ms oleg213-client.virtnet: 1 118588 2.16 MB/sec execute 261 sec latency 17.937 ms oleg213-client.virtnet: 1 118951 2.15 MB/sec execute 262 sec latency 17.330 ms oleg213-client.virtnet: 1 119526 2.15 MB/sec execute 263 sec latency 13.864 ms oleg213-client.virtnet: 1 120264 2.17 MB/sec execute 264 sec latency 16.311 ms oleg213-client.virtnet: 1 120837 2.18 MB/sec execute 265 sec latency 12.186 ms oleg213-client.virtnet: 1 121357 2.17 MB/sec execute 266 sec latency 13.979 ms oleg213-client.virtnet: 1 121709 2.17 MB/sec execute 267 sec latency 17.589 ms oleg213-client.virtnet: 1 122132 2.18 MB/sec execute 268 sec latency 18.311 ms oleg213-client.virtnet: 1 122476 2.17 MB/sec execute 269 sec latency 18.598 ms oleg213-client.virtnet: 1 122987 2.17 MB/sec execute 270 sec latency 9.893 ms oleg213-client.virtnet: 1 123510 2.18 MB/sec execute 271 sec latency 16.007 ms oleg213-client.virtnet: 1 124271 2.19 MB/sec execute 272 sec latency 12.996 ms oleg213-client.virtnet: 1 124597 2.19 MB/sec execute 273 sec latency 17.445 ms oleg213-client.virtnet: 1 124922 2.18 MB/sec execute 274 sec latency 17.240 ms oleg213-client.virtnet: 1 125339 2.18 MB/sec execute 275 sec latency 18.039 ms oleg213-client.virtnet: 1 125808 2.18 MB/sec execute 276 sec latency 27.708 ms oleg213-client.virtnet: 1 126158 2.18 MB/sec execute 277 sec latency 20.871 ms oleg213-client.virtnet: 1 126571 2.17 MB/sec execute 278 sec latency 18.800 ms oleg213-client.virtnet: 1 127084 2.18 MB/sec execute 279 sec latency 15.730 ms oleg213-client.virtnet: 1 127764 2.20 MB/sec execute 280 sec latency 14.965 ms oleg213-client.virtnet: 1 128051 2.19 MB/sec execute 281 sec latency 17.355 ms oleg213-client.virtnet: 1 128394 2.19 MB/sec execute 282 sec latency 17.655 ms oleg213-client.virtnet: 1 128677 2.18 MB/sec execute 283 sec latency 24.672 ms oleg213-client.virtnet: 1 129162 2.19 MB/sec execute 284 sec latency 16.687 ms oleg213-client.virtnet: 1 129772 2.18 MB/sec execute 285 sec latency 9.183 ms oleg213-client.virtnet: 1 130169 2.18 MB/sec execute 286 sec latency 16.459 ms oleg213-client.virtnet: 1 130721 2.19 MB/sec execute 287 sec latency 18.477 ms oleg213-client.virtnet: 1 131359 2.20 MB/sec execute 288 sec latency 16.693 ms oleg213-client.virtnet: 1 131750 2.20 MB/sec execute 289 sec latency 15.693 ms oleg213-client.virtnet: 1 132420 2.20 MB/sec execute 290 sec latency 13.048 ms oleg213-client.virtnet: 1 133143 2.20 MB/sec execute 291 sec latency 8.861 ms oleg213-client.virtnet: 1 133910 2.21 MB/sec execute 292 sec latency 9.271 ms oleg213-client.virtnet: 1 135001 2.23 MB/sec execute 293 sec latency 9.302 ms oleg213-client.virtnet: 1 135487 2.23 MB/sec execute 294 sec latency 12.065 ms oleg213-client.virtnet: 1 136058 2.23 MB/sec execute 295 sec latency 13.710 ms oleg213-client.virtnet: 1 136809 2.23 MB/sec execute 296 sec latency 9.350 ms oleg213-client.virtnet: 1 137594 2.24 MB/sec execute 297 sec latency 8.888 ms oleg213-client.virtnet: 1 138585 2.26 MB/sec execute 298 sec latency 9.799 ms oleg213-client.virtnet: 1 139115 2.25 MB/sec execute 299 sec latency 13.390 ms oleg213-client.virtnet: 1 cleanup 300 sec oleg213-client.virtnet: 0 cleanup 300 sec oleg213-client.virtnet: oleg213-client.virtnet: Operation Count AvgLat MaxLat oleg213-client.virtnet: ---------------------------------------- oleg213-client.virtnet: NTCreateX 21783 8.295 19744.655 oleg213-client.virtnet: Close 15973 1.188 5.121 oleg213-client.virtnet: Rename 924 8.497 20.326 oleg213-client.virtnet: Unlink 4428 2.022 11.487 oleg213-client.virtnet: Qpathinfo 19712 1.466 27.690 oleg213-client.virtnet: Qfileinfo 3416 0.136 2.441 oleg213-client.virtnet: Qfsinfo 3628 0.147 14.209 oleg213-client.virtnet: Sfileinfo 1784 3.785 13.333 oleg213-client.virtnet: Find 7601 0.409 11.362 oleg213-client.virtnet: WriteX 10725 2.888 19338.562 oleg213-client.virtnet: ReadX 34075 0.012 2.303 oleg213-client.virtnet: LockX 70 1.315 2.263 oleg213-client.virtnet: UnlockX 70 1.338 2.239 oleg213-client.virtnet: Flush 1529 7.418 543.808 oleg213-client.virtnet: oleg213-client.virtnet: Throughput 2.25492 MB/sec 1 clients 1 procs max_latency=19744.664 ms oleg213-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg213-client.virtnet at Thu Jun 20 20:49:06 EDT 2024 with return code 0 oleg213-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg213-client.virtnet oleg213-client.virtnet: /mnt/lustre/d70b.replay-single/oleg213-client.virtnet /mnt/lustre/d70b.replay-single/oleg213-client.virtnet oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg213-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg213-client.virtnet: removed directory: 'clients/client0' oleg213-client.virtnet: removed directory: 'clients' oleg213-client.virtnet: removed 'client.txt' oleg213-client.virtnet: /mnt/lustre/d70b.replay-single/oleg213-client.virtnet oleg213-client.virtnet: dbench successfully finished