== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 13:17:40 (1721409460) Starting client oleg357-client.virtnet: -o user_xattr,flock oleg357-server@tcp:/lustre /mnt/lustre Started clients oleg357-client.virtnet: 192.168.203.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg357-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=19371 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2232 1285456 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26016 3556284 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1424 3592384 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27440 7148668 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg357-server Stopping /mnt/lustre-mds1 (opts:) on oleg357-server reboot facets: mds1 Failover mds1 to oleg357-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2192 1285496 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1285828 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26396 3569856 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1608 3592832 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28004 7162688 1% /mnt/lustre oleg357-client.virtnet: looking for dbench program oleg357-client.virtnet: /usr/bin/dbench oleg357-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg357-client.virtnet oleg357-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg357-client.virtnet' oleg357-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg357-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg357-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg357-client.virtnet at Fri Jul 19 13:17:40 EDT 2024 oleg357-client.virtnet: waiting for dbench pid 19396 oleg357-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg357-client.virtnet: oleg357-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg357-client.virtnet: failed to create barrier semaphore oleg357-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg357-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg357-client.virtnet: releasing clients oleg357-client.virtnet: 1 525 18.05 MB/sec warmup 1 sec latency 10.278 ms oleg357-client.virtnet: 1 1257 11.55 MB/sec warmup 2 sec latency 9.219 ms oleg357-client.virtnet: 1 2063 8.87 MB/sec warmup 3 sec latency 8.937 ms oleg357-client.virtnet: 1 3011 8.38 MB/sec warmup 4 sec latency 7.912 ms oleg357-client.virtnet: 1 3949 8.01 MB/sec warmup 5 sec latency 11.418 ms oleg357-client.virtnet: 1 4606 7.00 MB/sec warmup 6 sec latency 10.572 ms oleg357-client.virtnet: 1 5350 6.49 MB/sec warmup 7 sec latency 8.790 ms oleg357-client.virtnet: 1 6155 6.01 MB/sec warmup 8 sec latency 16.218 ms oleg357-client.virtnet: 1 7357 6.54 MB/sec warmup 9 sec latency 8.040 ms oleg357-client.virtnet: 1 7997 5.99 MB/sec warmup 10 sec latency 9.797 ms oleg357-client.virtnet: 1 8786 5.86 MB/sec warmup 11 sec latency 8.552 ms oleg357-client.virtnet: 1 9591 5.58 MB/sec warmup 12 sec latency 11.400 ms oleg357-client.virtnet: 1 10792 5.92 MB/sec warmup 13 sec latency 8.048 ms oleg357-client.virtnet: 1 11390 5.62 MB/sec warmup 14 sec latency 10.936 ms oleg357-client.virtnet: 1 12143 5.56 MB/sec warmup 15 sec latency 10.273 ms oleg357-client.virtnet: 1 12939 5.30 MB/sec warmup 16 sec latency 8.530 ms oleg357-client.virtnet: 1 14008 5.46 MB/sec warmup 17 sec latency 8.631 ms oleg357-client.virtnet: 1 14657 5.41 MB/sec warmup 18 sec latency 9.855 ms oleg357-client.virtnet: 1 15350 5.24 MB/sec warmup 19 sec latency 9.258 ms oleg357-client.virtnet: 1 16085 5.14 MB/sec warmup 20 sec latency 9.348 ms oleg357-client.virtnet: 1 16969 5.16 MB/sec warmup 21 sec latency 10.871 ms oleg357-client.virtnet: 1 17991 5.28 MB/sec warmup 22 sec latency 9.402 ms oleg357-client.virtnet: 1 18636 5.10 MB/sec warmup 23 sec latency 8.308 ms oleg357-client.virtnet: 1 19409 5.08 MB/sec warmup 24 sec latency 8.524 ms oleg357-client.virtnet: 1 20197 4.95 MB/sec warmup 25 sec latency 8.178 ms oleg357-client.virtnet: 1 21404 5.16 MB/sec warmup 26 sec latency 8.124 ms oleg357-client.virtnet: 1 22003 5.03 MB/sec warmup 27 sec latency 12.431 ms oleg357-client.virtnet: 1 22732 5.03 MB/sec warmup 28 sec latency 9.477 ms oleg357-client.virtnet: 1 23477 4.88 MB/sec warmup 29 sec latency 8.504 ms oleg357-client.virtnet: 1 24500 4.96 MB/sec warmup 30 sec latency 8.732 ms oleg357-client.virtnet: 1 25267 4.99 MB/sec warmup 31 sec latency 9.187 ms oleg357-client.virtnet: 1 25907 4.89 MB/sec warmup 32 sec latency 10.078 ms oleg357-client.virtnet: 1 26668 4.85 MB/sec warmup 33 sec latency 10.344 ms oleg357-client.virtnet: 1 27353 4.78 MB/sec warmuptest_70b fail mds2 2 times Failing mds2 on oleg357-server Stopping /mnt/lustre-mds2 (opts:) on oleg357-server reboot facets: mds2 Failover mds2 to oleg357-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1796 1285892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36428 3567672 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12800 3592964 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49228 7160636 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg357-server Stopping /mnt/lustre-mds1 (opts:) on oleg357-server reboot facets: mds1 Failover mds1 to oleg357-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 34 sec latency 98.602 ms oleg357-client.virtnet: 1 28222 4.84 MB/sec warmup 35 sec latency 416.693 ms oleg357-client.virtnet: 1 28858 4.83 MB/sec warmup 36 sec latency 11.308 ms oleg357-client.virtnet: 1 29159 4.71 MB/sec warmup 37 sec latency 570.614 ms oleg357-client.virtnet: 1 29159 4.59 MB/sec warmup 38 sec latency 1570.786 ms oleg357-client.virtnet: 1 29159 4.47 MB/sec warmup 39 sec latency 2570.941 ms oleg357-client.virtnet: 1 29159 4.36 MB/sec warmup 40 sec latency 3571.118 ms oleg357-client.virtnet: 1 29159 4.25 MB/sec warmup 41 sec latency 4571.341 ms oleg357-client.virtnet: 1 29159 4.15 MB/sec warmup 42 sec latency 5571.594 ms oleg357-client.virtnet: 1 29159 4.05 MB/sec warmup 43 sec latency 6571.803 ms oleg357-client.virtnet: 1 29159 3.96 MB/sec warmup 44 sec latency 7571.989 ms oleg357-client.virtnet: 1 29159 3.87 MB/sec warmup 45 sec latency 8572.183 ms oleg357-client.virtnet: 1 29159 3.79 MB/sec warmup 46 sec latency 9572.385 ms oleg357-client.virtnet: 1 29159 3.71 MB/sec warmup 47 sec latency 10572.522 ms oleg357-client.virtnet: 1 29159 3.63 MB/sec warmup 48 sec latency 11572.709 ms oleg357-client.virtnet: 1 29159 3.56 MB/sec warmup 49 sec latency 12572.818 ms oleg357-client.virtnet: 1 29159 3.49 MB/sec warmup 50 sec latency 13572.985 ms oleg357-client.virtnet: 1 29159 3.42 MB/sec warmup 51 sec latency 14573.110 ms oleg357-client.virtnet: 1 29159 3.35 MB/sec warmup 52 sec latency 15573.287 ms oleg357-client.virtnet: 1 29159 3.29 MB/sec warmup 53 sec latency 16573.493 ms oleg357-client.virtnet: 1 29159 3.23 MB/sec warmup 54 sec latency 17573.776 ms oleg357-client.virtnet: 1 29159 3.17 MB/sec warmup 55 sec latency 18573.904 ms oleg357-client.virtnet: 1 29834 3.20 MB/sec warmup 56 sec latency 18680.578 ms oleg357-client.virtnet: 1 30555 3.15 MB/sec warmup 57 sec latency 9.827 ms oleg357-client.virtnet: 1 31586 3.23 MB/sec warmup 58 sec latency 8.821 ms oleg357-client.virtnet: 1 32337 3.27 MB/sec warmup 59 sec latency 9.695 ms oleg357-client.virtnet: 1 33657 4.44 MB/sec execute 1 sec latency 12.109 ms oleg357-client.virtnet: 1 34395 3.16 MB/sec execute 2 sec latency 8.797 ms oleg357-client.virtnet: 1 35571 5.61 MB/sec execute 3 sec latency 8.425 ms oleg357-client.virtnet: 1 36186 4.63 MB/sec execute 4 sec latency 11.068 ms oleg357-client.virtnet: 1 36904 4.67 MB/sec execute 5 sec latency 9.749 ms oleg357-client.virtnet: 1 37649 4.00 MB/sec execute 6 sec latency 9.135 ms oleg357-client.virtnet: 1 38280 4.19 MB/sec execute 7 sec latency 12.136 ms oleg357-client.virtnet: 1 39053 4.47 MB/sec execute 8 sec latency 17.848 ms oleg357-client.virtnet: 1 39494 4.16 MB/sec execute 9 sec latency 13.050 ms oleg357-client.virtnet: 1 40052 3.84 MB/sec execute 10 sec latency 14.219 ms oleg357-client.virtnet: 1 40630 3.87 MB/sec execute 11 sec latency 13.158 ms oleg357-client.virtnet: 1 41193 3.59 MB/sec execute 12 sec latency 16.535 ms oleg357-client.virtnet: 1 41633 3.49 MB/sec execute 13 sec latency 13.881 ms oleg357-client.virtnet: 1 42484 3.78 MB/sec execute 14 sec latency 16.188 ms oleg357-client.virtnet: 1 43059 3.77 MB/sec execute 15 sec latency 12.503 ms oleg357-client.virtnet: 1 43733 3.66 MB/sec execute 16 sec latency 9.623 ms oleg357-client.virtnet: 1 44431 3.64 MB/sec execute 17 sec latency 10.663 ms oleg357-client.virtnet: 1 45316 3.75 MB/sec execute 18 sec latency 8.076 ms oleg357-client.virtnet: 1 46373 3.97 MB/sec execute 19 sec latency 9.454 ms oleg357-client.virtnet: 1 46935 3.80 Moleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36432 3567876 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12928 3592980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49360 7160856 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg357-server Stopping /mnt/lustre-mds2 (opts:) on oleg357-server reboot facets: mds2 Failover mds2 to oleg357-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36556 3569912 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12796 3592648 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49352 7162560 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg357-server Stopping /mnt/lustre-mds1 (opts:) on oleg357-server reboot facets: mds1 B/sec execute 20 sec latency 11.150 ms oleg357-client.virtnet: 1 47647 3.84 MB/sec execute 21 sec latency 10.356 ms oleg357-client.virtnet: 1 48408 3.70 MB/sec execute 22 sec latency 9.066 ms oleg357-client.virtnet: 1 49494 3.92 MB/sec execute 23 sec latency 9.113 ms oleg357-client.virtnet: 1 50138 3.95 MB/sec execute 24 sec latency 9.228 ms oleg357-client.virtnet: 1 50792 3.87 MB/sec execute 25 sec latency 10.009 ms oleg357-client.virtnet: 1 51501 3.85 MB/sec execute 26 sec latency 9.134 ms oleg357-client.virtnet: 1 52286 3.81 MB/sec execute 27 sec latency 7.470 ms oleg357-client.virtnet: 1 53446 4.06 MB/sec execute 28 sec latency 8.870 ms oleg357-client.virtnet: 1 53961 3.94 MB/sec execute 29 sec latency 70.597 ms oleg357-client.virtnet: 1 54711 3.97 MB/sec execute 30 sec latency 9.180 ms oleg357-client.virtnet: 1 55387 3.86 MB/sec execute 31 sec latency 9.078 ms oleg357-client.virtnet: 1 55405 3.74 MB/sec execute 32 sec latency 964.921 ms oleg357-client.virtnet: 1 55405 3.63 MB/sec execute 33 sec latency 1965.086 ms oleg357-client.virtnet: 1 55405 3.52 MB/sec execute 34 sec latency 2965.257 ms oleg357-client.virtnet: 1 55405 3.42 MB/sec execute 35 sec latency 3965.404 ms oleg357-client.virtnet: 1 55405 3.32 MB/sec execute 36 sec latency 4965.578 ms oleg357-client.virtnet: 1 55405 3.23 MB/sec execute 37 sec latency 5965.830 ms oleg357-client.virtnet: 1 55405 3.15 MB/sec execute 38 sec latency 6966.006 ms oleg357-client.virtnet: 1 55405 3.07 MB/sec execute 39 sec latency 7966.164 ms oleg357-client.virtnet: 1 55405 2.99 MB/sec execute 40 sec latency 8966.329 ms oleg357-client.virtnet: 1 55405 2.92 MB/sec execute 41 sec latency 9966.491 ms oleg357-client.virtnet: 1 55405 2.85 MB/sec execute 42 sec latency 10966.751 ms oleg357-client.virtnet: 1 55405 2.78 MB/sec execute 43 sec latency 11966.943 ms oleg357-client.virtnet: 1 55405 2.72 MB/sec execute 44 sec latency 12967.160 ms oleg357-client.virtnet: 1 55405 2.66 MB/sec execute 45 sec latency 13967.366 ms oleg357-client.virtnet: 1 55405 2.60 MB/sec execute 46 sec latency 14967.627 ms oleg357-client.virtnet: 1 55405 2.55 MB/sec execute 47 sec latency 15967.839 ms oleg357-client.virtnet: 1 55405 2.49 MB/sec execute 48 sec latency 16968.046 ms oleg357-client.virtnet: 1 55405 2.44 MB/sec execute 49 sec latency 17968.292 ms oleg357-client.virtnet: 1 55611 2.42 MB/sec execute 50 sec latency 18702.805 ms oleg357-client.virtnet: 1 56425 2.49 MB/sec execute 51 sec latency 12.119 ms oleg357-client.virtnet: 1 57172 2.56 MB/sec execute 52 sec latency 10.097 ms oleg357-client.virtnet: 1 57679 2.52 MB/sec execute 53 sec latency 17.689 ms oleg357-client.virtnet: 1 58260 2.56 MB/sec execute 54 sec latency 14.040 ms oleg357-client.virtnet: 1 58895 2.52 MB/sec execute 55 sec latency 12.192 ms oleg357-client.virtnet: 1 59772 2.60 MB/sec execute 56 sec latency 8.990 ms oleg357-client.virtnet: 1 60551 2.66 MB/sec execute 57 sec latency 11.405 ms oleg357-client.virtnet: 1 61107 2.63 MB/sec execute 58 sec latency 11.779 ms oleg357-client.virtnet: 1 61828 2.66 MB/sec execute 59 sec latency 9.114 ms oleg357-client.virtnet: 1 62495 2.63 MB/sec execute 60 sec latency 13.832 ms oleg357-client.virtnet: 1 63516 2.71 MB/sec execute 61 sec latency 8.840 ms oleg357-client.virtnet: 1 64258 2.76 MB/sec execute 62 sec latency 8.955 ms oleg357-client.virtnet: 1 64862 2.73 MB/sec execute 63 sec latency 10.415 ms oleg357-client.virtnet: 1 65613 2.76 MB/sec execute 64 sec latency 9.491 ms oleg357-client.viFailover mds1 to oleg357-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1848 1285840 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36384 3570124 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13180 3592812 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7162936 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg357-server Stopping /mnt/lustre-mds2 (opts:) on oleg357-server reboot facets: mds2 Failover mds2 to oleg357-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36268 3569948 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12952 3592104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49220 7162052 1% /mnt/lustre rtnet: 1 66393 2.76 MB/sec execute 65 sec latency 7.929 ms oleg357-client.virtnet: 1 67545 2.86 MB/sec execute 66 sec latency 9.102 ms oleg357-client.virtnet: 1 68127 2.84 MB/sec execute 67 sec latency 10.040 ms oleg357-client.virtnet: 1 68863 2.87 MB/sec execute 68 sec latency 8.874 ms oleg357-client.virtnet: 1 69604 2.84 MB/sec execute 69 sec latency 9.010 ms oleg357-client.virtnet: 1 70624 2.91 MB/sec execute 70 sec latency 8.765 ms oleg357-client.virtnet: 1 71358 2.95 MB/sec execute 71 sec latency 9.345 ms oleg357-client.virtnet: 1 72000 2.94 MB/sec execute 72 sec latency 10.322 ms oleg357-client.virtnet: 1 72763 2.94 MB/sec execute 73 sec latency 9.181 ms oleg357-client.virtnet: 1 73558 2.94 MB/sec execute 74 sec latency 7.590 ms oleg357-client.virtnet: 1 74686 3.03 MB/sec execute 75 sec latency 9.076 ms oleg357-client.virtnet: 1 75316 3.01 MB/sec execute 76 sec latency 11.965 ms oleg357-client.virtnet: 1 76102 3.04 MB/sec execute 77 sec latency 8.649 ms oleg357-client.virtnet: 1 76887 3.02 MB/sec execute 78 sec latency 9.553 ms oleg357-client.virtnet: 1 77908 3.08 MB/sec execute 79 sec latency 11.208 ms oleg357-client.virtnet: 1 78564 3.10 MB/sec execute 80 sec latency 9.157 ms oleg357-client.virtnet: 1 79265 3.08 MB/sec execute 81 sec latency 9.819 ms oleg357-client.virtnet: 1 79977 3.09 MB/sec execute 82 sec latency 9.148 ms oleg357-client.virtnet: 1 80504 3.07 MB/sec execute 83 sec latency 362.510 ms oleg357-client.virtnet: 1 81638 3.16 MB/sec execute 84 sec latency 398.273 ms oleg357-client.virtnet: 1 82164 3.14 MB/sec execute 85 sec latency 9.732 ms oleg357-client.virtnet: 1 82303 3.10 MB/sec execute 86 sec latency 861.844 ms oleg357-client.virtnet: 1 82303 3.07 MB/sec execute 87 sec latency 1862.093 ms oleg357-client.virtnet: 1 82303 3.03 MB/sec execute 88 sec latency 2862.268 ms oleg357-client.virtnet: 1 82303 3.00 MB/sec execute 89 sec latency 3862.455 ms oleg357-client.virtnet: 1 82303 2.97 MB/sec execute 90 sec latency 4862.700 ms oleg357-client.virtnet: 1 82303 2.93 MB/sec execute 91 sec latency 5862.896 ms oleg357-client.virtnet: 1 82303 2.90 MB/sec execute 92 sec latency 6863.118 ms oleg357-client.virtnet: 1 82303 2.87 MB/sec execute 93 sec latency 7863.318 ms oleg357-client.virtnet: 1 82303 2.84 MB/sec execute 94 sec latency 8863.547 ms oleg357-client.virtnet: 1 82303 2.81 MB/sec execute 95 sec latency 9863.766 ms oleg357-client.virtnet: 1 82303 2.78 MB/sec execute 96 sec latency 10863.983 ms oleg357-client.virtnet: 1 82303 2.75 MB/sec execute 97 sec latency 11864.124 ms oleg357-client.virtnet: 1 82303 2.72 MB/sec execute 98 sec latency 12864.261 ms oleg357-client.virtnet: 1 82303 2.70 MB/sec execute 99 sec latency 13864.430 ms oleg357-client.virtnet: 1 82303 2.67 MB/sec execute 100 sec latency 14864.575 ms oleg357-client.virtnet: 1 82303 2.64 MB/sec execute 101 sec latency 15864.833 ms oleg357-client.virtnet: 1 82303 2.62 MB/sec execute 102 sec latency 16865.026 ms oleg357-client.virtnet: 1 82303 2.59 MB/sec execute 103 sec latency 17865.207 ms oleg357-client.virtnet: 1 82342 2.57 MB/sec execute 104 sec latency 18789.829 ms oleg357-client.virtnet: 1 83048 2.59 MB/sec execute 105 sec latency 9.588 ms oleg357-client.virtnet: 1 83718 2.57 MB/sec execute 106 sec latency 12.363 ms oleg357-client.virtnet: 1 84621 2.61 MB/sec execute 107 sec latency 8.813 ms oleg357-client.virtnet: 1 85515 2.65 MB/sec execute 108 sec latency 9.504 ms oleg357-client.virtnet: 1 86126 2.63 MB/sec execute 109 sec latenctest_70b fail mds1 7 times Failing mds1 on oleg357-server Stopping /mnt/lustre-mds1 (opts:) on oleg357-server reboot facets: mds1 Failover mds1 to oleg357-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36400 3569784 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13024 3590908 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49424 7160692 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg357-server Stopping /mnt/lustre-mds2 (opts:) on oleg357-server reboot facets: mds2 Failover mds2 to oleg357-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 y 9.860 ms oleg357-client.virtnet: 1 86868 2.65 MB/sec execute 110 sec latency 10.180 ms oleg357-client.virtnet: 1 87613 2.64 MB/sec execute 111 sec latency 8.592 ms oleg357-client.virtnet: 1 88741 2.71 MB/sec execute 112 sec latency 10.398 ms oleg357-client.virtnet: 1 89231 2.70 MB/sec execute 113 sec latency 10.984 ms oleg357-client.virtnet: 1 89879 2.69 MB/sec execute 114 sec latency 15.191 ms oleg357-client.virtnet: 1 90587 2.70 MB/sec execute 115 sec latency 11.089 ms oleg357-client.virtnet: 1 91381 2.71 MB/sec execute 116 sec latency 10.442 ms oleg357-client.virtnet: 1 92485 2.77 MB/sec execute 117 sec latency 9.200 ms oleg357-client.virtnet: 1 93060 2.75 MB/sec execute 118 sec latency 10.160 ms oleg357-client.virtnet: 1 93828 2.77 MB/sec execute 119 sec latency 9.088 ms oleg357-client.virtnet: 1 94640 2.76 MB/sec execute 120 sec latency 9.393 ms oleg357-client.virtnet: 1 95619 2.80 MB/sec execute 121 sec latency 14.696 ms oleg357-client.virtnet: 1 96234 2.81 MB/sec execute 122 sec latency 10.001 ms oleg357-client.virtnet: 1 96866 2.81 MB/sec execute 123 sec latency 10.994 ms oleg357-client.virtnet: 1 97511 2.81 MB/sec execute 124 sec latency 15.432 ms oleg357-client.virtnet: 1 98272 2.80 MB/sec execute 125 sec latency 8.443 ms oleg357-client.virtnet: 1 99374 2.86 MB/sec execute 126 sec latency 10.398 ms oleg357-client.virtnet: 1 99937 2.85 MB/sec execute 127 sec latency 9.200 ms oleg357-client.virtnet: 1 100593 2.85 MB/sec execute 128 sec latency 11.469 ms oleg357-client.virtnet: 1 101351 2.85 MB/sec execute 129 sec latency 9.785 ms oleg357-client.virtnet: 1 102291 2.88 MB/sec execute 130 sec latency 9.118 ms oleg357-client.virtnet: 1 103222 2.91 MB/sec execute 131 sec latency 9.058 ms oleg357-client.virtnet: 1 103846 2.90 MB/sec execute 132 sec latency 11.338 ms oleg357-client.virtnet: 1 104608 2.91 MB/sec execute 133 sec latency 8.673 ms oleg357-client.virtnet: 1 105364 2.90 MB/sec execute 134 sec latency 10.436 ms oleg357-client.virtnet: 1 106542 2.96 MB/sec execute 135 sec latency 10.717 ms oleg357-client.virtnet: 1 107151 2.95 MB/sec execute 136 sec latency 14.939 ms oleg357-client.virtnet: 1 107860 2.96 MB/sec execute 137 sec latency 8.506 ms oleg357-client.virtnet: 1 108259 2.94 MB/sec execute 138 sec latency 332.764 ms oleg357-client.virtnet: 1 109020 2.94 MB/sec execute 139 sec latency 8.942 ms oleg357-client.virtnet: 1 110035 2.99 MB/sec execute 140 sec latency 169.436 ms oleg357-client.virtnet: 1 110035 2.97 MB/sec execute 141 sec latency 1169.673 ms oleg357-client.virtnet: 1 110035 2.95 MB/sec execute 142 sec latency 2169.898 ms oleg357-client.virtnet: 1 110035 2.92 MB/sec execute 143 sec latency 3170.114 ms oleg357-client.virtnet: 1 110035 2.90 MB/sec execute 144 sec latency 4170.327 ms oleg357-client.virtnet: 1 110035 2.88 MB/sec execute 145 sec latency 5170.550 ms oleg357-client.virtnet: 1 110035 2.86 MB/sec execute 146 sec latency 6170.730 ms oleg357-client.virtnet: 1 110035 2.85 MB/sec execute 147 sec latency 7170.916 ms oleg357-client.virtnet: 1 110035 2.83 MB/sec execute 148 sec latency 8171.078 ms oleg357-client.virtnet: 1 110035 2.81 MB/sec execute 149 sec latency 9171.228 ms oleg357-client.virtnet: 1 110035 2.79 MB/sec execute 150 sec latency 10171.382 ms oleg357-client.virtnet: 1 110035 2.77 MB/sec execute 151 sec latency 11171.499 ms oleg357-client.virtnet: 1 110035 2.75 MB/sec execute 152 sec latency 12171.636 ms oleg357-client.virtnet: 1 110035 2.73 MB/sec execute 153 sec latency 13171.828 ms oleg357-client.virtnet: 1 110035 2.72 MB/sec execuoleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36544 3569088 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12928 3593060 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49472 7162148 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg357-server Stopping /mnt/lustre-mds1 (opts:) on oleg357-server reboot facets: mds1 Failover mds1 to oleg357-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36428 3569004 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12784 3593016 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49212 7162020 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg357-server Stopping /mnt/lustre-mds2 (opts:) on oleg357-server reboot facets: mds2 te 154 sec latency 14172.026 ms oleg357-client.virtnet: 1 110035 2.70 MB/sec execute 155 sec latency 15172.203 ms oleg357-client.virtnet: 1 110035 2.68 MB/sec execute 156 sec latency 16172.398 ms oleg357-client.virtnet: 1 110035 2.66 MB/sec execute 157 sec latency 17172.561 ms oleg357-client.virtnet: 1 110035 2.65 MB/sec execute 158 sec latency 18172.758 ms oleg357-client.virtnet: 1 110344 2.64 MB/sec execute 159 sec latency 18602.060 ms oleg357-client.virtnet: 1 110884 2.63 MB/sec execute 160 sec latency 13.663 ms oleg357-client.virtnet: 1 111576 2.64 MB/sec execute 161 sec latency 11.045 ms oleg357-client.virtnet: 1 112349 2.63 MB/sec execute 162 sec latency 9.161 ms oleg357-client.virtnet: 1 113373 2.66 MB/sec execute 163 sec latency 9.052 ms oleg357-client.virtnet: 1 113991 2.68 MB/sec execute 164 sec latency 10.416 ms oleg357-client.virtnet: 1 114651 2.67 MB/sec execute 165 sec latency 9.340 ms oleg357-client.virtnet: 1 115399 2.68 MB/sec execute 166 sec latency 10.901 ms oleg357-client.virtnet: 1 116293 2.69 MB/sec execute 167 sec latency 14.162 ms oleg357-client.virtnet: 1 117335 2.72 MB/sec execute 168 sec latency 8.393 ms oleg357-client.virtnet: 1 117900 2.71 MB/sec execute 169 sec latency 11.587 ms oleg357-client.virtnet: 1 118622 2.72 MB/sec execute 170 sec latency 9.657 ms oleg357-client.virtnet: 1 119422 2.72 MB/sec execute 171 sec latency 8.668 ms oleg357-client.virtnet: 1 120442 2.75 MB/sec execute 172 sec latency 9.180 ms oleg357-client.virtnet: 1 120948 2.76 MB/sec execute 173 sec latency 16.351 ms oleg357-client.virtnet: 1 121337 2.74 MB/sec execute 174 sec latency 16.682 ms oleg357-client.virtnet: 1 121958 2.75 MB/sec execute 175 sec latency 15.267 ms oleg357-client.virtnet: 1 122481 2.74 MB/sec execute 176 sec latency 15.417 ms oleg357-client.virtnet: 1 123400 2.76 MB/sec execute 177 sec latency 9.050 ms oleg357-client.virtnet: 1 124437 2.79 MB/sec execute 178 sec latency 9.427 ms oleg357-client.virtnet: 1 124972 2.78 MB/sec execute 179 sec latency 11.738 ms oleg357-client.virtnet: 1 125698 2.79 MB/sec execute 180 sec latency 8.756 ms oleg357-client.virtnet: 1 126530 2.78 MB/sec execute 181 sec latency 8.560 ms oleg357-client.virtnet: 1 127674 2.82 MB/sec execute 182 sec latency 8.877 ms oleg357-client.virtnet: 1 128238 2.82 MB/sec execute 183 sec latency 9.066 ms oleg357-client.virtnet: 1 128955 2.81 MB/sec execute 184 sec latency 9.538 ms oleg357-client.virtnet: 1 129743 2.82 MB/sec execute 185 sec latency 8.967 ms oleg357-client.virtnet: 1 130524 2.83 MB/sec execute 186 sec latency 13.499 ms oleg357-client.virtnet: 1 131381 2.85 MB/sec execute 187 sec latency 11.180 ms oleg357-client.virtnet: 1 131746 2.84 MB/sec execute 188 sec latency 14.822 ms oleg357-client.virtnet: 1 132418 2.84 MB/sec execute 189 sec latency 14.934 ms oleg357-client.virtnet: 1 133137 2.84 MB/sec execute 190 sec latency 11.259 ms oleg357-client.virtnet: 1 133769 2.84 MB/sec execute 191 sec latency 12.262 ms oleg357-client.virtnet: 1 134841 2.88 MB/sec execute 192 sec latency 10.525 ms oleg357-client.virtnet: 1 135332 2.87 MB/sec execute 193 sec latency 11.351 ms oleg357-client.virtnet: 1 135974 2.86 MB/sec execute 194 sec latency 10.628 ms oleg357-client.virtnet: 1 136613 2.87 MB/sec execute 195 sec latency 12.937 ms oleg357-client.virtnet: 1 137403 2.87 MB/sec execute 196 sec latency 7.903 ms oleg357-client.virtnet: 1 137986 2.88 MB/sec execute 197 sec latency 579.260 ms oleg357-client.virtnet: 1 137986 2.86 MB/sec execute 198 sec latency 1579.454 ms oleg357-client.virtnet: 1 137986 2.85 MB/sec eFailover mds2 to oleg357-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36424 3569756 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12792 3593076 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49216 7162832 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg357-server Stopping /mnt/lustre-mds1 (opts:) on oleg357-server reboot facets: mds1 Failover mds1 to oleg357-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 xecute 199 sec latency 2579.645 ms oleg357-client.virtnet: 1 137986 2.83 MB/sec execute 200 sec latency 3579.930 ms oleg357-client.virtnet: 1 137986 2.82 MB/sec execute 201 sec latency 4580.103 ms oleg357-client.virtnet: 1 137986 2.81 MB/sec execute 202 sec latency 5580.351 ms oleg357-client.virtnet: 1 137986 2.79 MB/sec execute 203 sec latency 6580.579 ms oleg357-client.virtnet: 1 137986 2.78 MB/sec execute 204 sec latency 7580.726 ms oleg357-client.virtnet: 1 137986 2.77 MB/sec execute 205 sec latency 8580.915 ms oleg357-client.virtnet: 1 137986 2.75 MB/sec execute 206 sec latency 9581.047 ms oleg357-client.virtnet: 1 137986 2.74 MB/sec execute 207 sec latency 10581.168 ms oleg357-client.virtnet: 1 137986 2.73 MB/sec execute 208 sec latency 11581.348 ms oleg357-client.virtnet: 1 137986 2.71 MB/sec execute 209 sec latency 12581.535 ms oleg357-client.virtnet: 1 137986 2.70 MB/sec execute 210 sec latency 13581.708 ms oleg357-client.virtnet: 1 137986 2.69 MB/sec execute 211 sec latency 14581.896 ms oleg357-client.virtnet: 1 137986 2.67 MB/sec execute 212 sec latency 15582.086 ms oleg357-client.virtnet: 1 137986 2.66 MB/sec execute 213 sec latency 16582.323 ms oleg357-client.virtnet: 1 137986 2.65 MB/sec execute 214 sec latency 17582.518 ms oleg357-client.virtnet: 1 137986 2.64 MB/sec execute 215 sec latency 18582.727 ms oleg357-client.virtnet: 1 138506 2.65 MB/sec execute 216 sec latency 19002.379 ms oleg357-client.virtnet: 1 139053 2.64 MB/sec execute 217 sec latency 14.895 ms oleg357-client.virtnet: 1 139802 2.65 MB/sec execute 218 sec latency 9.632 ms oleg357-client.virtnet: 1 140515 2.64 MB/sec execute 219 sec latency 9.430 ms oleg357-client.virtnet: 1 141541 2.66 MB/sec execute 220 sec latency 10.586 ms oleg357-client.virtnet: 1 142314 2.68 MB/sec execute 221 sec latency 9.208 ms oleg357-client.virtnet: 1 142896 2.67 MB/sec execute 222 sec latency 10.679 ms oleg357-client.virtnet: 1 143604 2.68 MB/sec execute 223 sec latency 9.165 ms oleg357-client.virtnet: 1 144318 2.67 MB/sec execute 224 sec latency 14.176 ms oleg357-client.virtnet: 1 145087 2.69 MB/sec execute 225 sec latency 15.365 ms oleg357-client.virtnet: 1 145772 2.70 MB/sec execute 226 sec latency 11.614 ms oleg357-client.virtnet: 1 146418 2.70 MB/sec execute 227 sec latency 10.080 ms oleg357-client.virtnet: 1 146990 2.70 MB/sec execute 228 sec latency 16.915 ms oleg357-client.virtnet: 1 147390 2.69 MB/sec execute 229 sec latency 14.724 ms oleg357-client.virtnet: 1 148254 2.70 MB/sec execute 230 sec latency 9.426 ms oleg357-client.virtnet: 1 149293 2.73 MB/sec execute 231 sec latency 8.460 ms oleg357-client.virtnet: 1 149932 2.72 MB/sec execute 232 sec latency 9.792 ms oleg357-client.virtnet: 1 150733 2.73 MB/sec execute 233 sec latency 8.286 ms oleg357-client.virtnet: 1 151577 2.73 MB/sec execute 234 sec latency 8.027 ms oleg357-client.virtnet: 1 152751 2.76 MB/sec execute 235 sec latency 8.157 ms oleg357-client.virtnet: 1 153341 2.75 MB/sec execute 236 sec latency 9.726 ms oleg357-client.virtnet: 1 154130 2.76 MB/sec execute 237 sec latency 8.006 ms oleg357-client.virtnet: 1 154909 2.76 MB/sec execute 238 sec latency 11.201 ms oleg357-client.virtnet: 1 156071 2.79 MB/sec execute 239 sec latency 8.452 ms oleg357-client.virtnet: 1 156649 2.79 MB/sec execute 240 sec latency 8.734 ms oleg357-client.virtnet: 1 157368 2.78 MB/sec execute 241 sec latency 9.348 ms oleg357-client.virtnet: 1 158192 2.79 MB/sec execute 242 sec latency 9.012 ms oleg357-client.virtnet: 1 159266 2.80 MB/sec execute 243 sec latency 8.379 ms oleg357-client.virtnet: 1 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 160056 2.82 MB/sec execute 244 sec latency 8.968 ms oleg357-client.virtnet: 1 160690 2.81 MB/sec execute 245 sec latency 11.622 ms oleg357-client.virtnet: 1 161398 2.82 MB/sec execute 246 sec latency 11.873 ms oleg357-client.virtnet: 1 161975 2.81 MB/sec execute 247 sec latency 10.010 ms oleg357-client.virtnet: 1 162616 2.82 MB/sec execute 248 sec latency 16.422 ms oleg357-client.virtnet: 1 163400 2.84 MB/sec execute 249 sec latency 16.043 ms oleg357-client.virtnet: 1 163986 2.83 MB/sec execute 250 sec latency 8.538 ms oleg357-client.virtnet: 1 164434 2.82 MB/sec execute 251 sec latency 17.222 ms oleg357-client.virtnet: 1 165009 2.83 MB/sec execute 252 sec latency 17.579 ms oleg357-client.virtnet: 1 165602 2.82 MB/sec execute 253 sec latency 14.913 ms oleg357-client.virtnet: 1 166184 2.83 MB/sec execute 254 sec latency 16.977 ms oleg357-client.virtnet: 1 166885 2.84 MB/sec execute 255 sec latency 16.388 ms oleg357-client.virtnet: 1 167226 2.84 MB/sec execute 256 sec latency 17.224 ms oleg357-client.virtnet: 1 167885 2.83 MB/sec execute 257 sec latency 12.181 ms oleg357-client.virtnet: 1 168679 2.83 MB/sec execute 258 sec latency 9.592 ms oleg357-client.virtnet: 1 169298 2.83 MB/sec execute 259 sec latency 15.133 ms oleg357-client.virtnet: 1 170242 2.86 MB/sec execute 260 sec latency 15.957 ms oleg357-client.virtnet: 1 170758 2.85 MB/sec execute 261 sec latency 13.605 ms oleg357-client.virtnet: 1 171331 2.85 MB/sec execute 262 sec latency 12.545 ms oleg357-client.virtnet: 1 171845 2.85 MB/sec execute 263 sec latency 15.603 ms oleg357-client.virtnet: 1 172479 2.84 MB/sec execute 264 sec latency 9.040 ms oleg357-client.virtnet: 1 173206 2.86 MB/sec execute 265 sec latency 13.433 ms oleg357-client.virtnet: 1 174168 2.87 MB/sec execute 266 sec latency 8.219 ms oleg357-client.virtnet: 1 174632 2.86 MB/sec execute 267 sec latency 16.322 ms oleg357-client.virtnet: 1 175147 2.86 MB/sec execute 268 sec latency 12.162 ms oleg357-client.virtnet: 1 175931 2.86 MB/sec execute 269 sec latency 8.004 ms oleg357-client.virtnet: 1 176707 2.87 MB/sec execute 270 sec latency 11.238 ms oleg357-client.virtnet: 1 177529 2.88 MB/sec execute 271 sec latency 16.782 ms oleg357-client.virtnet: 1 178103 2.88 MB/sec execute 272 sec latency 11.588 ms oleg357-client.virtnet: 1 178851 2.89 MB/sec execute 273 sec latency 10.530 ms oleg357-client.virtnet: 1 179695 2.88 MB/sec execute 274 sec latency 8.583 ms oleg357-client.virtnet: 1 180538 2.90 MB/sec execute 275 sec latency 12.293 ms oleg357-client.virtnet: 1 181313 2.91 MB/sec execute 276 sec latency 8.553 ms oleg357-client.virtnet: 1 181972 2.90 MB/sec execute 277 sec latency 10.530 ms oleg357-client.virtnet: 1 182616 2.90 MB/sec execute 278 sec latency 11.470 ms oleg357-client.virtnet: 1 183342 2.90 MB/sec execute 279 sec latency 10.863 ms oleg357-client.virtnet: 1 184300 2.92 MB/sec execute 280 sec latency 10.489 ms oleg357-client.virtnet: 1 184960 2.92 MB/sec execute 281 sec latency 8.485 ms oleg357-client.virtnet: 1 185665 2.92 MB/sec execute 282 sec latency 10.740 ms oleg357-client.virtnet: 1 186435 2.92 MB/sec execute 283 sec latency 9.909 ms oleg357-client.virtnet: 1 187430 2.94 MB/sec execute 284 sec latency 7.802 ms oleg357-client.virtnet: 1 188225 2.95 MB/sec execute 285 sec latency 15.019 ms oleg357-client.virtnet: 1 188608 2.94 MB/sec execute 286 sec latency 17.269 ms oleg357-client.virtnet: 1 189257 2.94 MB/sec execute 287 sec latency 9.669 ms oleg357-client.virtnet: 1 190078 2.94 MB/sec execute 288 sec latency 9.550 ms oleg357-client.virtnet: 1 191192 2.95 MB/sec execute 289 sec latency 8.310 ms oleg357-client.virtnet: 1 191935 2.96 MB/sec execute 290 sec latency 9.425 ms oleg357-client.virtnet: 1 192524 2.96 MB/sec execute 291 sec latency 10.316 ms oleg357-client.virtnet: 1 193326 2.96 MB/sec execute 292 sec latency 8.403 ms oleg357-client.virtnet: 1 194109 2.96 MB/sec execute 293 sec latency 10.114 ms oleg357-client.virtnet: 1 195207 2.98 MB/sec execute 294 sec latency 11.554 ms oleg357-client.virtnet: 1 195812 2.98 MB/sec execute 295 sec latency 16.652 ms oleg357-client.virtnet: 1 196576 2.99 MB/sec execute 296 sec latency 8.860 ms oleg357-client.virtnet: 1 197392 2.98 MB/sec execute 297 sec latency 12.594 ms oleg357-client.virtnet: 1 198457 3.00 MB/sec execute 298 sec latency 8.551 ms oleg357-client.virtnet: 1 199120 3.00 MB/sec execute 299 sec latency 8.940 ms oleg357-client.virtnet: 1 cleanup 300 sec oleg357-client.virtnet: 0 cleanup 300 sec oleg357-client.virtnet: oleg357-client.virtnet: Operation Count AvgLat MaxLat oleg357-client.virtnet: ---------------------------------------- oleg357-client.virtnet: NTCreateX 28863 6.242 18789.815 oleg357-client.virtnet: Close 21199 0.875 5.495 oleg357-client.virtnet: Rename 1227 6.651 17.819 oleg357-client.virtnet: Unlink 5834 1.566 5.329 oleg357-client.virtnet: Qpathinfo 26196 1.866 19002.365 oleg357-client.virtnet: Qfileinfo 4565 0.111 2.125 oleg357-client.virtnet: Qfsinfo 4807 0.112 9.016 oleg357-client.virtnet: Sfileinfo 2362 2.868 8.436 oleg357-client.virtnet: Find 10123 0.318 7.305 oleg357-client.virtnet: WriteX 14279 0.858 7.053 oleg357-client.virtnet: ReadX 45280 0.011 0.763 oleg357-client.virtnet: LockX 94 0.993 1.845 oleg357-client.virtnet: UnlockX 94 1.003 2.152 oleg357-client.virtnet: Flush 2023 5.043 398.258 oleg357-client.virtnet: oleg357-client.virtnet: Throughput 3.00273 MB/sec 1 clients 1 procs max_latency=19002.379 ms oleg357-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg357-client.virtnet at Fri Jul 19 13:23:41 EDT 2024 with return code 0 oleg357-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg357-client.virtnet oleg357-client.virtnet: /mnt/lustre/d70b.replay-single/oleg357-client.virtnet /mnt/lustre/d70b.replay-single/oleg357-client.virtnet oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg357-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg357-client.virtnet: removed directory: 'clients/client0' oleg357-client.virtnet: removed directory: 'clients' oleg357-client.virtnet: removed 'client.txt' oleg357-client.virtnet: /mnt/lustre/d70b.replay-single/oleg357-client.virtnet oleg357-client.virtnet: dbench successfully finished