== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 09:42:58 (1747402978) Starting client oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre Started clients oleg227-client.virtnet: 192.168.202.127@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg227-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=19810 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2280 1285408 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1808 1285880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1608 3593992 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26000 3581020 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27608 7175012 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg227-server Stopping /mnt/lustre-mds1 (opts:) on oleg227-server reboot facets: mds1 Failover mds1 to oleg227-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1285448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1764 1285924 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14180 3592012 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37428 3569028 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51608 7161040 1% /mnt/lustre oleg227-client.virtnet: looking for dbench program oleg227-client.virtnet: /usr/bin/dbench oleg227-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg227-client.virtnet oleg227-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg227-client.virtnet' oleg227-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg227-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg227-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg227-client.virtnet at Fri May 16 09:42:59 EDT 2025 oleg227-client.virtnet: waiting for dbench pid 19834 oleg227-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg227-client.virtnet: oleg227-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg227-client.virtnet: failed to create barrier semaphore oleg227-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg227-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg227-client.virtnet: releasing clients oleg227-client.virtnet: 1 493 17.21 MB/sec warmup 1 sec latency 12.550 ms oleg227-client.virtnet: 1 937 10.89 MB/sec warmup 2 sec latency 16.832 ms oleg227-client.virtnet: 1 1481 8.68 MB/sec warmup 3 sec latency 15.268 ms oleg227-client.virtnet: 1 1884 6.59 MB/sec warmup 4 sec latency 13.750 ms oleg227-client.virtnet: 1 2337 5.57 MB/sec warmup 5 sec latency 96.319 ms oleg227-client.virtnet: 1 3018 5.59 MB/sec warmup 6 sec latency 14.955 ms oleg227-client.virtnet: 1 3525 5.48 MB/sec warmup 7 sec latency 610.945 ms oleg227-client.virtnet: 1 3525 4.80 MB/sec warmup 8 sec latency 1611.147 ms oleg227-client.virtnet: 1 3525 4.26 MB/sec warmup 9 sec latency 2611.338 ms oleg227-client.virtnet: 1 3525 3.84 MB/sec warmup 10 sec latency 3611.588 ms oleg227-client.virtnet: 1 3525 3.49 MB/sec warmup 11 sec latency 4611.803 ms oleg227-client.virtnet: 1 3525 3.20 MB/sec warmup 12 sec latency 5612.013 ms oleg227-client.virtnet: 1 3525 2.95 MB/sec warmup 13 sec latency 6612.299 ms oleg227-client.virtnet: 1 3525 2.74 MB/sec warmup 14 sec latency 7612.538 ms oleg227-client.virtnet: 1 3525 2.56 MB/sec warmup 15 sec latency 8612.798 ms oleg227-client.virtnet: 1 3525 2.40 MB/sec warmup 16 sec latency 9613.097 ms oleg227-client.virtnet: 1 3525 2.26 MB/sec warmup 17 sec latency 10613.369 ms oleg227-client.virtnet: 1 3525 2.13 MB/sec warmup 18 sec latency 11613.602 ms oleg227-client.virtnet: 1 3525 2.02 MB/sec warmup 19 sec latency 12613.806 ms oleg227-client.virtnet: 1 3525 1.92 MB/sec warmup 20 sec latency 13614.046 ms oleg227-client.virtnet: 1 3525 1.83 MB/sec warmup 21 sec latency 14614.258 ms oleg227-client.virtnet: 1 3525 1.74 MB/sec warmup 22 sec latency 15614.441 ms oleg227-client.virtnet: 1 3525 1.67 MB/sec warmup 23 sec latency 16614.686 ms oleg227-client.virtnet: 1 3525 1.60 MB/sec warmup 24 sec latency 17614.965 ms oleg227-client.virtnet: 1 3525 1.53 MB/sec warmup 25 sec latency 18615.275 ms oleg227-client.virtnet: 1 3525 1.48 MB/sec warmup 26 sec latency 19615.625 ms oleg227-client.virtnet: 1 3525 1.42 MB/sec warmup 27 sec latency 20615.943 ms oleg227-client.virtnet: 1 3797 1.42 MB/sec warmup 28 sec latency 21201.384 ms oleg227-client.virtnet: 1 4338 1.40 MB/sec warmup 29 sec latency 11.167 ms oleg227-client.virtnet: 1 5031 1.50 MB/sec warmup 30 sec latency 9.815 ms oleg227-client.virtnet: 1 5670 1.48 MB/sec warmup 31 sec latency 12.401 ms oleg227-client.virtnet: 1 6481 1.64 MB/sec warmup 32 sec latency 10.512 ms oleg227-client.virtnet: 1 7317 1.78 MB/sec warmup 33 sec latetest_70b fail mds2 2 times Failing mds2 on oleg227-server Stopping /mnt/lustre-mds2 (opts:) on oleg227-server reboot facets: mds2 Failover mds2 to oleg227-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11912 3594116 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37440 3566648 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49352 7160764 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg227-server Stopping /mnt/lustre-mds1 (opts:) on oleg227-server reboot facets: mds1 Failover mds1 to oleg227-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 ncy 15.283 ms oleg227-client.virtnet: 1 7834 1.75 MB/sec warmup 34 sec latency 14.425 ms oleg227-client.virtnet: 1 8514 1.84 MB/sec warmup 35 sec latency 10.185 ms oleg227-client.virtnet: 1 9204 1.80 MB/sec warmup 36 sec latency 12.184 ms oleg227-client.virtnet: 1 10092 1.94 MB/sec warmup 37 sec latency 11.063 ms oleg227-client.virtnet: 1 10871 2.05 MB/sec warmup 38 sec latency 11.761 ms oleg227-client.virtnet: 1 11386 2.02 MB/sec warmup 39 sec latency 10.931 ms oleg227-client.virtnet: 1 12107 2.08 MB/sec warmup 40 sec latency 9.558 ms oleg227-client.virtnet: 1 12819 2.05 MB/sec warmup 41 sec latency 9.142 ms oleg227-client.virtnet: 1 13840 2.18 MB/sec warmup 42 sec latency 11.577 ms oleg227-client.virtnet: 1 14591 2.27 MB/sec warmup 43 sec latency 9.534 ms oleg227-client.virtnet: 1 15174 2.23 MB/sec warmup 44 sec latency 11.621 ms oleg227-client.virtnet: 1 15890 2.28 MB/sec warmup 45 sec latency 8.906 ms oleg227-client.virtnet: 1 16673 2.29 MB/sec warmup 46 sec latency 8.615 ms oleg227-client.virtnet: 1 17857 2.45 MB/sec warmup 47 sec latency 8.508 ms oleg227-client.virtnet: 1 18411 2.43 MB/sec warmup 48 sec latency 8.827 ms oleg227-client.virtnet: 1 18832 2.42 MB/sec warmup 49 sec latency 15.457 ms oleg227-client.virtnet: 1 19406 2.44 MB/sec warmup 50 sec latency 15.519 ms oleg227-client.virtnet: 1 19630 2.39 MB/sec warmup 51 sec latency 642.960 ms oleg227-client.virtnet: 1 19630 2.35 MB/sec warmup 52 sec latency 1643.244 ms oleg227-client.virtnet: 1 19630 2.30 MB/sec warmup 53 sec latency 2643.569 ms oleg227-client.virtnet: 1 19630 2.26 MB/sec warmup 54 sec latency 3643.821 ms oleg227-client.virtnet: 1 20287 2.26 MB/sec warmup 55 sec latency 3747.074 ms oleg227-client.virtnet: 1 21387 2.40 MB/sec warmup 56 sec latency 12.084 ms oleg227-client.virtnet: 1 21723 2.38 MB/sec warmup 57 sec latency 17.357 ms oleg227-client.virtnet: 1 22114 2.35 MB/sec warmup 58 sec latency 15.861 ms oleg227-client.virtnet: 1 22700 2.38 MB/sec warmup 59 sec latency 14.427 ms oleg227-client.virtnet: 1 23793 2.25 MB/sec execute 1 sec latency 212.978 ms oleg227-client.virtnet: 1 24944 6.12 MB/sec execute 2 sec latency 8.890 ms oleg227-client.virtnet: 1 25099 4.47 MB/sec execute 3 sec latency 743.502 ms oleg227-client.virtnet: 1 25099 3.35 MB/sec execute 4 sec latency 1743.729 ms oleg227-client.virtnet: 1 25099 2.68 MB/sec execute 5 sec latency 2743.966 ms oleg227-client.virtnet: 1 25099 2.23 MB/sec execute 6 sec latency 3744.215 ms oleg227-client.virtnet: 1 25099 1.91 MB/sec execute 7 sec latency 4744.415 ms oleg227-client.virtnet: 1 25099 1.67 MB/sec execute 8 sec latency 5744.642 ms oleg227-client.virtnet: 1 25099 1.49 MB/sec execute 9 sec latency 6744.879 ms oleg227-client.virtnet: 1 25099 1.34 MB/sec execute 10 sec latency 7745.095 ms oleg227-client.virtnet: 1 25099 1.22 MB/sec execute 11 sec latency 8745.337 ms oleg227-client.virtnet: 1 25099 1.12 MB/sec execute 12 sec latency 9745.613 ms oleg227-client.virtnet: 1 25099 1.03 MB/sec execute 13 sec latency 10745.776 ms oleg227-client.virtnet: 1 25099 0.96 MB/sec execute 14 sec latency 11745.946 ms oleg227-client.virtnet: 1 25099 0.89 MB/sec execute 15 sec latency 12746.150 ms oleg227-client.virtnet: 1 25099 0.84 MB/sec execute 16 sec latency 13746.360 ms oleg227-client.virtnet: 1 25099 0.79 MB/sec execute 17 sec latency 14746.508 ms oleg227-client.virtnet: 1 25099 0.74 MB/sec execute 18 sec latency 15746.745 ms oleg227-client.virtnet: 1 25099 0.71 MB/sec execute oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2204 1285484 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11892 3594092 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37428 3566936 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49320 7161028 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg227-server Stopping /mnt/lustre-mds2 (opts:) on oleg227-server reboot facets: mds2 Failover mds2 to oleg227-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1285444 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11776 3593944 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37436 3566280 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49212 7160224 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg227-server Stopping /mnt/lustre-mds1 (opts:) on oleg227-server reboot facets: mds1 19 sec latency 16746.890 ms oleg227-client.virtnet: 1 25099 0.67 MB/sec execute 20 sec latency 17746.979 ms oleg227-client.virtnet: 1 25099 0.64 MB/sec execute 21 sec latency 18747.178 ms oleg227-client.virtnet: 1 25099 0.61 MB/sec execute 22 sec latency 19747.411 ms oleg227-client.virtnet: 1 25329 0.60 MB/sec execute 23 sec latency 20194.079 ms oleg227-client.virtnet: 1 25874 0.61 MB/sec execute 24 sec latency 15.855 ms oleg227-client.virtnet: 1 26594 0.76 MB/sec execute 25 sec latency 9.092 ms oleg227-client.virtnet: 1 27329 0.80 MB/sec execute 26 sec latency 8.834 ms oleg227-client.virtnet: 1 28443 1.15 MB/sec execute 27 sec latency 9.210 ms oleg227-client.virtnet: 1 28948 1.17 MB/sec execute 28 sec latency 10.449 ms oleg227-client.virtnet: 1 29625 1.20 MB/sec execute 29 sec latency 9.821 ms oleg227-client.virtnet: 1 30388 1.28 MB/sec execute 30 sec latency 14.236 ms oleg227-client.virtnet: 1 31288 1.42 MB/sec execute 31 sec latency 10.488 ms oleg227-client.virtnet: 1 32242 1.61 MB/sec execute 32 sec latency 8.216 ms oleg227-client.virtnet: 1 32787 1.59 MB/sec execute 33 sec latency 11.823 ms oleg227-client.virtnet: 1 33534 1.68 MB/sec execute 34 sec latency 8.806 ms oleg227-client.virtnet: 1 34319 1.68 MB/sec execute 35 sec latency 8.627 ms oleg227-client.virtnet: 1 35341 1.85 MB/sec execute 36 sec latency 10.085 ms oleg227-client.virtnet: 1 36014 1.92 MB/sec execute 37 sec latency 8.854 ms oleg227-client.virtnet: 1 36695 1.92 MB/sec execute 38 sec latency 12.101 ms oleg227-client.virtnet: 1 37346 1.96 MB/sec execute 39 sec latency 11.399 ms oleg227-client.virtnet: 1 38266 2.05 MB/sec execute 40 sec latency 8.021 ms oleg227-client.virtnet: 1 39296 2.19 MB/sec execute 41 sec latency 8.577 ms oleg227-client.virtnet: 1 39873 2.16 MB/sec execute 42 sec latency 9.128 ms oleg227-client.virtnet: 1 40626 2.22 MB/sec execute 43 sec latency 11.018 ms oleg227-client.virtnet: 1 41417 2.21 MB/sec execute 44 sec latency 9.297 ms oleg227-client.virtnet: 1 42560 2.39 MB/sec execute 45 sec latency 8.091 ms oleg227-client.virtnet: 1 42712 2.35 MB/sec execute 46 sec latency 777.448 ms oleg227-client.virtnet: 1 42712 2.30 MB/sec execute 47 sec latency 1777.627 ms oleg227-client.virtnet: 1 42712 2.25 MB/sec execute 48 sec latency 2777.812 ms oleg227-client.virtnet: 1 42974 2.23 MB/sec execute 49 sec latency 3303.553 ms oleg227-client.virtnet: 1 43578 2.20 MB/sec execute 50 sec latency 10.752 ms oleg227-client.virtnet: 1 44318 2.25 MB/sec execute 51 sec latency 10.073 ms oleg227-client.virtnet: 1 45052 2.24 MB/sec execute 52 sec latency 8.319 ms oleg227-client.virtnet: 1 46216 2.40 MB/sec execute 53 sec latency 8.673 ms oleg227-client.virtnet: 1 46749 2.38 MB/sec execute 54 sec latency 10.410 ms oleg227-client.virtnet: 1 47352 2.37 MB/sec execute 55 sec latency 108.103 ms oleg227-client.virtnet: 1 48091 2.39 MB/sec execute 56 sec latency 14.651 ms oleg227-client.virtnet: 1 48318 2.35 MB/sec execute 57 sec latency 725.068 ms oleg227-client.virtnet: 1 48318 2.31 MB/sec execute 58 sec latency 1725.245 ms oleg227-client.virtnet: 1 48318 2.27 MB/sec execute 59 sec latency 2725.394 ms oleg227-client.virtnet: 1 48318 2.24 MB/sec execute 60 sec latency 3725.666 ms oleg227-client.virtnet: 1 48318 2.20 MB/sec execute 61 sec latency 4725.915 ms oleg227-client.virtnet: 1 48318 2.16 MB/sec execute 62 sec latency 5726.108 ms oleg227-client.virtnet: 1 48318 2.13 MB/sec execute 63 sec latency 6726.296 ms oleg227-client.virtnet: 1 48318 2.Failover mds1 to oleg227-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2204 1285484 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11792 3593956 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37268 3569336 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49060 7163292 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg227-server Stopping /mnt/lustre-mds2 (opts:) on oleg227-server reboot facets: mds2 Failover mds2 to oleg227-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1285444 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12084 3593552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37456 3568976 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49540 7162528 1% /mnt/lustre 10 MB/sec execute 64 sec latency 7726.494 ms oleg227-client.virtnet: 1 48318 2.06 MB/sec execute 65 sec latency 8726.717 ms oleg227-client.virtnet: 1 48318 2.03 MB/sec execute 66 sec latency 9726.979 ms oleg227-client.virtnet: 1 48318 2.00 MB/sec execute 67 sec latency 10727.148 ms oleg227-client.virtnet: 1 48318 1.97 MB/sec execute 68 sec latency 11727.258 ms oleg227-client.virtnet: 1 48318 1.94 MB/sec execute 69 sec latency 12727.358 ms oleg227-client.virtnet: 1 48318 1.92 MB/sec execute 70 sec latency 13727.605 ms oleg227-client.virtnet: 1 48318 1.89 MB/sec execute 71 sec latency 14727.782 ms oleg227-client.virtnet: 1 48318 1.86 MB/sec execute 72 sec latency 15727.978 ms oleg227-client.virtnet: 1 48318 1.84 MB/sec execute 73 sec latency 16728.135 ms oleg227-client.virtnet: 1 48318 1.81 MB/sec execute 74 sec latency 17728.351 ms oleg227-client.virtnet: 1 48318 1.79 MB/sec execute 75 sec latency 18728.609 ms oleg227-client.virtnet: 1 48318 1.77 MB/sec execute 76 sec latency 19728.749 ms oleg227-client.virtnet: 1 48347 1.74 MB/sec execute 77 sec latency 20685.323 ms oleg227-client.virtnet: 1 49169 1.81 MB/sec execute 78 sec latency 9.206 ms oleg227-client.virtnet: 1 50030 1.87 MB/sec execute 79 sec latency 10.622 ms oleg227-client.virtnet: 1 50515 1.85 MB/sec execute 80 sec latency 11.790 ms oleg227-client.virtnet: 1 51270 1.89 MB/sec execute 81 sec latency 9.145 ms oleg227-client.virtnet: 1 52068 1.88 MB/sec execute 82 sec latency 8.789 ms oleg227-client.virtnet: 1 53202 1.99 MB/sec execute 83 sec latency 8.779 ms oleg227-client.virtnet: 1 53756 1.98 MB/sec execute 84 sec latency 8.747 ms oleg227-client.virtnet: 1 54467 1.99 MB/sec execute 85 sec latency 9.303 ms oleg227-client.virtnet: 1 55252 2.00 MB/sec execute 86 sec latency 9.009 ms oleg227-client.virtnet: 1 56222 2.06 MB/sec execute 87 sec latency 8.411 ms oleg227-client.virtnet: 1 57121 2.11 MB/sec execute 88 sec latency 9.014 ms oleg227-client.virtnet: 1 57760 2.10 MB/sec execute 89 sec latency 10.394 ms oleg227-client.virtnet: 1 58569 2.12 MB/sec execute 90 sec latency 8.649 ms oleg227-client.virtnet: 1 59376 2.13 MB/sec execute 91 sec latency 7.823 ms oleg227-client.virtnet: 1 60540 2.22 MB/sec execute 92 sec latency 8.639 ms oleg227-client.virtnet: 1 61120 2.21 MB/sec execute 93 sec latency 9.372 ms oleg227-client.virtnet: 1 61889 2.23 MB/sec execute 94 sec latency 8.973 ms oleg227-client.virtnet: 1 62689 2.23 MB/sec execute 95 sec latency 8.667 ms oleg227-client.virtnet: 1 63756 2.30 MB/sec execute 96 sec latency 10.658 ms oleg227-client.virtnet: 1 64368 2.31 MB/sec execute 97 sec latency 9.067 ms oleg227-client.virtnet: 1 65071 2.31 MB/sec execute 98 sec latency 9.355 ms oleg227-client.virtnet: 1 65765 2.32 MB/sec execute 99 sec latency 74.003 ms oleg227-client.virtnet: 1 65765 2.29 MB/sec execute 100 sec latency 1074.162 ms oleg227-client.virtnet: 1 65765 2.27 MB/sec execute 101 sec latency 2074.315 ms oleg227-client.virtnet: 1 65765 2.25 MB/sec execute 102 sec latency 3074.475 ms oleg227-client.virtnet: 1 65933 2.23 MB/sec execute 103 sec latency 3896.774 ms oleg227-client.virtnet: 1 66861 2.27 MB/sec execute 104 sec latency 10.484 ms oleg227-client.virtnet: 1 67745 2.31 MB/sec execute 105 sec latency 9.346 ms oleg227-client.virtnet: 1 68355 2.30 MB/sec execute 106 sec latency 13.938 ms oleg227-client.virtnet: 1 69112 2.32 MB/sec execute 107 sec latency 9.084 ms oleg227-client.virtnet: 1 69837 2.32 MB/sec execute 108 sec latency 54.035 ms oleg227-client.virtest_70b fail mds1 7 times Failing mds1 on oleg227-server Stopping /mnt/lustre-mds1 (opts:) on oleg227-server reboot facets: mds1 Failover mds1 to oleg227-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2196 1285492 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12016 3594064 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37276 3569264 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49292 7163328 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg227-server Stopping /mnt/lustre-mds2 (opts:) on oleg227-server reboot facets: mds2 Failover mds2 to oleg227-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 tnet: 1 70443 2.35 MB/sec execute 109 sec latency 414.826 ms oleg227-client.virtnet: 1 71332 2.38 MB/sec execute 110 sec latency 9.908 ms oleg227-client.virtnet: 1 71587 2.36 MB/sec execute 111 sec latency 561.181 ms oleg227-client.virtnet: 1 71587 2.34 MB/sec execute 112 sec latency 1561.317 ms oleg227-client.virtnet: 1 71587 2.32 MB/sec execute 113 sec latency 2561.471 ms oleg227-client.virtnet: 1 71587 2.30 MB/sec execute 114 sec latency 3561.618 ms oleg227-client.virtnet: 1 71587 2.28 MB/sec execute 115 sec latency 4561.801 ms oleg227-client.virtnet: 1 71587 2.26 MB/sec execute 116 sec latency 5561.981 ms oleg227-client.virtnet: 1 71587 2.24 MB/sec execute 117 sec latency 6562.176 ms oleg227-client.virtnet: 1 71587 2.22 MB/sec execute 118 sec latency 7562.459 ms oleg227-client.virtnet: 1 71587 2.20 MB/sec execute 119 sec latency 8562.637 ms oleg227-client.virtnet: 1 71587 2.19 MB/sec execute 120 sec latency 9562.798 ms oleg227-client.virtnet: 1 71587 2.17 MB/sec execute 121 sec latency 10562.961 ms oleg227-client.virtnet: 1 71587 2.15 MB/sec execute 122 sec latency 11563.216 ms oleg227-client.virtnet: 1 71587 2.13 MB/sec execute 123 sec latency 12563.347 ms oleg227-client.virtnet: 1 71587 2.12 MB/sec execute 124 sec latency 13563.579 ms oleg227-client.virtnet: 1 71587 2.10 MB/sec execute 125 sec latency 14563.778 ms oleg227-client.virtnet: 1 71587 2.08 MB/sec execute 126 sec latency 15563.893 ms oleg227-client.virtnet: 1 71587 2.07 MB/sec execute 127 sec latency 16563.988 ms oleg227-client.virtnet: 1 71587 2.05 MB/sec execute 128 sec latency 17564.225 ms oleg227-client.virtnet: 1 71587 2.03 MB/sec execute 129 sec latency 18564.483 ms oleg227-client.virtnet: 1 71587 2.02 MB/sec execute 130 sec latency 19564.727 ms oleg227-client.virtnet: 1 71587 2.00 MB/sec execute 131 sec latency 20564.892 ms oleg227-client.virtnet: 1 71906 1.99 MB/sec execute 132 sec latency 20880.023 ms oleg227-client.virtnet: 1 72494 2.01 MB/sec execute 133 sec latency 14.112 ms oleg227-client.virtnet: 1 72911 2.00 MB/sec execute 134 sec latency 17.226 ms oleg227-client.virtnet: 1 73367 1.99 MB/sec execute 135 sec latency 12.608 ms oleg227-client.virtnet: 1 74203 2.03 MB/sec execute 136 sec latency 12.343 ms oleg227-client.virtnet: 1 74867 2.05 MB/sec execute 137 sec latency 11.994 ms oleg227-client.virtnet: 1 75346 2.04 MB/sec execute 138 sec latency 12.712 ms oleg227-client.virtnet: 1 75803 2.04 MB/sec execute 139 sec latency 15.792 ms oleg227-client.virtnet: 1 76305 2.05 MB/sec execute 140 sec latency 15.168 ms oleg227-client.virtnet: 1 76710 2.04 MB/sec execute 141 sec latency 11.065 ms oleg227-client.virtnet: 1 77135 2.04 MB/sec execute 142 sec latency 15.251 ms oleg227-client.virtnet: 1 78176 2.09 MB/sec execute 143 sec latency 9.803 ms oleg227-client.virtnet: 1 78624 2.09 MB/sec execute 144 sec latency 11.453 ms oleg227-client.virtnet: 1 79212 2.09 MB/sec execute 145 sec latency 15.700 ms oleg227-client.virtnet: 1 79913 2.10 MB/sec execute 146 sec latency 9.804 ms oleg227-client.virtnet: 1 80588 2.10 MB/sec execute 147 sec latency 9.328 ms oleg227-client.virtnet: 1 81738 2.15 MB/sec execute 148 sec latency 9.039 ms oleg227-client.virtnet: 1 82192 2.15 MB/sec execute 149 sec latency 13.083 ms oleg227-client.virtnet: 1 82814 2.14 MB/sec execute 150 sec latency 11.400 ms oleg227-client.virtnet: 1 83506 2.15 MB/sec execute 151 sec latency 10.376 ms oleg227-client.virtnet: 1 84127 2.15 MB/sec execute 152 sec latency 9.752 ms oleg227-client.virtnet: 1 85268 2.20 MB/sec execoleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11948 3593156 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37456 3569080 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49404 7162236 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg227-server Stopping /mnt/lustre-mds1 (opts:) on oleg227-server reboot facets: mds1 Failover mds1 to oleg227-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2204 1285484 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11936 3594144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37460 3568940 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49396 7163084 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg227-server Stopping /mnt/lustre-mds2 (opts:) on oleg227-server reboot facets: mds2 ute 153 sec latency 9.341 ms oleg227-client.virtnet: 1 85844 2.20 MB/sec execute 154 sec latency 10.199 ms oleg227-client.virtnet: 1 86312 2.20 MB/sec execute 155 sec latency 277.528 ms oleg227-client.virtnet: 1 86312 2.18 MB/sec execute 156 sec latency 1277.749 ms oleg227-client.virtnet: 1 86312 2.17 MB/sec execute 157 sec latency 2277.941 ms oleg227-client.virtnet: 1 86312 2.16 MB/sec execute 158 sec latency 3278.120 ms oleg227-client.virtnet: 1 86801 2.16 MB/sec execute 159 sec latency 3615.991 ms oleg227-client.virtnet: 1 87578 2.16 MB/sec execute 160 sec latency 10.383 ms oleg227-client.virtnet: 1 88743 2.21 MB/sec execute 161 sec latency 8.741 ms oleg227-client.virtnet: 1 89319 2.21 MB/sec execute 162 sec latency 9.897 ms oleg227-client.virtnet: 1 89975 2.21 MB/sec execute 163 sec latency 13.560 ms oleg227-client.virtnet: 1 90394 2.21 MB/sec execute 164 sec latency 431.967 ms oleg227-client.virtnet: 1 91140 2.21 MB/sec execute 165 sec latency 8.818 ms oleg227-client.virtnet: 1 92281 2.26 MB/sec execute 166 sec latency 9.988 ms oleg227-client.virtnet: 1 92331 2.25 MB/sec execute 167 sec latency 950.581 ms oleg227-client.virtnet: 1 92331 2.24 MB/sec execute 168 sec latency 1950.741 ms oleg227-client.virtnet: 1 92331 2.22 MB/sec execute 169 sec latency 2950.915 ms oleg227-client.virtnet: 1 92331 2.21 MB/sec execute 170 sec latency 3951.113 ms oleg227-client.virtnet: 1 92331 2.20 MB/sec execute 171 sec latency 4951.352 ms oleg227-client.virtnet: 1 92331 2.18 MB/sec execute 172 sec latency 5951.542 ms oleg227-client.virtnet: 1 92331 2.17 MB/sec execute 173 sec latency 6951.742 ms oleg227-client.virtnet: 1 92331 2.16 MB/sec execute 174 sec latency 7951.967 ms oleg227-client.virtnet: 1 92331 2.15 MB/sec execute 175 sec latency 8952.181 ms oleg227-client.virtnet: 1 92331 2.13 MB/sec execute 176 sec latency 9952.375 ms oleg227-client.virtnet: 1 92331 2.12 MB/sec execute 177 sec latency 10952.573 ms oleg227-client.virtnet: 1 92331 2.11 MB/sec execute 178 sec latency 11952.685 ms oleg227-client.virtnet: 1 92331 2.10 MB/sec execute 179 sec latency 12952.775 ms oleg227-client.virtnet: 1 92331 2.09 MB/sec execute 180 sec latency 13952.936 ms oleg227-client.virtnet: 1 92331 2.07 MB/sec execute 181 sec latency 14953.113 ms oleg227-client.virtnet: 1 92331 2.06 MB/sec execute 182 sec latency 15953.331 ms oleg227-client.virtnet: 1 92331 2.05 MB/sec execute 183 sec latency 16953.547 ms oleg227-client.virtnet: 1 92331 2.04 MB/sec execute 184 sec latency 17953.664 ms oleg227-client.virtnet: 1 92331 2.03 MB/sec execute 185 sec latency 18953.850 ms oleg227-client.virtnet: 1 92331 2.02 MB/sec execute 186 sec latency 19954.074 ms oleg227-client.virtnet: 1 92518 2.01 MB/sec execute 187 sec latency 20601.610 ms oleg227-client.virtnet: 1 93012 2.01 MB/sec execute 188 sec latency 11.225 ms oleg227-client.virtnet: 1 93741 2.02 MB/sec execute 189 sec latency 9.433 ms oleg227-client.virtnet: 1 94441 2.01 MB/sec execute 190 sec latency 9.423 ms oleg227-client.virtnet: 1 95474 2.04 MB/sec execute 191 sec latency 8.157 ms oleg227-client.virtnet: 1 96206 2.06 MB/sec execute 192 sec latency 9.301 ms oleg227-client.virtnet: 1 96797 2.06 MB/sec execute 193 sec latency 10.951 ms oleg227-client.virtnet: 1 97547 2.07 MB/sec execute 194 sec latency 10.139 ms oleg227-client.virtnet: 1 98286 2.07 MB/sec execute 195 sec latency 16.075 ms oleg227-client.virtnet: 1 99481 2.11 MB/sec execute 196 sec latency 8.628 ms oleg227-client.virtnet: 1 100082 2.11 MB/sec execute 197 sec latency 9.987 ms oleg227-clFailover mds2 to oleg227-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11944 3594048 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37284 3569224 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49228 7163272 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg227-server Stopping /mnt/lustre-mds1 (opts:) on oleg227-server reboot facets: mds1 Failover mds1 to oleg227-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 ient.virtnet: 1 100815 2.12 MB/sec execute 198 sec latency 8.905 ms oleg227-client.virtnet: 1 101554 2.12 MB/sec execute 199 sec latency 8.521 ms oleg227-client.virtnet: 1 102554 2.14 MB/sec execute 200 sec latency 8.172 ms oleg227-client.virtnet: 1 103310 2.16 MB/sec execute 201 sec latency 8.672 ms oleg227-client.virtnet: 1 103959 2.16 MB/sec execute 202 sec latency 9.806 ms oleg227-client.virtnet: 1 104707 2.17 MB/sec execute 203 sec latency 8.805 ms oleg227-client.virtnet: 1 105507 2.17 MB/sec execute 204 sec latency 7.863 ms oleg227-client.virtnet: 1 106655 2.21 MB/sec execute 205 sec latency 8.629 ms oleg227-client.virtnet: 1 107195 2.20 MB/sec execute 206 sec latency 10.205 ms oleg227-client.virtnet: 1 107916 2.22 MB/sec execute 207 sec latency 9.930 ms oleg227-client.virtnet: 1 108633 2.21 MB/sec execute 208 sec latency 9.747 ms oleg227-client.virtnet: 1 109626 2.23 MB/sec execute 209 sec latency 9.513 ms oleg227-client.virtnet: 1 110148 2.24 MB/sec execute 210 sec latency 455.785 ms oleg227-client.virtnet: 1 110148 2.23 MB/sec execute 211 sec latency 1455.974 ms oleg227-client.virtnet: 1 110148 2.22 MB/sec execute 212 sec latency 2456.154 ms oleg227-client.virtnet: 1 110293 2.22 MB/sec execute 213 sec latency 3201.810 ms oleg227-client.virtnet: 1 110885 2.21 MB/sec execute 214 sec latency 10.012 ms oleg227-client.virtnet: 1 111622 2.22 MB/sec execute 215 sec latency 8.877 ms oleg227-client.virtnet: 1 112408 2.22 MB/sec execute 216 sec latency 8.632 ms oleg227-client.virtnet: 1 113558 2.26 MB/sec execute 217 sec latency 9.221 ms oleg227-client.virtnet: 1 114141 2.26 MB/sec execute 218 sec latency 9.521 ms oleg227-client.virtnet: 1 114628 2.25 MB/sec execute 219 sec latency 301.104 ms oleg227-client.virtnet: 1 115347 2.26 MB/sec execute 220 sec latency 9.520 ms oleg227-client.virtnet: 1 115755 2.25 MB/sec execute 221 sec latency 499.137 ms oleg227-client.virtnet: 1 115755 2.24 MB/sec execute 222 sec latency 1499.304 ms oleg227-client.virtnet: 1 115755 2.23 MB/sec execute 223 sec latency 2499.475 ms oleg227-client.virtnet: 1 115755 2.22 MB/sec execute 224 sec latency 3499.638 ms oleg227-client.virtnet: 1 115755 2.21 MB/sec execute 225 sec latency 4499.812 ms oleg227-client.virtnet: 1 115755 2.20 MB/sec execute 226 sec latency 5499.996 ms oleg227-client.virtnet: 1 115755 2.19 MB/sec execute 227 sec latency 6500.176 ms oleg227-client.virtnet: 1 115755 2.18 MB/sec execute 228 sec latency 7500.382 ms oleg227-client.virtnet: 1 115755 2.17 MB/sec execute 229 sec latency 8500.640 ms oleg227-client.virtnet: 1 115755 2.16 MB/sec execute 230 sec latency 9500.841 ms oleg227-client.virtnet: 1 115755 2.15 MB/sec execute 231 sec latency 10501.071 ms oleg227-client.virtnet: 1 115755 2.14 MB/sec execute 232 sec latency 11501.307 ms oleg227-client.virtnet: 1 115755 2.14 MB/sec execute 233 sec latency 12501.484 ms oleg227-client.virtnet: 1 115755 2.13 MB/sec execute 234 sec latency 13501.670 ms oleg227-client.virtnet: 1 115755 2.12 MB/sec execute 235 sec latency 14501.967 ms oleg227-client.virtnet: 1 115755 2.11 MB/sec execute 236 sec latency 15502.155 ms oleg227-client.virtnet: 1 115755 2.10 MB/sec execute 237 sec latency 16502.363 ms oleg227-client.virtnet: 1 115755 2.09 MB/sec execute 238 sec latency 17502.641 ms oleg227-client.virtnet: 1 115755 2.08 MB/sec execute 239 sec latency 18502.911 ms oleg227-client.virtnet: 1 115755 2.07 MB/sec execute 240 sec latency 19503.080 ms oleg227-client.virtnet: 1 115755 2.06 MB/sec execute 241 sec latency 20503.194 ms oleg227-client.virtnet: 1 116477 2.08 MB/sec exoleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec ecute 242 sec latency 20730.124 ms oleg227-client.virtnet: 1 117390 2.10 MB/sec execute 243 sec latency 8.999 ms oleg227-client.virtnet: 1 117982 2.10 MB/sec execute 244 sec latency 12.341 ms oleg227-client.virtnet: 1 118715 2.11 MB/sec execute 245 sec latency 9.317 ms oleg227-client.virtnet: 1 119509 2.11 MB/sec execute 246 sec latency 8.924 ms oleg227-client.virtnet: 1 120665 2.14 MB/sec execute 247 sec latency 8.173 ms oleg227-client.virtnet: 1 121285 2.14 MB/sec execute 248 sec latency 8.951 ms oleg227-client.virtnet: 1 122036 2.15 MB/sec execute 249 sec latency 11.916 ms oleg227-client.virtnet: 1 122760 2.14 MB/sec execute 250 sec latency 9.089 ms oleg227-client.virtnet: 1 123655 2.16 MB/sec execute 251 sec latency 12.540 ms oleg227-client.virtnet: 1 124552 2.18 MB/sec execute 252 sec latency 9.352 ms oleg227-client.virtnet: 1 125141 2.17 MB/sec execute 253 sec latency 12.803 ms oleg227-client.virtnet: 1 125884 2.18 MB/sec execute 254 sec latency 9.092 ms oleg227-client.virtnet: 1 126654 2.18 MB/sec execute 255 sec latency 7.416 ms oleg227-client.virtnet: 1 127814 2.21 MB/sec execute 256 sec latency 8.542 ms oleg227-client.virtnet: 1 128410 2.21 MB/sec execute 257 sec latency 10.244 ms oleg227-client.virtnet: 1 129160 2.22 MB/sec execute 258 sec latency 8.904 ms oleg227-client.virtnet: 1 129908 2.22 MB/sec execute 259 sec latency 8.487 ms oleg227-client.virtnet: 1 131003 2.24 MB/sec execute 260 sec latency 8.241 ms oleg227-client.virtnet: 1 131716 2.25 MB/sec execute 261 sec latency 8.903 ms oleg227-client.virtnet: 1 132397 2.25 MB/sec execute 262 sec latency 10.397 ms oleg227-client.virtnet: 1 133137 2.25 MB/sec execute 263 sec latency 8.850 ms oleg227-client.virtnet: 1 133874 2.26 MB/sec execute 264 sec latency 8.231 ms oleg227-client.virtnet: 1 135014 2.29 MB/sec execute 265 sec latency 8.758 ms oleg227-client.virtnet: 1 135567 2.28 MB/sec execute 266 sec latency 13.512 ms oleg227-client.virtnet: 1 136301 2.29 MB/sec execute 267 sec latency 8.658 ms oleg227-client.virtnet: 1 137033 2.28 MB/sec execute 268 sec latency 8.402 ms oleg227-client.virtnet: 1 138093 2.31 MB/sec execute 269 sec latency 10.490 ms oleg227-client.virtnet: 1 138784 2.32 MB/sec execute 270 sec latency 10.079 ms oleg227-client.virtnet: 1 139405 2.32 MB/sec execute 271 sec latency 11.992 ms oleg227-client.virtnet: 1 140156 2.32 MB/sec execute 272 sec latency 9.222 ms oleg227-client.virtnet: 1 140951 2.32 MB/sec execute 273 sec latency 7.483 ms oleg227-client.virtnet: 1 142076 2.35 MB/sec execute 274 sec latency 8.501 ms oleg227-client.virtnet: 1 142648 2.35 MB/sec execute 275 sec latency 13.087 ms oleg227-client.virtnet: 1 143368 2.35 MB/sec execute 276 sec latency 29.648 ms oleg227-client.virtnet: 1 144115 2.35 MB/sec execute 277 sec latency 9.284 ms oleg227-client.virtnet: 1 145236 2.37 MB/sec execute 278 sec latency 8.644 ms oleg227-client.virtnet: 1 145892 2.38 MB/sec execute 279 sec latency 8.622 ms oleg227-client.virtnet: 1 146595 2.38 MB/sec execute 280 sec latency 8.340 ms oleg227-client.virtnet: 1 147332 2.38 MB/sec execute 281 sec latency 8.814 ms oleg227-client.virtnet: 1 148237 2.39 MB/sec execute 282 sec latency 7.423 ms oleg227-client.virtnet: 1 149265 2.41 MB/sec execute 283 sec latency 9.466 ms oleg227-client.virtnet: 1 149862 2.41 MB/sec execute 284 sec latency 12.308 ms oleg227-client.virtnet: 1 150621 2.41 MB/sec execute 285 sec latency 8.827 ms oleg227-client.virtnet: 1 151403 2.41 MB/sec execute 286 sec latency 8.694 ms oleg227-client.virtnet: 1 152560 2.44 MB/sec execute 287 sec latency 9.042 ms oleg227-client.virtnet: 1 153151 2.44 MB/sec execute 288 sec latency 8.935 ms oleg227-client.virtnet: 1 153824 2.44 MB/sec execute 289 sec latency 11.474 ms oleg227-client.virtnet: 1 154613 2.44 MB/sec execute 290 sec latency 9.007 ms oleg227-client.virtnet: 1 155554 2.46 MB/sec execute 291 sec latency 8.712 ms oleg227-client.virtnet: 1 156454 2.47 MB/sec execute 292 sec latency 9.088 ms oleg227-client.virtnet: 1 157073 2.46 MB/sec execute 293 sec latency 9.648 ms oleg227-client.virtnet: 1 157826 2.47 MB/sec execute 294 sec latency 8.876 ms oleg227-client.virtnet: 1 158593 2.47 MB/sec execute 295 sec latency 7.561 ms oleg227-client.virtnet: 1 159759 2.50 MB/sec execute 296 sec latency 9.461 ms oleg227-client.virtnet: 1 160122 2.49 MB/sec execute 297 sec latency 14.136 ms oleg227-client.virtnet: 1 160703 2.49 MB/sec execute 298 sec latency 10.648 ms oleg227-client.virtnet: 1 161374 2.49 MB/sec execute 299 sec latency 13.962 ms oleg227-client.virtnet: 1 cleanup 300 sec oleg227-client.virtnet: 0 cleanup 300 sec oleg227-client.virtnet: oleg227-client.virtnet: Operation Count AvgLat MaxLat oleg227-client.virtnet: ---------------------------------------- oleg227-client.virtnet: NTCreateX 24020 5.937 20880.010 oleg227-client.virtnet: Close 17669 2.029 20730.102 oleg227-client.virtnet: Rename 1018 9.413 3201.795 oleg227-client.virtnet: Unlink 4850 5.701 20194.056 oleg227-client.virtnet: Qpathinfo 21844 2.532 20601.600 oleg227-client.virtnet: Qfileinfo 3807 0.107 1.318 oleg227-client.virtnet: Qfsinfo 3988 0.154 2.846 oleg227-client.virtnet: Sfileinfo 1965 2.790 8.869 oleg227-client.virtnet: Find 8430 0.317 10.574 oleg227-client.virtnet: WriteX 11900 0.812 10.480 oleg227-client.virtnet: ReadX 37666 0.010 1.091 oleg227-client.virtnet: LockX 78 0.854 1.424 oleg227-client.virtnet: UnlockX 78 0.863 1.416 oleg227-client.virtnet: Flush 1684 5.385 431.923 oleg227-client.virtnet: oleg227-client.virtnet: Throughput 2.49367 MB/sec 1 clients 1 procs max_latency=20880.023 ms oleg227-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg227-client.virtnet at Fri May 16 09:49:00 EDT 2025 with return code 0 oleg227-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg227-client.virtnet oleg227-client.virtnet: /mnt/lustre/d70b.replay-single/oleg227-client.virtnet /mnt/lustre/d70b.replay-single/oleg227-client.virtnet oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg227-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg227-client.virtnet: removed directory: 'clients/client0' oleg227-client.virtnet: removed directory: 'clients' oleg227-client.virtnet: removed 'client.txt' oleg227-client.virtnet: /mnt/lustre/d70b.replay-single/oleg227-client.virtnet oleg227-client.virtnet: dbench successfully finished