== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 12:11:49 (1737133909) Starting client oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre Started clients oleg307-client.virtnet: 192.168.203.107@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg307-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=19205 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26024 3556276 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3594196 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27440 7150472 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg307-server Stopping /mnt/lustre-mds1 (opts:) on oleg307-server reboot facets: mds1 Failover mds1 to oleg307-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2196 1285492 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26316 3568744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5416 3594244 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 31732 7162988 1% /mnt/lustre oleg307-client.virtnet: looking for dbench program oleg307-client.virtnet: /usr/bin/dbench oleg307-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg307-client.virtnet oleg307-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg307-client.virtnet' oleg307-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg307-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg307-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg307-client.virtnet at Fri Jan 17 12:11:50 EST 2025 oleg307-client.virtnet: waiting for dbench pid 19230 oleg307-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg307-client.virtnet: oleg307-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg307-client.virtnet: failed to create barrier semaphore oleg307-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg307-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg307-client.virtnet: releasing clients oleg307-client.virtnet: 1 248 9.40 MB/sec warmup 1 sec latency 46.619 ms oleg307-client.virtnet: 1 469 8.21 MB/sec warmup 2 sec latency 31.673 ms oleg307-client.virtnet: 1 779 7.12 MB/sec warmup 3 sec latency 34.756 ms oleg307-client.virtnet: 1 1043 5.72 MB/sec warmup 4 sec latency 23.772 ms oleg307-client.virtnet: 1 1418 5.20 MB/sec warmup 5 sec latency 19.359 ms oleg307-client.virtnet: 1 1803 4.39 MB/sec warmup 6 sec latency 15.971 ms oleg307-client.virtnet: 1 2225 3.83 MB/sec warmup 7 sec latency 28.036 ms oleg307-client.virtnet: 1 2610 3.63 MB/sec warmup 8 sec latency 16.838 ms oleg307-client.virtnet: 1 3353 3.93 MB/sec warmup 9 sec latency 23.958 ms oleg307-client.virtnet: 1 3814 3.98 MB/sec warmup 10 sec latency 19.717 ms oleg307-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 114.266 ms oleg307-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 1114.585 ms oleg307-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 2114.917 ms oleg307-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 3115.119 ms oleg307-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 4115.321 ms oleg307-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 5115.616 ms oleg307-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 6116.101 ms oleg307-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 7116.231 ms oleg307-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 8116.364 ms oleg307-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 9116.548 ms oleg307-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 10116.706 ms oleg307-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 11117.047 ms oleg307-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 12117.208 ms oleg307-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 13117.368 ms oleg307-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 14117.527 ms oleg307-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 15117.681 ms oleg307-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 16117.827 ms oleg307-client.virtnet: 1 4083 1.43 MB/sec warmup 28 sec latency 17118.004 ms oleg307-client.virtnet: 1 4496 1.41 MB/sec warmup 29 sec latency 17350.728 ms oleg307-client.virtnet: 1 5158 1.51 MB/sec warmup 30 sec latency 12.567 ms oleg307-client.virtnet: 1 5763 1.48 MB/sec warmup 31 sec latency 9.993 ms oleg307-client.virtnet: 1 6489 1.64 MB/sec warmup 32 sec latency 11.564 ms oleg307-client.virtnet: 1 7326 1.78 MB/sec warmup 33 sec latency 16.32test_70b fail mds2 2 times Failing mds2 on oleg307-server Stopping /mnt/lustre-mds2 (opts:) on oleg307-server reboot facets: mds2 Failover mds2 to oleg307-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37304 3566944 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12044 3593856 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49348 7160800 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg307-server Stopping /mnt/lustre-mds1 (opts:) on oleg307-server reboot facets: mds1 Failover mds1 to oleg307-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 9 ms oleg307-client.virtnet: 1 7778 1.75 MB/sec warmup 34 sec latency 41.779 ms oleg307-client.virtnet: 1 8256 1.75 MB/sec warmup 35 sec latency 126.834 ms oleg307-client.virtnet: 1 8770 1.79 MB/sec warmup 36 sec latency 16.792 ms oleg307-client.virtnet: 1 8909 1.74 MB/sec warmup 37 sec latency 704.497 ms oleg307-client.virtnet: 1 8909 1.70 MB/sec warmup 38 sec latency 1705.222 ms oleg307-client.virtnet: 1 8909 1.66 MB/sec warmup 39 sec latency 2706.725 ms oleg307-client.virtnet: 1 8909 1.61 MB/sec warmup 40 sec latency 3706.949 ms oleg307-client.virtnet: 1 8909 1.57 MB/sec warmup 41 sec latency 4707.115 ms oleg307-client.virtnet: 1 8909 1.54 MB/sec warmup 42 sec latency 5707.449 ms oleg307-client.virtnet: 1 8909 1.50 MB/sec warmup 43 sec latency 6707.713 ms oleg307-client.virtnet: 1 8909 1.47 MB/sec warmup 44 sec latency 7709.216 ms oleg307-client.virtnet: 1 8909 1.43 MB/sec warmup 45 sec latency 8709.377 ms oleg307-client.virtnet: 1 8909 1.40 MB/sec warmup 46 sec latency 9709.585 ms oleg307-client.virtnet: 1 8909 1.37 MB/sec warmup 47 sec latency 10710.110 ms oleg307-client.virtnet: 1 8909 1.34 MB/sec warmup 48 sec latency 11710.268 ms oleg307-client.virtnet: 1 8909 1.32 MB/sec warmup 49 sec latency 12710.453 ms oleg307-client.virtnet: 1 8909 1.29 MB/sec warmup 50 sec latency 13710.631 ms oleg307-client.virtnet: 1 8909 1.27 MB/sec warmup 51 sec latency 14710.780 ms oleg307-client.virtnet: 1 8909 1.24 MB/sec warmup 52 sec latency 15710.927 ms oleg307-client.virtnet: 1 8909 1.22 MB/sec warmup 53 sec latency 16711.072 ms oleg307-client.virtnet: 1 8909 1.20 MB/sec warmup 54 sec latency 17711.223 ms oleg307-client.virtnet: 1 8909 1.17 MB/sec warmup 55 sec latency 18711.432 ms oleg307-client.virtnet: 1 9171 1.16 MB/sec warmup 56 sec latency 19177.215 ms oleg307-client.virtnet: 1 9870 1.23 MB/sec warmup 57 sec latency 12.180 ms oleg307-client.virtnet: 1 10708 1.32 MB/sec warmup 58 sec latency 14.703 ms oleg307-client.virtnet: 1 11124 1.33 MB/sec warmup 59 sec latency 11.907 ms oleg307-client.virtnet: 1 12248 4.34 MB/sec execute 1 sec latency 14.500 ms oleg307-client.virtnet: 1 12788 2.43 MB/sec execute 2 sec latency 22.253 ms oleg307-client.virtnet: 1 13238 2.38 MB/sec execute 3 sec latency 17.435 ms oleg307-client.virtnet: 1 14164 4.14 MB/sec execute 4 sec latency 14.896 ms oleg307-client.virtnet: 1 14583 3.65 MB/sec execute 5 sec latency 15.809 ms oleg307-client.virtnet: 1 15008 3.13 MB/sec execute 6 sec latency 18.093 ms oleg307-client.virtnet: 1 15465 2.92 MB/sec execute 7 sec latency 14.521 ms oleg307-client.virtnet: 1 16004 2.95 MB/sec execute 8 sec latency 13.496 ms oleg307-client.virtnet: 1 16552 2.80 MB/sec execute 9 sec latency 12.733 ms oleg307-client.virtnet: 1 17188 3.08 MB/sec execute 10 sec latency 16.469 ms oleg307-client.virtnet: 1 17850 3.27 MB/sec execute 11 sec latency 17.378 ms oleg307-client.virtnet: 1 18182 3.11 MB/sec execute 12 sec latency 13.077 ms oleg307-client.virtnet: 1 18618 2.92 MB/sec execute 13 sec latency 18.513 ms oleg307-client.virtnet: 1 18994 2.82 MB/sec execute 14 sec latency 21.503 ms oleg307-client.virtnet: 1 19479 2.85 MB/sec execute 15 sec latency 14.542 ms oleg307-client.virtnet: 1 19939 2.70 MB/sec execute 16 sec latency 11.632 ms oleg307-client.virtnet: 1 20438 2.74 MB/sec execute 17 sec latency 14.588 ms oleg307-client.virtnet: 1 21326 3.05 MB/sec execute 18 sec latency 16.903 ms oleg307-client.virtnet: 1 21721 2.98 MB/sec executeoleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1832 1285856 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37476 3567904 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11752 3594156 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49228 7162060 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg307-server Stopping /mnt/lustre-mds2 (opts:) on oleg307-server reboot facets: mds2 Failover mds2 to oleg307-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1285896 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37688 3568740 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11752 3594236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49440 7162976 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg307-server Stopping /mnt/lustre-mds1 (opts:) on oleg307-server reboot facets: mds1 19 sec latency 16.699 ms oleg307-client.virtnet: 1 22112 2.85 MB/sec execute 20 sec latency 13.699 ms oleg307-client.virtnet: 1 22512 2.79 MB/sec execute 21 sec latency 14.712 ms oleg307-client.virtnet: 1 22955 2.81 MB/sec execute 22 sec latency 15.735 ms oleg307-client.virtnet: 1 23489 2.71 MB/sec execute 23 sec latency 12.439 ms oleg307-client.virtnet: 1 24085 2.81 MB/sec execute 24 sec latency 14.153 ms oleg307-client.virtnet: 1 24855 2.96 MB/sec execute 25 sec latency 17.241 ms oleg307-client.virtnet: 1 25183 2.90 MB/sec execute 26 sec latency 18.588 ms oleg307-client.virtnet: 1 25388 2.80 MB/sec execute 27 sec latency 22.416 ms oleg307-client.virtnet: 1 25785 2.73 MB/sec execute 28 sec latency 22.187 ms oleg307-client.virtnet: 1 26328 2.78 MB/sec execute 29 sec latency 13.987 ms oleg307-client.virtnet: 1 26855 2.71 MB/sec execute 30 sec latency 12.274 ms oleg307-client.virtnet: 1 27353 2.69 MB/sec execute 31 sec latency 190.444 ms oleg307-client.virtnet: 1 28223 2.82 MB/sec execute 32 sec latency 407.338 ms oleg307-client.virtnet: 1 28683 2.86 MB/sec execute 33 sec latency 18.020 ms oleg307-client.virtnet: 1 28826 2.79 MB/sec execute 34 sec latency 522.991 ms oleg307-client.virtnet: 1 28826 2.71 MB/sec execute 35 sec latency 1523.149 ms oleg307-client.virtnet: 1 28826 2.63 MB/sec execute 36 sec latency 2523.297 ms oleg307-client.virtnet: 1 28826 2.56 MB/sec execute 37 sec latency 3523.891 ms oleg307-client.virtnet: 1 28826 2.49 MB/sec execute 38 sec latency 4524.053 ms oleg307-client.virtnet: 1 28826 2.43 MB/sec execute 39 sec latency 5524.284 ms oleg307-client.virtnet: 1 28826 2.37 MB/sec execute 40 sec latency 6524.474 ms oleg307-client.virtnet: 1 28826 2.31 MB/sec execute 41 sec latency 7524.731 ms oleg307-client.virtnet: 1 28826 2.26 MB/sec execute 42 sec latency 8524.939 ms oleg307-client.virtnet: 1 28826 2.20 MB/sec execute 43 sec latency 9525.093 ms oleg307-client.virtnet: 1 28826 2.15 MB/sec execute 44 sec latency 10525.236 ms oleg307-client.virtnet: 1 28826 2.11 MB/sec execute 45 sec latency 11525.704 ms oleg307-client.virtnet: 1 28826 2.06 MB/sec execute 46 sec latency 12527.210 ms oleg307-client.virtnet: 1 28826 2.02 MB/sec execute 47 sec latency 13527.721 ms oleg307-client.virtnet: 1 28826 1.97 MB/sec execute 48 sec latency 14527.874 ms oleg307-client.virtnet: 1 28826 1.93 MB/sec execute 49 sec latency 15528.046 ms oleg307-client.virtnet: 1 28826 1.90 MB/sec execute 50 sec latency 16528.300 ms oleg307-client.virtnet: 1 28826 1.86 MB/sec execute 51 sec latency 17528.515 ms oleg307-client.virtnet: 1 28826 1.82 MB/sec execute 52 sec latency 18528.717 ms oleg307-client.virtnet: 1 28989 1.79 MB/sec execute 53 sec latency 19091.192 ms oleg307-client.virtnet: 1 29485 1.79 MB/sec execute 54 sec latency 13.056 ms oleg307-client.virtnet: 1 30074 1.82 MB/sec execute 55 sec latency 11.750 ms oleg307-client.virtnet: 1 30747 1.82 MB/sec execute 56 sec latency 10.339 ms oleg307-client.virtnet: 1 31688 1.91 MB/sec execute 57 sec latency 9.177 ms oleg307-client.virtnet: 1 32289 1.96 MB/sec execute 58 sec latency 11.562 ms oleg307-client.virtnet: 1 32792 1.94 MB/sec execute 59 sec latency 11.336 ms oleg307-client.virtnet: 1 33457 1.98 MB/sec execute 60 sec latency 10.379 ms oleg307-client.virtnet: 1 34135 1.96 MB/sec execute 61 sec latency 10.394 ms oleg307-client.virtnet: 1 34987 2.04 MB/sec execute 62 sec latency 10.440 ms oleg307-client.virtnet: 1 35795 2.11 MB/sec execute 63 sec latency 11.188 ms oleg307-clientFailover mds1 to oleg307-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1832 1285856 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37452 3568956 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11788 3594240 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49240 7163196 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg307-server Stopping /mnt/lustre-mds2 (opts:) on oleg307-server reboot facets: mds2 Failover mds2 to oleg307-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 .virtnet: 1 36281 2.08 MB/sec execute 64 sec latency 12.031 ms oleg307-client.virtnet: 1 36767 2.08 MB/sec execute 65 sec latency 11.805 ms oleg307-client.virtnet: 1 37291 2.10 MB/sec execute 66 sec latency 12.697 ms oleg307-client.virtnet: 1 37877 2.09 MB/sec execute 67 sec latency 12.782 ms oleg307-client.virtnet: 1 38514 2.14 MB/sec execute 68 sec latency 20.437 ms oleg307-client.virtnet: 1 39233 2.19 MB/sec execute 69 sec latency 12.411 ms oleg307-client.virtnet: 1 39544 2.17 MB/sec execute 70 sec latency 13.677 ms oleg307-client.virtnet: 1 39989 2.15 MB/sec execute 71 sec latency 13.731 ms oleg307-client.virtnet: 1 40544 2.18 MB/sec execute 72 sec latency 11.438 ms oleg307-client.virtnet: 1 41063 2.16 MB/sec execute 73 sec latency 10.617 ms oleg307-client.virtnet: 1 41612 2.16 MB/sec execute 74 sec latency 11.701 ms oleg307-client.virtnet: 1 42571 2.26 MB/sec execute 75 sec latency 11.424 ms oleg307-client.virtnet: 1 43012 2.25 MB/sec execute 76 sec latency 11.880 ms oleg307-client.virtnet: 1 43510 2.23 MB/sec execute 77 sec latency 13.112 ms oleg307-client.virtnet: 1 44097 2.26 MB/sec execute 78 sec latency 12.492 ms oleg307-client.virtnet: 1 44611 2.24 MB/sec execute 79 sec latency 12.028 ms oleg307-client.virtnet: 1 45168 2.24 MB/sec execute 80 sec latency 15.258 ms oleg307-client.virtnet: 1 46145 2.33 MB/sec execute 81 sec latency 11.636 ms oleg307-client.virtnet: 1 46532 2.32 MB/sec execute 82 sec latency 14.441 ms oleg307-client.virtnet: 1 46921 2.30 MB/sec execute 83 sec latency 19.421 ms oleg307-client.virtnet: 1 47223 2.29 MB/sec execute 84 sec latency 21.772 ms oleg307-client.virtnet: 1 47619 2.30 MB/sec execute 85 sec latency 19.098 ms oleg307-client.virtnet: 1 47932 2.28 MB/sec execute 86 sec latency 21.077 ms oleg307-client.virtnet: 1 48175 2.25 MB/sec execute 87 sec latency 381.078 ms oleg307-client.virtnet: 1 48644 2.25 MB/sec execute 88 sec latency 13.900 ms oleg307-client.virtnet: 1 49323 2.29 MB/sec execute 89 sec latency 18.542 ms oleg307-client.virtnet: 1 49413 2.27 MB/sec execute 90 sec latency 868.648 ms oleg307-client.virtnet: 1 49413 2.25 MB/sec execute 91 sec latency 1868.882 ms oleg307-client.virtnet: 1 49413 2.23 MB/sec execute 92 sec latency 2869.230 ms oleg307-client.virtnet: 1 49413 2.20 MB/sec execute 93 sec latency 3869.408 ms oleg307-client.virtnet: 1 49413 2.18 MB/sec execute 94 sec latency 4869.560 ms oleg307-client.virtnet: 1 49413 2.16 MB/sec execute 95 sec latency 5869.759 ms oleg307-client.virtnet: 1 49413 2.13 MB/sec execute 96 sec latency 6869.913 ms oleg307-client.virtnet: 1 49413 2.11 MB/sec execute 97 sec latency 7870.139 ms oleg307-client.virtnet: 1 49413 2.09 MB/sec execute 98 sec latency 8871.355 ms oleg307-client.virtnet: 1 49413 2.07 MB/sec execute 99 sec latency 9871.503 ms oleg307-client.virtnet: 1 49413 2.05 MB/sec execute 100 sec latency 10871.657 ms oleg307-client.virtnet: 1 49413 2.03 MB/sec execute 101 sec latency 11871.864 ms oleg307-client.virtnet: 1 49413 2.01 MB/sec execute 102 sec latency 12872.119 ms oleg307-client.virtnet: 1 49413 1.99 MB/sec execute 103 sec latency 13872.384 ms oleg307-client.virtnet: 1 49413 1.97 MB/sec execute 104 sec latency 14872.600 ms oleg307-client.virtnet: 1 49413 1.95 MB/sec execute 105 sec latency 15872.957 ms oleg307-client.virtnet: 1 49413 1.93 MB/sec execute 106 sec latency 16873.120 ms oleg307-client.virtnet: 1 49413 1.91 MB/sec execute 107 sec latency 17873.304 ms oleg307-client.virtnet: 1 49413 1.90 MB/sec executoleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1285896 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 39588 3567432 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11944 3593176 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 51532 7160608 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg307-server Stopping /mnt/lustre-mds1 (opts:) on oleg307-server reboot facets: mds1 Failover mds1 to oleg307-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec e 108 sec latency 18873.551 ms oleg307-client.virtnet: 1 49848 1.91 MB/sec execute 109 sec latency 18962.736 ms oleg307-client.virtnet: 1 50153 1.90 MB/sec execute 110 sec latency 21.769 ms oleg307-client.virtnet: 1 50547 1.89 MB/sec execute 111 sec latency 18.579 ms oleg307-client.virtnet: 1 50950 1.89 MB/sec execute 112 sec latency 14.208 ms oleg307-client.virtnet: 1 51385 1.90 MB/sec execute 113 sec latency 15.479 ms oleg307-client.virtnet: 1 51829 1.89 MB/sec execute 114 sec latency 14.554 ms oleg307-client.virtnet: 1 52244 1.89 MB/sec execute 115 sec latency 16.919 ms oleg307-client.virtnet: 1 52965 1.93 MB/sec execute 116 sec latency 16.457 ms oleg307-client.virtnet: 1 53416 1.95 MB/sec execute 117 sec latency 18.658 ms oleg307-client.virtnet: 1 53688 1.94 MB/sec execute 118 sec latency 19.590 ms oleg307-client.virtnet: 1 54024 1.92 MB/sec execute 119 sec latency 21.735 ms oleg307-client.virtnet: 1 54404 1.92 MB/sec execute 120 sec latency 13.811 ms oleg307-client.virtnet: 1 54812 1.93 MB/sec execute 121 sec latency 15.471 ms oleg307-client.virtnet: 1 55113 1.92 MB/sec execute 122 sec latency 19.135 ms oleg307-client.virtnet: 1 55580 1.91 MB/sec execute 123 sec latency 14.475 ms oleg307-client.virtnet: 1 56139 1.94 MB/sec execute 124 sec latency 19.799 ms oleg307-client.virtnet: 1 56899 1.97 MB/sec execute 125 sec latency 12.607 ms oleg307-client.virtnet: 1 57210 1.97 MB/sec execute 126 sec latency 18.327 ms oleg307-client.virtnet: 1 57578 1.95 MB/sec execute 127 sec latency 23.122 ms oleg307-client.virtnet: 1 57944 1.95 MB/sec execute 128 sec latency 16.724 ms oleg307-client.virtnet: 1 58383 1.96 MB/sec execute 129 sec latency 13.516 ms oleg307-client.virtnet: 1 58697 1.95 MB/sec execute 130 sec latency 19.800 ms oleg307-client.virtnet: 1 59126 1.94 MB/sec execute 131 sec latency 19.100 ms oleg307-client.virtnet: 1 59541 1.96 MB/sec execute 132 sec latency 29.985 ms oleg307-client.virtnet: 1 60129 1.97 MB/sec execute 133 sec latency 23.026 ms oleg307-client.virtnet: 1 60495 1.98 MB/sec execute 134 sec latency 23.945 ms oleg307-client.virtnet: 1 60759 1.98 MB/sec execute 135 sec latency 19.312 ms oleg307-client.virtnet: 1 61037 1.96 MB/sec execute 136 sec latency 29.640 ms oleg307-client.virtnet: 1 61255 1.95 MB/sec execute 137 sec latency 27.627 ms oleg307-client.virtnet: 1 61496 1.95 MB/sec execute 138 sec latency 35.487 ms oleg307-client.virtnet: 1 61791 1.96 MB/sec execute 139 sec latency 22.342 ms oleg307-client.virtnet: 1 61940 1.94 MB/sec execute 140 sec latency 35.463 ms oleg307-client.virtnet: 1 62126 1.93 MB/sec execute 141 sec latency 31.271 ms oleg307-client.virtnet: 1 62246 1.92 MB/sec execute 142 sec latency 49.078 ms oleg307-client.virtnet: 1 62428 1.91 MB/sec execute 143 sec latency 50.847 ms oleg307-client.virtnet: 1 62583 1.89 MB/sec execute 144 sec latency 39.528 ms oleg307-client.virtnet: 1 62738 1.89 MB/sec execute 145 sec latency 53.205 ms oleg307-client.virtnet: 1 62912 1.88 MB/sec execute 146 sec latency 40.565 ms oleg307-client.virtnet: 1 63103 1.89 MB/sec execute 147 sec latency 60.248 ms oleg307-client.virtnet: 1 63313 1.89 MB/sec execute 148 sec latency 81.427 ms oleg307-client.virtnet: 1 63538 1.88 MB/sec execute 149 sec latency 61.942 ms oleg307-client.virtnet: 1 63734 1.88 MB/sec execute 150 sec latency 62.662 ms oleg307-client.virtnet: 1 63890 1.88 MB/sec execute 151 sec latency 88.692 ms oleg307-client.virtnet: 1 64000 1.87 MB/sec execute 152 sec latency 75.361 ms oleg307-client.virtnet: 1 64150 1.87 MB/sec eUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1824 1285864 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37416 3569192 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11912 3592228 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49328 7161420 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg307-server Stopping /mnt/lustre-mds2 (opts:) on oleg307-server reboot facets: mds2 Failover mds2 to oleg307-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1285896 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37380 3569072 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12036 3593424 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49416 7162496 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg307-server Stopping /mnt/lustre-mds1 (opts:) on oleg307-server reboot facets: mds1 xecute 153 sec latency 46.058 ms oleg307-client.virtnet: 1 64246 1.86 MB/sec execute 154 sec latency 84.325 ms oleg307-client.virtnet: 1 64369 1.84 MB/sec execute 155 sec latency 60.448 ms oleg307-client.virtnet: 1 64521 1.83 MB/sec execute 156 sec latency 71.050 ms oleg307-client.virtnet: 1 64611 1.82 MB/sec execute 157 sec latency 49.935 ms oleg307-client.virtnet: 1 64744 1.81 MB/sec execute 158 sec latency 47.860 ms oleg307-client.virtnet: 1 64785 1.80 MB/sec execute 159 sec latency 634.504 ms oleg307-client.virtnet: 1 65053 1.80 MB/sec execute 160 sec latency 27.423 ms oleg307-client.virtnet: 1 65373 1.81 MB/sec execute 161 sec latency 22.648 ms oleg307-client.virtnet: 1 65458 1.80 MB/sec execute 162 sec latency 510.319 ms oleg307-client.virtnet: 1 65458 1.79 MB/sec execute 163 sec latency 1512.388 ms oleg307-client.virtnet: 1 65458 1.77 MB/sec execute 164 sec latency 2512.543 ms oleg307-client.virtnet: 1 65458 1.76 MB/sec execute 165 sec latency 3512.727 ms oleg307-client.virtnet: 1 65458 1.75 MB/sec execute 166 sec latency 4514.183 ms oleg307-client.virtnet: 1 65458 1.74 MB/sec execute 167 sec latency 5514.617 ms oleg307-client.virtnet: 1 65458 1.73 MB/sec execute 168 sec latency 6516.432 ms oleg307-client.virtnet: 1 65458 1.72 MB/sec execute 169 sec latency 7519.075 ms oleg307-client.virtnet: 1 65458 1.71 MB/sec execute 170 sec latency 8519.321 ms oleg307-client.virtnet: 1 65458 1.70 MB/sec execute 171 sec latency 9520.312 ms oleg307-client.virtnet: 1 65458 1.69 MB/sec execute 172 sec latency 10520.477 ms oleg307-client.virtnet: 1 65458 1.68 MB/sec execute 173 sec latency 11521.572 ms oleg307-client.virtnet: 1 65458 1.67 MB/sec execute 174 sec latency 12521.730 ms oleg307-client.virtnet: 1 65458 1.66 MB/sec execute 175 sec latency 13521.925 ms oleg307-client.virtnet: 1 65458 1.65 MB/sec execute 176 sec latency 14522.099 ms oleg307-client.virtnet: 1 65458 1.64 MB/sec execute 177 sec latency 15522.262 ms oleg307-client.virtnet: 1 65458 1.63 MB/sec execute 178 sec latency 16523.551 ms oleg307-client.virtnet: 1 65458 1.63 MB/sec execute 179 sec latency 17530.290 ms oleg307-client.virtnet: 1 65458 1.62 MB/sec execute 180 sec latency 18530.440 ms oleg307-client.virtnet: 1 65458 1.61 MB/sec execute 181 sec latency 19531.039 ms oleg307-client.virtnet: 1 65458 1.60 MB/sec execute 182 sec latency 20531.471 ms oleg307-client.virtnet: 1 65545 1.59 MB/sec execute 183 sec latency 20948.878 ms oleg307-client.virtnet: 1 65678 1.58 MB/sec execute 184 sec latency 38.011 ms oleg307-client.virtnet: 1 65811 1.57 MB/sec execute 185 sec latency 43.202 ms oleg307-client.virtnet: 1 65938 1.57 MB/sec execute 186 sec latency 69.095 ms oleg307-client.virtnet: 1 66221 1.57 MB/sec execute 187 sec latency 26.537 ms oleg307-client.virtnet: 1 66428 1.56 MB/sec execute 188 sec latency 31.540 ms oleg307-client.virtnet: 1 66779 1.58 MB/sec execute 189 sec latency 70.180 ms oleg307-client.virtnet: 1 67037 1.57 MB/sec execute 190 sec latency 65.899 ms oleg307-client.virtnet: 1 67247 1.57 MB/sec execute 191 sec latency 50.633 ms oleg307-client.virtnet: 1 67550 1.58 MB/sec execute 192 sec latency 23.348 ms oleg307-client.virtnet: 1 67784 1.58 MB/sec execute 193 sec latency 20.412 ms oleg307-client.virtnet: 1 67927 1.57 MB/sec execute 194 sec latency 68.085 ms oleg307-client.virtnet: 1 68142 1.57 MB/sec execute 195 sec latency 39.534 ms oleg307-client.virtnet: 1 68288 1.56 MB/sec execute 196 sec latency 52.819 ms oleg307-client.virtnet: 1 68521 1.56 MB/sec execute 197 sec latency 24.312 ms olFailover mds1 to oleg307-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1808 1285880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37416 3568528 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11924 3593412 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49340 7161940 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg307-server Stopping /mnt/lustre-mds2 (opts:) on oleg307-server reboot facets: mds2 Failover mds2 to oleg307-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 eg307-client.virtnet: 1 68837 1.57 MB/sec execute 198 sec latency 24.118 ms oleg307-client.virtnet: 1 69077 1.56 MB/sec execute 199 sec latency 25.144 ms oleg307-client.virtnet: 1 69296 1.55 MB/sec execute 200 sec latency 25.020 ms oleg307-client.virtnet: 1 69648 1.55 MB/sec execute 201 sec latency 16.768 ms oleg307-client.virtnet: 1 69913 1.55 MB/sec execute 202 sec latency 30.256 ms oleg307-client.virtnet: 1 70404 1.56 MB/sec execute 203 sec latency 30.938 ms oleg307-client.virtnet: 1 70942 1.58 MB/sec execute 204 sec latency 22.442 ms oleg307-client.virtnet: 1 71206 1.58 MB/sec execute 205 sec latency 24.637 ms oleg307-client.virtnet: 1 71384 1.57 MB/sec execute 206 sec latency 24.833 ms oleg307-client.virtnet: 1 71641 1.57 MB/sec execute 207 sec latency 36.225 ms oleg307-client.virtnet: 1 71789 1.56 MB/sec execute 208 sec latency 35.834 ms oleg307-client.virtnet: 1 71962 1.56 MB/sec execute 209 sec latency 40.540 ms oleg307-client.virtnet: 1 72213 1.55 MB/sec execute 210 sec latency 32.651 ms oleg307-client.virtnet: 1 72531 1.56 MB/sec execute 211 sec latency 24.744 ms oleg307-client.virtnet: 1 72731 1.55 MB/sec execute 212 sec latency 36.407 ms oleg307-client.virtnet: 1 72996 1.55 MB/sec execute 213 sec latency 18.650 ms oleg307-client.virtnet: 1 73354 1.55 MB/sec execute 214 sec latency 19.695 ms oleg307-client.virtnet: 1 73714 1.56 MB/sec execute 215 sec latency 18.759 ms oleg307-client.virtnet: 1 74164 1.56 MB/sec execute 216 sec latency 46.039 ms oleg307-client.virtnet: 1 74560 1.57 MB/sec execute 217 sec latency 24.885 ms oleg307-client.virtnet: 1 74845 1.57 MB/sec execute 218 sec latency 20.225 ms oleg307-client.virtnet: 1 75057 1.57 MB/sec execute 219 sec latency 23.681 ms oleg307-client.virtnet: 1 75334 1.56 MB/sec execute 220 sec latency 32.843 ms oleg307-client.virtnet: 1 75593 1.56 MB/sec execute 221 sec latency 25.339 ms oleg307-client.virtnet: 1 75824 1.56 MB/sec execute 222 sec latency 28.472 ms oleg307-client.virtnet: 1 76031 1.56 MB/sec execute 223 sec latency 507.066 ms oleg307-client.virtnet: 1 76245 1.56 MB/sec execute 224 sec latency 564.169 ms oleg307-client.virtnet: 1 76502 1.55 MB/sec execute 225 sec latency 21.583 ms oleg307-client.virtnet: 1 76698 1.54 MB/sec execute 226 sec latency 257.276 ms oleg307-client.virtnet: 1 76698 1.54 MB/sec execute 227 sec latency 1257.478 ms oleg307-client.virtnet: 1 76698 1.53 MB/sec execute 228 sec latency 2259.176 ms oleg307-client.virtnet: 1 76698 1.52 MB/sec execute 229 sec latency 3259.708 ms oleg307-client.virtnet: 1 76698 1.52 MB/sec execute 230 sec latency 4261.413 ms oleg307-client.virtnet: 1 76698 1.51 MB/sec execute 231 sec latency 5261.603 ms oleg307-client.virtnet: 1 76698 1.50 MB/sec execute 232 sec latency 6261.769 ms oleg307-client.virtnet: 1 76698 1.50 MB/sec execute 233 sec latency 7262.219 ms oleg307-client.virtnet: 1 76698 1.49 MB/sec execute 234 sec latency 8263.067 ms oleg307-client.virtnet: 1 76698 1.48 MB/sec execute 235 sec latency 9263.364 ms oleg307-client.virtnet: 1 76698 1.48 MB/sec execute 236 sec latency 10263.631 ms oleg307-client.virtnet: 1 76698 1.47 MB/sec execute 237 sec latency 11263.818 ms oleg307-client.virtnet: 1 76698 1.47 MB/sec execute 238 sec latency 12265.377 ms oleg307-client.virtnet: 1 76698 1.46 MB/sec execute 239 sec latency 13265.738 ms oleg307-client.virtnet: 1 76698 1.45 MB/sec execute 240 sec latency 14266.732 ms oleg307-client.virtnet: 1 76698 1.45 MB/sec execute 241 sec latency 15267.291 ms oleg307-client.virtnet: 1 76698 1.44 MB/sec oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec execute 242 sec latency 16268.075 ms oleg307-client.virtnet: 1 76698 1.44 MB/sec execute 243 sec latency 17268.388 ms oleg307-client.virtnet: 1 76698 1.43 MB/sec execute 244 sec latency 18268.586 ms oleg307-client.virtnet: 1 76698 1.42 MB/sec execute 245 sec latency 19268.915 ms oleg307-client.virtnet: 1 76698 1.42 MB/sec execute 246 sec latency 20269.474 ms oleg307-client.virtnet: 1 76818 1.41 MB/sec execute 247 sec latency 20733.389 ms oleg307-client.virtnet: 1 77102 1.42 MB/sec execute 248 sec latency 22.993 ms oleg307-client.virtnet: 1 77511 1.43 MB/sec execute 249 sec latency 42.432 ms oleg307-client.virtnet: 1 78009 1.44 MB/sec execute 250 sec latency 25.227 ms oleg307-client.virtnet: 1 78252 1.44 MB/sec execute 251 sec latency 29.612 ms oleg307-client.virtnet: 1 78478 1.44 MB/sec execute 252 sec latency 25.098 ms oleg307-client.virtnet: 1 78739 1.43 MB/sec execute 253 sec latency 23.859 ms oleg307-client.virtnet: 1 78966 1.43 MB/sec execute 254 sec latency 32.892 ms oleg307-client.virtnet: 1 79212 1.43 MB/sec execute 255 sec latency 27.005 ms oleg307-client.virtnet: 1 79492 1.44 MB/sec execute 256 sec latency 32.281 ms oleg307-client.virtnet: 1 79717 1.43 MB/sec execute 257 sec latency 18.634 ms oleg307-client.virtnet: 1 79933 1.43 MB/sec execute 258 sec latency 23.078 ms oleg307-client.virtnet: 1 80246 1.42 MB/sec execute 259 sec latency 19.663 ms oleg307-client.virtnet: 1 80483 1.42 MB/sec execute 260 sec latency 23.809 ms oleg307-client.virtnet: 1 80802 1.43 MB/sec execute 261 sec latency 31.479 ms oleg307-client.virtnet: 1 81273 1.43 MB/sec execute 262 sec latency 30.800 ms oleg307-client.virtnet: 1 81700 1.44 MB/sec execute 263 sec latency 22.557 ms oleg307-client.virtnet: 1 81898 1.44 MB/sec execute 264 sec latency 36.101 ms oleg307-client.virtnet: 1 82061 1.44 MB/sec execute 265 sec latency 39.009 ms oleg307-client.virtnet: 1 82319 1.43 MB/sec execute 266 sec latency 27.376 ms oleg307-client.virtnet: 1 82547 1.43 MB/sec execute 267 sec latency 28.663 ms oleg307-client.virtnet: 1 82785 1.43 MB/sec execute 268 sec latency 29.697 ms oleg307-client.virtnet: 1 83101 1.44 MB/sec execute 269 sec latency 20.138 ms oleg307-client.virtnet: 1 83330 1.43 MB/sec execute 270 sec latency 25.849 ms oleg307-client.virtnet: 1 83622 1.43 MB/sec execute 271 sec latency 23.832 ms oleg307-client.virtnet: 1 83963 1.43 MB/sec execute 272 sec latency 25.398 ms oleg307-client.virtnet: 1 84212 1.43 MB/sec execute 273 sec latency 20.944 ms oleg307-client.virtnet: 1 84644 1.44 MB/sec execute 274 sec latency 23.347 ms oleg307-client.virtnet: 1 85171 1.45 MB/sec execute 275 sec latency 26.094 ms oleg307-client.virtnet: 1 85418 1.45 MB/sec execute 276 sec latency 39.295 ms oleg307-client.virtnet: 1 85597 1.45 MB/sec execute 277 sec latency 26.775 ms oleg307-client.virtnet: 1 85844 1.44 MB/sec execute 278 sec latency 21.070 ms oleg307-client.virtnet: 1 86079 1.44 MB/sec execute 279 sec latency 33.121 ms oleg307-client.virtnet: 1 86303 1.44 MB/sec execute 280 sec latency 24.946 ms oleg307-client.virtnet: 1 86579 1.44 MB/sec execute 281 sec latency 41.474 ms oleg307-client.virtnet: 1 86797 1.44 MB/sec execute 282 sec latency 21.925 ms oleg307-client.virtnet: 1 87022 1.43 MB/sec execute 283 sec latency 31.355 ms oleg307-client.virtnet: 1 87262 1.43 MB/sec execute 284 sec latency 34.228 ms oleg307-client.virtnet: 1 87566 1.43 MB/sec execute 285 sec latency 20.282 ms oleg307-client.virtnet: 1 87832 1.43 MB/sec execute 286 sec latency 30.771 ms oleg307-client.virtnet: 1 88384 1.44 MB/sec execute 287 sec latency 23.698 ms oleg307-client.virtnet: 1 88809 1.45 MB/sec execute 288 sec latency 23.291 ms oleg307-client.virtnet: 1 89035 1.45 MB/sec execute 289 sec latency 24.086 ms oleg307-client.virtnet: 1 89235 1.45 MB/sec execute 290 sec latency 24.381 ms oleg307-client.virtnet: 1 89531 1.44 MB/sec execute 291 sec latency 31.448 ms oleg307-client.virtnet: 1 89866 1.44 MB/sec execute 292 sec latency 24.463 ms oleg307-client.virtnet: 1 90222 1.45 MB/sec execute 293 sec latency 22.156 ms oleg307-client.virtnet: 1 90483 1.45 MB/sec execute 294 sec latency 23.953 ms oleg307-client.virtnet: 1 90750 1.44 MB/sec execute 295 sec latency 21.732 ms oleg307-client.virtnet: 1 91083 1.44 MB/sec execute 296 sec latency 36.444 ms oleg307-client.virtnet: 1 91323 1.44 MB/sec execute 297 sec latency 22.518 ms oleg307-client.virtnet: 1 91893 1.45 MB/sec execute 298 sec latency 25.759 ms oleg307-client.virtnet: 1 92340 1.46 MB/sec execute 299 sec latency 17.671 ms oleg307-client.virtnet: 1 cleanup 300 sec oleg307-client.virtnet: 0 cleanup 301 sec oleg307-client.virtnet: oleg307-client.virtnet: Operation Count AvgLat MaxLat oleg307-client.virtnet: ---------------------------------------- oleg307-client.virtnet: NTCreateX 13965 13.291 20948.866 oleg307-client.virtnet: Close 10283 1.830 22.852 oleg307-client.virtnet: Rename 595 12.285 68.187 oleg307-client.virtnet: Unlink 2793 3.007 20.116 oleg307-client.virtnet: Qpathinfo 12682 3.511 18962.721 oleg307-client.virtnet: Qfileinfo 2222 0.162 6.720 oleg307-client.virtnet: Qfsinfo 2321 0.221 23.383 oleg307-client.virtnet: Sfileinfo 1134 6.082 39.504 oleg307-client.virtnet: Find 4900 0.535 71.031 oleg307-client.virtnet: WriteX 6923 1.590 38.316 oleg307-client.virtnet: ReadX 22063 0.016 5.330 oleg307-client.virtnet: LockX 46 1.941 5.386 oleg307-client.virtnet: UnlockX 46 2.045 10.623 oleg307-client.virtnet: Flush 967 13.101 634.492 oleg307-client.virtnet: oleg307-client.virtnet: Throughput 1.4626 MB/sec 1 clients 1 procs max_latency=20948.878 ms oleg307-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg307-client.virtnet at Fri Jan 17 12:17:51 EST 2025 with return code 0 oleg307-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg307-client.virtnet oleg307-client.virtnet: /mnt/lustre/d70b.replay-single/oleg307-client.virtnet /mnt/lustre/d70b.replay-single/oleg307-client.virtnet oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg307-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg307-client.virtnet: removed directory: 'clients/client0' oleg307-client.virtnet: removed directory: 'clients' oleg307-client.virtnet: removed 'client.txt' oleg307-client.virtnet: /mnt/lustre/d70b.replay-single/oleg307-client.virtnet oleg307-client.virtnet: dbench successfully finished