== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 11:44:11 (1730825051) Starting client oleg102-client.virtnet: -o user_xattr,flock oleg102-server@tcp:/lustre /mnt/lustre Started clients oleg102-client.virtnet: 192.168.201.102@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg102-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=19260 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26024 3580996 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1416 3593300 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27440 7174296 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg102-server Stopping /mnt/lustre-mds1 (opts:) on oleg102-server reboot facets: mds1 Failover mds1 to oleg102-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg102-client.virtnet: looking for dbench program oleg102-client.virtnet: /usr/bin/dbench oleg102-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg102-client.virtnet oleg102-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg102-client.virtnet' oleg102-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg102-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg102-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg102-client.virtnet at Tue Nov 5 11:44:12 EST 2024 oleg102-client.virtnet: waiting for dbench pid 19285 oleg102-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg102-client.virtnet: oleg102-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg102-client.virtnet: failed to create barrier semaphore oleg102-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg102-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg102-client.virtnet: releasing clients oleg102-client.virtnet: 1 289 10.70 MB/sec warmup 1 sec latency 22.950 ms oleg102-client.virtnet: 1 616 10.49 MB/sec warmup 2 sec latency 19.930 ms oleg102-client.virtnet: 1 927 7.26 MB/sec warmup 3 sec latency 17.958 ms oleg102-client.virtnet: 1 1396 6.50 MB/sec warmup 4 sec latency 15.433 ms oleg102-client.virtnet: 1 1808 5.27 MB/sec warmup 5 sec latency 16.360 ms oleg102-client.virtnet: 1 2373 4.65 MB/sec warmup 6 sec latency 10.898 ms oleg102-client.virtnet: 1 2932 4.78 MB/sec warmup 7 sec latency 17.560 ms oleg102-client.virtnet: 1 3551 4.80 MB/sec warmup 8 sec latency 18.185 ms oleg102-client.virtnet: 1 3849 4.42 MB/sec warmup 9 sec latency 17.793 ms oleg102-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 44.004 ms oleg102-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 1044.219 ms oleg102-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 2044.462 ms oleg102-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 3044.653 ms oleg102-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 4044.862 ms oleg102-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 5045.073 ms oleg102-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 6045.281 ms oleg102-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 7045.513 ms oleg102-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 8045.660 ms oleg102-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 9045.799 ms oleg102-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 10045.968 ms oleg102-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 11046.207 ms oleg102-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 12046.462 ms oleg102-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 13046.681 ms oleg102-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 14046.991 ms oleg102-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 15047.304 ms oleg102-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 16047.531 ms oleg102-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 17047.792 ms oleg102-client.virtnet: 1 4083 1.43 MB/sec warmup 28 sec latency 18048.020 ms oleg102-client.virtnet: 1 4083 1.38 MB/sec warmup 29 sec latency 19048.268 ms oleg102-client.virtnet: 1 4083 1.34 MB/sec warmup 30 sec latency 20048.523 ms oleg102-client.virtnet: 1 4458 1.32 MB/sec warmup 31 sec latency 20121.031 ms oleg102-client.virtnet: 1 4799 1.32 MB/sec warmup 32 sec latency 17.009 ms oleg102-client.virtnet: 1 5208 1.37 MB/sec warmup 33 sec lateUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2196 1285492 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36268 3570244 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12940 3593140 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49208 7163384 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg102-server Stopping /mnt/lustre-mds2 (opts:) on oleg102-server reboot facets: mds2 Failover mds2 to oleg102-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1796 1285892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36292 3567724 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13052 3592988 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49344 7160712 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg102-server Stopping /mnt/lustre-mds1 (opts:) on oleg102-server reboot facets: mds1 ncy 17.828 ms oleg102-client.virtnet: 1 5551 1.34 MB/sec warmup 34 sec latency 18.243 ms oleg102-client.virtnet: 1 5988 1.35 MB/sec warmup 35 sec latency 12.699 ms oleg102-client.virtnet: 1 6552 1.46 MB/sec warmup 36 sec latency 16.192 ms oleg102-client.virtnet: 1 7233 1.56 MB/sec warmup 37 sec latency 18.515 ms oleg102-client.virtnet: 1 7503 1.56 MB/sec warmup 38 sec latency 18.259 ms oleg102-client.virtnet: 1 7837 1.53 MB/sec warmup 39 sec latency 18.167 ms oleg102-client.virtnet: 1 8192 1.53 MB/sec warmup 40 sec latency 170.489 ms oleg102-client.virtnet: 1 8192 1.49 MB/sec warmup 41 sec latency 1170.788 ms oleg102-client.virtnet: 1 8192 1.46 MB/sec warmup 42 sec latency 2171.061 ms oleg102-client.virtnet: 1 8192 1.42 MB/sec warmup 43 sec latency 3171.301 ms oleg102-client.virtnet: 1 8192 1.39 MB/sec warmup 44 sec latency 4171.496 ms oleg102-client.virtnet: 1 8192 1.36 MB/sec warmup 45 sec latency 5171.766 ms oleg102-client.virtnet: 1 8192 1.33 MB/sec warmup 46 sec latency 6171.947 ms oleg102-client.virtnet: 1 8192 1.30 MB/sec warmup 47 sec latency 7172.166 ms oleg102-client.virtnet: 1 8192 1.27 MB/sec warmup 48 sec latency 8172.404 ms oleg102-client.virtnet: 1 8192 1.25 MB/sec warmup 49 sec latency 9172.651 ms oleg102-client.virtnet: 1 8192 1.22 MB/sec warmup 50 sec latency 10172.956 ms oleg102-client.virtnet: 1 8192 1.20 MB/sec warmup 51 sec latency 11173.286 ms oleg102-client.virtnet: 1 8192 1.18 MB/sec warmup 52 sec latency 12173.568 ms oleg102-client.virtnet: 1 8192 1.15 MB/sec warmup 53 sec latency 13173.876 ms oleg102-client.virtnet: 1 8192 1.13 MB/sec warmup 54 sec latency 14174.115 ms oleg102-client.virtnet: 1 8192 1.11 MB/sec warmup 55 sec latency 15174.321 ms oleg102-client.virtnet: 1 8192 1.09 MB/sec warmup 56 sec latency 16174.618 ms oleg102-client.virtnet: 1 8192 1.07 MB/sec warmup 57 sec latency 17174.812 ms oleg102-client.virtnet: 1 8192 1.05 MB/sec warmup 58 sec latency 18175.012 ms oleg102-client.virtnet: 1 8192 1.04 MB/sec warmup 59 sec latency 19175.220 ms oleg102-client.virtnet: 1 8192 0.00 MB/sec execute 1 sec latency 21175.574 ms oleg102-client.virtnet: 1 8192 0.00 MB/sec execute 2 sec latency 22175.812 ms oleg102-client.virtnet: 1 8192 0.00 MB/sec execute 3 sec latency 23176.023 ms oleg102-client.virtnet: 1 8192 0.00 MB/sec execute 4 sec latency 24176.189 ms oleg102-client.virtnet: 1 8192 0.00 MB/sec execute 5 sec latency 25176.339 ms oleg102-client.virtnet: 1 8192 0.00 MB/sec execute 6 sec latency 26176.569 ms oleg102-client.virtnet: 1 8508 0.44 MB/sec execute 7 sec latency 26445.800 ms oleg102-client.virtnet: 1 8814 0.41 MB/sec execute 8 sec latency 18.607 ms oleg102-client.virtnet: 1 9201 0.41 MB/sec execute 9 sec latency 10.879 ms oleg102-client.virtnet: 1 9569 0.57 MB/sec execute 10 sec latency 14.465 ms oleg102-client.virtnet: 1 10100 0.96 MB/sec execute 11 sec latency 16.718 ms oleg102-client.virtnet: 1 10673 1.29 MB/sec execute 12 sec latency 26.181 ms oleg102-client.virtnet: 1 10953 1.29 MB/sec execute 13 sec latency 19.953 ms oleg102-client.virtnet: 1 11241 1.23 MB/sec execute 14 sec latency 18.608 ms oleg102-client.virtnet: 1 11551 1.19 MB/sec execute 15 sec latency 22.833 ms oleg102-client.virtnet: 1 11892 1.20 MB/sec execute 16 sec latency 17.698 ms oleg102-client.virtnet: 1 12289 1.32 MB/sec execute 17 sec latency 17.385 ms oleg102-client.virtnet: 1 12632 1.26 MB/sec execute 18 sec latency 18.183 ms oleg102-client.virtnet: 1 1Failover mds1 to oleg102-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1824 1285864 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36436 3569864 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12784 3593324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49220 7163188 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg102-server Stopping /mnt/lustre-mds2 (opts:) on oleg102-server reboot facets: mds2 3030 1.27 MB/sec execute 19 sec latency 20.291 ms oleg102-client.virtnet: 1 13531 1.42 MB/sec execute 20 sec latency 15.821 ms oleg102-client.virtnet: 1 14146 1.64 MB/sec execute 21 sec latency 18.134 ms oleg102-client.virtnet: 1 14455 1.63 MB/sec execute 22 sec latency 18.065 ms oleg102-client.virtnet: 1 14675 1.58 MB/sec execute 23 sec latency 18.762 ms oleg102-client.virtnet: 1 15001 1.53 MB/sec execute 24 sec latency 20.478 ms oleg102-client.virtnet: 1 15365 1.53 MB/sec execute 25 sec latency 18.540 ms oleg102-client.virtnet: 1 15759 1.59 MB/sec execute 26 sec latency 19.021 ms oleg102-client.virtnet: 1 16046 1.54 MB/sec execute 27 sec latency 18.796 ms oleg102-client.virtnet: 1 16488 1.54 MB/sec execute 28 sec latency 11.063 ms oleg102-client.virtnet: 1 16850 1.56 MB/sec execute 29 sec latency 15.907 ms oleg102-client.virtnet: 1 17535 1.69 MB/sec execute 30 sec latency 17.533 ms oleg102-client.virtnet: 1 17951 1.78 MB/sec execute 31 sec latency 15.225 ms oleg102-client.virtnet: 1 18166 1.73 MB/sec execute 32 sec latency 18.013 ms oleg102-client.virtnet: 1 18476 1.69 MB/sec execute 33 sec latency 22.067 ms oleg102-client.virtnet: 1 18803 1.69 MB/sec execute 34 sec latency 18.469 ms oleg102-client.virtnet: 1 19237 1.73 MB/sec execute 35 sec latency 17.743 ms oleg102-client.virtnet: 1 19541 1.69 MB/sec execute 36 sec latency 18.251 ms oleg102-client.virtnet: 1 19934 1.65 MB/sec execute 37 sec latency 11.147 ms oleg102-client.virtnet: 1 20275 1.66 MB/sec execute 38 sec latency 15.303 ms oleg102-client.virtnet: 1 20949 1.76 MB/sec execute 39 sec latency 18.060 ms oleg102-client.virtnet: 1 21446 1.83 MB/sec execute 40 sec latency 17.052 ms oleg102-client.virtnet: 1 21713 1.82 MB/sec execute 41 sec latency 18.827 ms oleg102-client.virtnet: 1 22025 1.78 MB/sec execute 42 sec latency 20.328 ms oleg102-client.virtnet: 1 22309 1.76 MB/sec execute 43 sec latency 17.349 ms oleg102-client.virtnet: 1 22734 1.81 MB/sec execute 44 sec latency 17.206 ms oleg102-client.virtnet: 1 23045 1.77 MB/sec execute 45 sec latency 17.838 ms oleg102-client.virtnet: 1 23416 1.74 MB/sec execute 46 sec latency 11.233 ms oleg102-client.virtnet: 1 23780 1.74 MB/sec execute 47 sec latency 15.174 ms oleg102-client.virtnet: 1 24319 1.82 MB/sec execute 48 sec latency 17.083 ms oleg102-client.virtnet: 1 24600 1.82 MB/sec execute 49 sec latency 655.740 ms oleg102-client.virtnet: 1 25035 1.85 MB/sec execute 50 sec latency 13.501 ms oleg102-client.virtnet: 1 25286 1.84 MB/sec execute 51 sec latency 18.336 ms oleg102-client.virtnet: 1 25359 1.80 MB/sec execute 52 sec latency 663.808 ms oleg102-client.virtnet: 1 25359 1.77 MB/sec execute 53 sec latency 1664.075 ms oleg102-client.virtnet: 1 25359 1.73 MB/sec execute 54 sec latency 2664.287 ms oleg102-client.virtnet: 1 25359 1.70 MB/sec execute 55 sec latency 3664.467 ms oleg102-client.virtnet: 1 25359 1.67 MB/sec execute 56 sec latency 4664.715 ms oleg102-client.virtnet: 1 25359 1.64 MB/sec execute 57 sec latency 5664.970 ms oleg102-client.virtnet: 1 25359 1.61 MB/sec execute 58 sec latency 6665.236 ms oleg102-client.virtnet: 1 25359 1.59 MB/sec execute 59 sec latency 7665.457 ms oleg102-client.virtnet: 1 25359 1.56 MB/sec execute 60 sec latency 8665.642 ms oleg102-client.virtnet: 1 25359 1.54 MB/sec execute 61 sec latency 9665.891 ms oleg102-client.virtnet: 1 25359 1.51 MB/sec execute 62 sec latency 10666.153 ms oleg102-client.virtnet: 1 25359 1.49 MB/sec execute 63 sec latency 11666.416 ms oleg102-cFailover mds2 to oleg102-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1285896 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36268 3569272 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12940 3593048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49208 7162320 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg102-server Stopping /mnt/lustre-mds1 (opts:) on oleg102-server reboot facets: mds1 Failover mds1 to oleg102-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 lient.virtnet: 1 25359 1.46 MB/sec execute 64 sec latency 12666.698 ms oleg102-client.virtnet: 1 25359 1.44 MB/sec execute 65 sec latency 13666.913 ms oleg102-client.virtnet: 1 25359 1.42 MB/sec execute 66 sec latency 14667.167 ms oleg102-client.virtnet: 1 25359 1.40 MB/sec execute 67 sec latency 15667.492 ms oleg102-client.virtnet: 1 25359 1.38 MB/sec execute 68 sec latency 16667.756 ms oleg102-client.virtnet: 1 25359 1.36 MB/sec execute 69 sec latency 17667.903 ms oleg102-client.virtnet: 1 25359 1.34 MB/sec execute 70 sec latency 18668.099 ms oleg102-client.virtnet: 1 25359 1.32 MB/sec execute 71 sec latency 19668.242 ms oleg102-client.virtnet: 1 25359 1.30 MB/sec execute 72 sec latency 20668.488 ms oleg102-client.virtnet: 1 25359 1.28 MB/sec execute 73 sec latency 21668.660 ms oleg102-client.virtnet: 1 25359 1.27 MB/sec execute 74 sec latency 22668.859 ms oleg102-client.virtnet: 1 25359 1.25 MB/sec execute 75 sec latency 23669.144 ms oleg102-client.virtnet: 1 25359 1.23 MB/sec execute 76 sec latency 24669.424 ms oleg102-client.virtnet: 1 25359 1.22 MB/sec execute 77 sec latency 25669.727 ms oleg102-client.virtnet: 1 25494 1.20 MB/sec execute 78 sec latency 26224.943 ms oleg102-client.virtnet: 1 25798 1.20 MB/sec execute 79 sec latency 18.740 ms oleg102-client.virtnet: 1 26133 1.20 MB/sec execute 80 sec latency 17.233 ms oleg102-client.virtnet: 1 26513 1.22 MB/sec execute 81 sec latency 18.884 ms oleg102-client.virtnet: 1 26859 1.21 MB/sec execute 82 sec latency 17.471 ms oleg102-client.virtnet: 1 27264 1.21 MB/sec execute 83 sec latency 12.092 ms oleg102-client.virtnet: 1 27774 1.26 MB/sec execute 84 sec latency 15.269 ms oleg102-client.virtnet: 1 28498 1.31 MB/sec execute 85 sec latency 16.817 ms oleg102-client.virtnet: 1 28779 1.31 MB/sec execute 86 sec latency 19.574 ms oleg102-client.virtnet: 1 29083 1.30 MB/sec execute 87 sec latency 18.485 ms oleg102-client.virtnet: 1 29393 1.29 MB/sec execute 88 sec latency 27.115 ms oleg102-client.virtnet: 1 29813 1.32 MB/sec execute 89 sec latency 17.853 ms oleg102-client.virtnet: 1 30121 1.31 MB/sec execute 90 sec latency 18.001 ms oleg102-client.virtnet: 1 30519 1.30 MB/sec execute 91 sec latency 12.304 ms oleg102-client.virtnet: 1 30874 1.30 MB/sec execute 92 sec latency 15.328 ms oleg102-client.virtnet: 1 31425 1.35 MB/sec execute 93 sec latency 17.729 ms oleg102-client.virtnet: 1 32037 1.39 MB/sec execute 94 sec latency 17.990 ms oleg102-client.virtnet: 1 32326 1.39 MB/sec execute 95 sec latency 18.161 ms oleg102-client.virtnet: 1 32646 1.38 MB/sec execute 96 sec latency 18.833 ms oleg102-client.virtnet: 1 32941 1.37 MB/sec execute 97 sec latency 17.929 ms oleg102-client.virtnet: 1 33358 1.40 MB/sec execute 98 sec latency 17.707 ms oleg102-client.virtnet: 1 33668 1.39 MB/sec execute 99 sec latency 17.949 ms oleg102-client.virtnet: 1 34032 1.38 MB/sec execute 100 sec latency 12.805 ms oleg102-client.virtnet: 1 34419 1.38 MB/sec execute 101 sec latency 15.397 ms oleg102-client.virtnet: 1 34973 1.42 MB/sec execute 102 sec latency 16.956 ms oleg102-client.virtnet: 1 35631 1.45 MB/sec execute 103 sec latency 15.806 ms oleg102-client.virtnet: 1 35921 1.45 MB/sec execute 104 sec latency 17.685 ms oleg102-client.virtnet: 1 36236 1.44 MB/sec execute 105 sec latency 22.853 ms oleg102-client.virtnet: 1 36601 1.44 MB/sec execute 106 sec latency 15.646 ms oleg102-client.virtnet: 1 37019 1.46 MB/sec execute 107 sec latency 16.865 ms oleg102-client.virtnet: 1 37335 1.45 MB/sec oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1824 1285864 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36304 3569896 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13020 3592940 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49324 7162836 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg102-server Stopping /mnt/lustre-mds2 (opts:) on oleg102-server reboot facets: mds2 Failover mds2 to oleg102-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1796 1285892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36276 3570348 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12944 3593164 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49220 7163512 1% /mnt/lustre execute 108 sec latency 17.817 ms oleg102-client.virtnet: 1 37757 1.44 MB/sec execute 109 sec latency 10.944 ms oleg102-client.virtnet: 1 38223 1.47 MB/sec execute 110 sec latency 15.066 ms oleg102-client.virtnet: 1 38871 1.49 MB/sec execute 111 sec latency 17.431 ms oleg102-client.virtnet: 1 39307 1.52 MB/sec execute 112 sec latency 15.904 ms oleg102-client.virtnet: 1 39563 1.51 MB/sec execute 113 sec latency 17.100 ms oleg102-client.virtnet: 1 39914 1.50 MB/sec execute 114 sec latency 26.849 ms oleg102-client.virtnet: 1 40306 1.50 MB/sec execute 115 sec latency 17.826 ms oleg102-client.virtnet: 1 40722 1.51 MB/sec execute 116 sec latency 17.940 ms oleg102-client.virtnet: 1 41083 1.50 MB/sec execute 117 sec latency 17.616 ms oleg102-client.virtnet: 1 41475 1.50 MB/sec execute 118 sec latency 66.627 ms oleg102-client.virtnet: 1 41991 1.53 MB/sec execute 119 sec latency 17.036 ms oleg102-client.virtnet: 1 42620 1.56 MB/sec execute 120 sec latency 18.160 ms oleg102-client.virtnet: 1 42767 1.55 MB/sec execute 121 sec latency 657.161 ms oleg102-client.virtnet: 1 42767 1.54 MB/sec execute 122 sec latency 1657.351 ms oleg102-client.virtnet: 1 42767 1.53 MB/sec execute 123 sec latency 2657.702 ms oleg102-client.virtnet: 1 42767 1.52 MB/sec execute 124 sec latency 3657.983 ms oleg102-client.virtnet: 1 42767 1.50 MB/sec execute 125 sec latency 4658.138 ms oleg102-client.virtnet: 1 42767 1.49 MB/sec execute 126 sec latency 5658.381 ms oleg102-client.virtnet: 1 42767 1.48 MB/sec execute 127 sec latency 6658.616 ms oleg102-client.virtnet: 1 42767 1.47 MB/sec execute 128 sec latency 7658.913 ms oleg102-client.virtnet: 1 42767 1.46 MB/sec execute 129 sec latency 8659.268 ms oleg102-client.virtnet: 1 42767 1.45 MB/sec execute 130 sec latency 9659.606 ms oleg102-client.virtnet: 1 42767 1.43 MB/sec execute 131 sec latency 10659.954 ms oleg102-client.virtnet: 1 42767 1.42 MB/sec execute 132 sec latency 11660.217 ms oleg102-client.virtnet: 1 42767 1.41 MB/sec execute 133 sec latency 12660.506 ms oleg102-client.virtnet: 1 42767 1.40 MB/sec execute 134 sec latency 13660.808 ms oleg102-client.virtnet: 1 42767 1.39 MB/sec execute 135 sec latency 14661.155 ms oleg102-client.virtnet: 1 42767 1.38 MB/sec execute 136 sec latency 15661.499 ms oleg102-client.virtnet: 1 42767 1.37 MB/sec execute 137 sec latency 16661.823 ms oleg102-client.virtnet: 1 42767 1.36 MB/sec execute 138 sec latency 17662.066 ms oleg102-client.virtnet: 1 42767 1.35 MB/sec execute 139 sec latency 18662.216 ms oleg102-client.virtnet: 1 42767 1.34 MB/sec execute 140 sec latency 19662.399 ms oleg102-client.virtnet: 1 42767 1.33 MB/sec execute 141 sec latency 20662.598 ms oleg102-client.virtnet: 1 42767 1.32 MB/sec execute 142 sec latency 21662.861 ms oleg102-client.virtnet: 1 42767 1.31 MB/sec execute 143 sec latency 22663.068 ms oleg102-client.virtnet: 1 42767 1.31 MB/sec execute 144 sec latency 23663.317 ms oleg102-client.virtnet: 1 42767 1.30 MB/sec execute 145 sec latency 24663.563 ms oleg102-client.virtnet: 1 42767 1.29 MB/sec execute 146 sec latency 25663.792 ms oleg102-client.virtnet: 1 42921 1.29 MB/sec execute 147 sec latency 26062.975 ms oleg102-client.virtnet: 1 43175 1.28 MB/sec execute 148 sec latency 17.649 ms oleg102-client.virtnet: 1 43541 1.28 MB/sec execute 149 sec latency 21.657 ms oleg102-client.virtnet: 1 44032 1.30 MB/sec execute 150 sec latency 17.787 ms oleg102-client.virtnet: 1 44542 1.29 MB/sec execute 151 sec latency 12.332 ms oleg102-client.virtnet: 1 44955 1.29 MB/sec execute 152 sec latentest_70b fail mds1 7 times Failing mds1 on oleg102-server Stopping /mnt/lustre-mds1 (opts:) on oleg102-server reboot facets: mds1 Failover mds1 to oleg102-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1824 1285864 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36272 3570208 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12936 3593084 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49208 7163292 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg102-server Stopping /mnt/lustre-mds2 (opts:) on oleg102-server reboot facets: mds2 cy 11.075 ms oleg102-client.virtnet: 1 45372 1.31 MB/sec execute 153 sec latency 16.342 ms oleg102-client.virtnet: 1 46218 1.34 MB/sec execute 154 sec latency 10.984 ms oleg102-client.virtnet: 1 46491 1.34 MB/sec execute 155 sec latency 17.812 ms oleg102-client.virtnet: 1 46814 1.34 MB/sec execute 156 sec latency 16.321 ms oleg102-client.virtnet: 1 47091 1.33 MB/sec execute 157 sec latency 25.669 ms oleg102-client.virtnet: 1 47421 1.33 MB/sec execute 158 sec latency 31.317 ms oleg102-client.virtnet: 1 47806 1.34 MB/sec execute 159 sec latency 18.476 ms oleg102-client.virtnet: 1 48232 1.34 MB/sec execute 160 sec latency 16.661 ms oleg102-client.virtnet: 1 48601 1.34 MB/sec execute 161 sec latency 15.566 ms oleg102-client.virtnet: 1 49138 1.36 MB/sec execute 162 sec latency 18.048 ms oleg102-client.virtnet: 1 49760 1.39 MB/sec execute 163 sec latency 17.728 ms oleg102-client.virtnet: 1 50038 1.39 MB/sec execute 164 sec latency 17.499 ms oleg102-client.virtnet: 1 50355 1.38 MB/sec execute 165 sec latency 17.228 ms oleg102-client.virtnet: 1 50669 1.38 MB/sec execute 166 sec latency 20.240 ms oleg102-client.virtnet: 1 51090 1.39 MB/sec execute 167 sec latency 17.695 ms oleg102-client.virtnet: 1 51524 1.39 MB/sec execute 168 sec latency 16.546 ms oleg102-client.virtnet: 1 52080 1.39 MB/sec execute 169 sec latency 10.741 ms oleg102-client.virtnet: 1 52665 1.41 MB/sec execute 170 sec latency 16.038 ms oleg102-client.virtnet: 1 53309 1.43 MB/sec execute 171 sec latency 20.445 ms oleg102-client.virtnet: 1 53605 1.43 MB/sec execute 172 sec latency 16.596 ms oleg102-client.virtnet: 1 53913 1.43 MB/sec execute 173 sec latency 17.306 ms oleg102-client.virtnet: 1 54215 1.42 MB/sec execute 174 sec latency 21.452 ms oleg102-client.virtnet: 1 54661 1.44 MB/sec execute 175 sec latency 17.380 ms oleg102-client.virtnet: 1 55035 1.43 MB/sec execute 176 sec latency 17.104 ms oleg102-client.virtnet: 1 55522 1.43 MB/sec execute 177 sec latency 10.775 ms oleg102-client.virtnet: 1 56167 1.46 MB/sec execute 178 sec latency 17.886 ms oleg102-client.virtnet: 1 56782 1.48 MB/sec execute 179 sec latency 19.513 ms oleg102-client.virtnet: 1 57093 1.47 MB/sec execute 180 sec latency 19.427 ms oleg102-client.virtnet: 1 57469 1.47 MB/sec execute 181 sec latency 21.267 ms oleg102-client.virtnet: 1 57770 1.47 MB/sec execute 182 sec latency 18.800 ms oleg102-client.virtnet: 1 58233 1.48 MB/sec execute 183 sec latency 16.258 ms oleg102-client.virtnet: 1 58563 1.47 MB/sec execute 184 sec latency 17.073 ms oleg102-client.virtnet: 1 59055 1.47 MB/sec execute 185 sec latency 10.920 ms oleg102-client.virtnet: 1 59673 1.49 MB/sec execute 186 sec latency 16.501 ms oleg102-client.virtnet: 1 60291 1.51 MB/sec execute 187 sec latency 325.537 ms oleg102-client.virtnet: 1 60628 1.51 MB/sec execute 188 sec latency 18.193 ms oleg102-client.virtnet: 1 60844 1.51 MB/sec execute 189 sec latency 104.061 ms oleg102-client.virtnet: 1 60844 1.50 MB/sec execute 190 sec latency 1104.293 ms oleg102-client.virtnet: 1 60844 1.49 MB/sec execute 191 sec latency 2104.579 ms oleg102-client.virtnet: 1 60844 1.48 MB/sec execute 192 sec latency 3104.832 ms oleg102-client.virtnet: 1 60844 1.48 MB/sec execute 193 sec latency 4105.074 ms oleg102-client.virtnet: 1 60844 1.47 MB/sec execute 194 sec latency 5105.305 ms oleg102-client.virtnet: 1 60844 1.46 MB/sec execute 195 sec latency 6105.584 ms oleg102-client.virtnet: 1 60844 1.45 MB/sec execute 196 sec latency 7105.782 ms oleg102-client.virtnet: 1 60844 1.45 MB/sec executeFailover mds2 to oleg102-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1285896 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36432 3570080 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12784 3593492 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49216 7163572 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg102-server Stopping /mnt/lustre-mds1 (opts:) on oleg102-server reboot facets: mds1 Failover mds1 to oleg102-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 197 sec latency 8106.059 ms oleg102-client.virtnet: 1 60844 1.44 MB/sec execute 198 sec latency 9106.257 ms oleg102-client.virtnet: 1 60844 1.43 MB/sec execute 199 sec latency 10106.512 ms oleg102-client.virtnet: 1 60844 1.42 MB/sec execute 200 sec latency 11106.776 ms oleg102-client.virtnet: 1 60844 1.42 MB/sec execute 201 sec latency 12107.019 ms oleg102-client.virtnet: 1 60844 1.41 MB/sec execute 202 sec latency 13107.313 ms oleg102-client.virtnet: 1 60844 1.40 MB/sec execute 203 sec latency 14107.542 ms oleg102-client.virtnet: 1 60844 1.40 MB/sec execute 204 sec latency 15107.750 ms oleg102-client.virtnet: 1 60844 1.39 MB/sec execute 205 sec latency 16107.903 ms oleg102-client.virtnet: 1 60844 1.38 MB/sec execute 206 sec latency 17108.196 ms oleg102-client.virtnet: 1 60844 1.38 MB/sec execute 207 sec latency 18108.438 ms oleg102-client.virtnet: 1 60844 1.37 MB/sec execute 208 sec latency 19108.615 ms oleg102-client.virtnet: 1 60882 1.36 MB/sec execute 209 sec latency 19987.417 ms oleg102-client.virtnet: 1 61209 1.36 MB/sec execute 210 sec latency 17.795 ms oleg102-client.virtnet: 1 61533 1.36 MB/sec execute 211 sec latency 17.176 ms oleg102-client.virtnet: 1 61921 1.37 MB/sec execute 212 sec latency 17.199 ms oleg102-client.virtnet: 1 62271 1.36 MB/sec execute 213 sec latency 17.546 ms oleg102-client.virtnet: 1 62698 1.36 MB/sec execute 214 sec latency 13.060 ms oleg102-client.virtnet: 1 63523 1.39 MB/sec execute 215 sec latency 12.623 ms oleg102-client.virtnet: 1 64082 1.41 MB/sec execute 216 sec latency 20.126 ms oleg102-client.virtnet: 1 64305 1.40 MB/sec execute 217 sec latency 17.349 ms oleg102-client.virtnet: 1 64784 1.40 MB/sec execute 218 sec latency 17.643 ms oleg102-client.virtnet: 1 65279 1.41 MB/sec execute 219 sec latency 17.280 ms oleg102-client.virtnet: 1 65593 1.41 MB/sec execute 220 sec latency 17.670 ms oleg102-client.virtnet: 1 65968 1.40 MB/sec execute 221 sec latency 11.589 ms oleg102-client.virtnet: 1 66342 1.40 MB/sec execute 222 sec latency 16.523 ms oleg102-client.virtnet: 1 67057 1.42 MB/sec execute 223 sec latency 18.066 ms oleg102-client.virtnet: 1 67666 1.44 MB/sec execute 224 sec latency 18.052 ms oleg102-client.virtnet: 1 68201 1.44 MB/sec execute 225 sec latency 13.047 ms oleg102-client.virtnet: 1 68508 1.44 MB/sec execute 226 sec latency 18.588 ms oleg102-client.virtnet: 1 68913 1.45 MB/sec execute 227 sec latency 17.809 ms oleg102-client.virtnet: 1 69297 1.44 MB/sec execute 228 sec latency 14.819 ms oleg102-client.virtnet: 1 69755 1.44 MB/sec execute 229 sec latency 30.005 ms oleg102-client.virtnet: 1 70181 1.45 MB/sec execute 230 sec latency 15.781 ms oleg102-client.virtnet: 1 70795 1.46 MB/sec execute 231 sec latency 17.504 ms oleg102-client.virtnet: 1 71225 1.47 MB/sec execute 232 sec latency 18.197 ms oleg102-client.virtnet: 1 71460 1.47 MB/sec execute 233 sec latency 17.844 ms oleg102-client.virtnet: 1 71892 1.47 MB/sec execute 234 sec latency 13.393 ms oleg102-client.virtnet: 1 72389 1.48 MB/sec execute 235 sec latency 17.508 ms oleg102-client.virtnet: 1 72691 1.47 MB/sec execute 236 sec latency 18.266 ms oleg102-client.virtnet: 1 73158 1.47 MB/sec execute 237 sec latency 11.670 ms oleg102-client.virtnet: 1 73590 1.47 MB/sec execute 238 sec latency 15.539 ms oleg102-client.virtnet: 1 74268 1.49 MB/sec execute 239 sec latency 17.830 ms oleg102-client.virtnet: 1 74700 1.50 MB/sec execute 240 sec latency 13.164 ms oleg102-client.virtnet: 1 74953 1.50 MB/sec execute 241 sec latency 18.509 ms oleg102-client.virtnet: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 75273 1.49 MB/sec execute 242 sec latency 25.454 ms oleg102-client.virtnet: 1 75616 1.50 MB/sec execute 243 sec latency 17.145 ms oleg102-client.virtnet: 1 76050 1.50 MB/sec execute 244 sec latency 15.917 ms oleg102-client.virtnet: 1 76377 1.50 MB/sec execute 245 sec latency 18.128 ms oleg102-client.virtnet: 1 76809 1.49 MB/sec execute 246 sec latency 10.723 ms oleg102-client.virtnet: 1 77352 1.51 MB/sec execute 247 sec latency 12.471 ms oleg102-client.virtnet: 1 77939 1.52 MB/sec execute 248 sec latency 17.449 ms oleg102-client.virtnet: 1 78331 1.53 MB/sec execute 249 sec latency 18.239 ms oleg102-client.virtnet: 1 78549 1.52 MB/sec execute 250 sec latency 17.422 ms oleg102-client.virtnet: 1 78887 1.52 MB/sec execute 251 sec latency 17.687 ms oleg102-client.virtnet: 1 79230 1.52 MB/sec execute 252 sec latency 15.592 ms oleg102-client.virtnet: 1 79616 1.52 MB/sec execute 253 sec latency 18.402 ms oleg102-client.virtnet: 1 79927 1.52 MB/sec execute 254 sec latency 18.085 ms oleg102-client.virtnet: 1 80340 1.51 MB/sec execute 255 sec latency 11.051 ms oleg102-client.virtnet: 1 80694 1.52 MB/sec execute 256 sec latency 15.959 ms oleg102-client.virtnet: 1 81426 1.54 MB/sec execute 257 sec latency 17.323 ms oleg102-client.virtnet: 1 81836 1.55 MB/sec execute 258 sec latency 16.125 ms oleg102-client.virtnet: 1 82062 1.54 MB/sec execute 259 sec latency 18.493 ms oleg102-client.virtnet: 1 82382 1.54 MB/sec execute 260 sec latency 20.708 ms oleg102-client.virtnet: 1 82711 1.54 MB/sec execute 261 sec latency 19.230 ms oleg102-client.virtnet: 1 83244 1.55 MB/sec execute 262 sec latency 17.528 ms oleg102-client.virtnet: 1 83964 1.55 MB/sec execute 263 sec latency 10.767 ms oleg102-client.virtnet: 1 84392 1.56 MB/sec execute 264 sec latency 16.246 ms oleg102-client.virtnet: 1 85123 1.57 MB/sec execute 265 sec latency 18.316 ms oleg102-client.virtnet: 1 85637 1.57 MB/sec execute 266 sec latency 10.919 ms oleg102-client.virtnet: 1 85973 1.57 MB/sec execute 267 sec latency 30.130 ms oleg102-client.virtnet: 1 86323 1.57 MB/sec execute 268 sec latency 16.366 ms oleg102-client.virtnet: 1 86710 1.58 MB/sec execute 269 sec latency 17.941 ms oleg102-client.virtnet: 1 87020 1.57 MB/sec execute 270 sec latency 18.480 ms oleg102-client.virtnet: 1 87442 1.57 MB/sec execute 271 sec latency 11.535 ms oleg102-client.virtnet: 1 88112 1.59 MB/sec execute 272 sec latency 12.589 ms oleg102-client.virtnet: 1 88802 1.60 MB/sec execute 273 sec latency 13.867 ms oleg102-client.virtnet: 1 89069 1.60 MB/sec execute 274 sec latency 18.429 ms oleg102-client.virtnet: 1 89472 1.59 MB/sec execute 275 sec latency 14.688 ms oleg102-client.virtnet: 1 89880 1.59 MB/sec execute 276 sec latency 16.222 ms oleg102-client.virtnet: 1 90265 1.60 MB/sec execute 277 sec latency 17.750 ms oleg102-client.virtnet: 1 90570 1.59 MB/sec execute 278 sec latency 18.496 ms oleg102-client.virtnet: 1 90979 1.59 MB/sec execute 279 sec latency 11.328 ms oleg102-client.virtnet: 1 91335 1.59 MB/sec execute 280 sec latency 16.486 ms oleg102-client.virtnet: 1 92081 1.61 MB/sec execute 281 sec latency 18.199 ms oleg102-client.virtnet: 1 92551 1.62 MB/sec execute 282 sec latency 10.724 ms oleg102-client.virtnet: 1 92795 1.61 MB/sec execute 283 sec latency 17.009 ms oleg102-client.virtnet: 1 93158 1.61 MB/sec execute 284 sec latency 19.019 ms oleg102-client.virtnet: 1 93683 1.62 MB/sec execute 285 sec latency 17.961 ms oleg102-client.virtnet: 1 94002 1.62 MB/sec execute 286 sec latency 16.629 ms oleg102-client.virtnet: 1 94431 1.61 MB/sec execute 287 sec latency 10.778 ms oleg102-client.virtnet: 1 94828 1.61 MB/sec execute 288 sec latency 16.578 ms oleg102-client.virtnet: 1 95476 1.63 MB/sec execute 289 sec latency 17.614 ms oleg102-client.virtnet: 1 95973 1.64 MB/sec execute 290 sec latency 17.896 ms oleg102-client.virtnet: 1 96495 1.64 MB/sec execute 291 sec latency 25.080 ms oleg102-client.virtnet: 1 96826 1.63 MB/sec execute 292 sec latency 17.391 ms oleg102-client.virtnet: 1 97259 1.64 MB/sec execute 293 sec latency 18.487 ms oleg102-client.virtnet: 1 97796 1.64 MB/sec execute 294 sec latency 13.100 ms oleg102-client.virtnet: 1 98213 1.64 MB/sec execute 295 sec latency 15.982 ms oleg102-client.virtnet: 1 98732 1.65 MB/sec execute 296 sec latency 15.630 ms oleg102-client.virtnet: 1 99336 1.66 MB/sec execute 297 sec latency 17.821 ms oleg102-client.virtnet: 1 99634 1.66 MB/sec execute 298 sec latency 17.725 ms oleg102-client.virtnet: 1 99937 1.66 MB/sec execute 299 sec latency 17.418 ms oleg102-client.virtnet: 1 cleanup 300 sec oleg102-client.virtnet: 0 cleanup 300 sec oleg102-client.virtnet: oleg102-client.virtnet: Operation Count AvgLat MaxLat oleg102-client.virtnet: ---------------------------------------- oleg102-client.virtnet: NTCreateX 15936 13.889 26445.788 oleg102-client.virtnet: Close 11700 1.672 17.420 oleg102-client.virtnet: Rename 675 11.562 20.416 oleg102-client.virtnet: Unlink 3220 2.659 8.344 oleg102-client.virtnet: Qpathinfo 14445 1.986 15.813 oleg102-client.virtnet: Qfileinfo 2518 0.169 2.455 oleg102-client.virtnet: Qfsinfo 2644 0.214 31.296 oleg102-client.virtnet: Sfileinfo 1293 5.243 13.245 oleg102-client.virtnet: Find 5579 0.542 13.659 oleg102-client.virtnet: WriteX 7852 1.459 19.520 oleg102-client.virtnet: ReadX 25004 0.015 2.552 oleg102-client.virtnet: LockX 52 1.660 2.050 oleg102-client.virtnet: UnlockX 52 1.734 2.376 oleg102-client.virtnet: Flush 1114 9.539 655.722 oleg102-client.virtnet: oleg102-client.virtnet: Throughput 1.6563 MB/sec 1 clients 1 procs max_latency=26445.800 ms oleg102-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg102-client.virtnet at Tue Nov 5 11:50:13 EST 2024 with return code 0 oleg102-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg102-client.virtnet oleg102-client.virtnet: /mnt/lustre/d70b.replay-single/oleg102-client.virtnet /mnt/lustre/d70b.replay-single/oleg102-client.virtnet oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg102-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg102-client.virtnet: removed directory: 'clients/client0' oleg102-client.virtnet: removed directory: 'clients' oleg102-client.virtnet: removed 'client.txt' oleg102-client.virtnet: /mnt/lustre/d70b.replay-single/oleg102-client.virtnet oleg102-client.virtnet: dbench successfully finished