== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 00:10:28 (1722485428) Starting client oleg128-client.virtnet: -o user_xattr,flock oleg128-server@tcp:/lustre /mnt/lustre Started clients oleg128-client.virtnet: 192.168.201.128@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg128-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=19324 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2284 1285404 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1808 1285880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26024 3556276 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5340 3594700 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 31364 7150976 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg128-server Stopping /mnt/lustre-mds1 (opts:) on oleg128-server reboot facets: mds1 Failover mds1 to oleg128-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 oleg128-client.virtnet: looking for dbench program oleg128-client.virtnet: /usr/bin/dbench oleg128-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg128-client.virtnet oleg128-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg128-client.virtnet' oleg128-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg128-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg128-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg128-client.virtnet at Thu Aug 1 00:10:29 EDT 2024 oleg128-client.virtnet: waiting for dbench pid 19349 oleg128-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg128-client.virtnet: oleg128-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg128-client.virtnet: failed to create barrier semaphore oleg128-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg128-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg128-client.virtnet: releasing clients oleg128-client.virtnet: 1 287 10.70 MB/sec warmup 1 sec latency 22.639 ms oleg128-client.virtnet: 1 569 9.55 MB/sec warmup 2 sec latency 20.021 ms oleg128-client.virtnet: 1 916 7.26 MB/sec warmup 3 sec latency 17.973 ms oleg128-client.virtnet: 1 1383 6.50 MB/sec warmup 4 sec latency 16.673 ms oleg128-client.virtnet: 1 1719 5.25 MB/sec warmup 5 sec latency 18.676 ms oleg128-client.virtnet: 1 2128 4.44 MB/sec warmup 6 sec latency 11.838 ms oleg128-client.virtnet: 1 2428 4.01 MB/sec warmup 7 sec latency 198.163 ms oleg128-client.virtnet: 1 2428 3.51 MB/sec warmup 8 sec latency 1198.352 ms oleg128-client.virtnet: 1 2428 3.12 MB/sec warmup 9 sec latency 2198.544 ms oleg128-client.virtnet: 1 2428 2.81 MB/sec warmup 10 sec latency 3198.769 ms oleg128-client.virtnet: 1 2428 2.55 MB/sec warmup 11 sec latency 4198.984 ms oleg128-client.virtnet: 1 2428 2.34 MB/sec warmup 12 sec latency 5199.278 ms oleg128-client.virtnet: 1 2428 2.16 MB/sec warmup 13 sec latency 6199.492 ms oleg128-client.virtnet: 1 2428 2.01 MB/sec warmup 14 sec latency 7199.705 ms oleg128-client.virtnet: 1 2428 1.87 MB/sec warmup 15 sec latency 8199.923 ms oleg128-client.virtnet: 1 2428 1.76 MB/sec warmup 16 sec latency 9200.178 ms oleg128-client.virtnet: 1 2428 1.65 MB/sec warmup 17 sec latency 10200.383 ms oleg128-client.virtnet: 1 2428 1.56 MB/sec warmup 18 sec latency 11200.586 ms oleg128-client.virtnet: 1 2428 1.48 MB/sec warmup 19 sec latency 12200.780 ms oleg128-client.virtnet: 1 2428 1.40 MB/sec warmup 20 sec latency 13200.985 ms oleg128-client.virtnet: 1 2428 1.34 MB/sec warmup 21 sec latency 14201.182 ms oleg128-client.virtnet: 1 2428 1.28 MB/sec warmup 22 sec latency 15201.382 ms oleg128-client.virtnet: 1 2428 1.22 MB/sec warmup 23 sec latency 16201.605 ms oleg128-client.virtnet: 1 2428 1.17 MB/sec warmup 24 sec latency 17201.861 ms oleg128-client.virtnet: 1 2428 1.12 MB/sec warmup 25 sec latency 18202.103 ms oleg128-client.virtnet: 1 2428 1.08 MB/sec warmup 26 sec latency 19202.382 ms oleg128-client.virtnet: 1 2428 1.04 MB/sec warmup 27 sec latency 20202.612 ms oleg128-client.virtnet: 1 2428 1.00 MB/sec warmup 28 sec latency 21202.829 ms oleg128-client.virtnet: 1 2428 0.97 MB/sec warmup 29 sec latency 22203.059 ms oleg128-client.virtnet: 1 2428 0.94 MB/sec warmup 30 sec latency 23203.341 ms oleg128-client.virtnet: 1 2428 0.91 MB/sec warmup 31 sec latency 24203.586 ms oleg128-client.virtnet: 1 2428 0.88 MB/sec warmup 32 sec latency 25203.844 ms oleg128-client.virtnet: 1 2428 0.85 MB/sec warmup pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1285444 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1764 1285924 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37484 3569228 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11988 3592992 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49472 7162220 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg128-server Stopping /mnt/lustre-mds2 (opts:) on oleg128-server reboot facets: mds2 Failover mds2 to oleg128-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2228 1285460 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37408 3568572 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11936 3591028 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49344 7159600 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg128-server Stopping /mnt/lustre-mds1 (opts:) on oleg128-server reboot facets: mds1 33 sec latency 26204.051 ms oleg128-client.virtnet: 1 2428 0.83 MB/sec warmup 34 sec latency 27204.266 ms oleg128-client.virtnet: 1 2428 0.80 MB/sec warmup 35 sec latency 28204.471 ms oleg128-client.virtnet: 1 2428 0.78 MB/sec warmup 36 sec latency 29204.688 ms oleg128-client.virtnet: 1 2454 0.76 MB/sec warmup 37 sec latency 30079.017 ms oleg128-client.virtnet: 1 3012 0.88 MB/sec warmup 38 sec latency 16.892 ms oleg128-client.virtnet: 1 3645 0.99 MB/sec warmup 39 sec latency 16.936 ms oleg128-client.virtnet: 1 3923 1.00 MB/sec warmup 40 sec latency 16.853 ms oleg128-client.virtnet: 1 4224 0.98 MB/sec warmup 41 sec latency 17.734 ms oleg128-client.virtnet: 1 4503 0.97 MB/sec warmup 42 sec latency 18.520 ms oleg128-client.virtnet: 1 4944 1.05 MB/sec warmup 43 sec latency 16.591 ms oleg128-client.virtnet: 1 5250 1.03 MB/sec warmup 44 sec latency 18.451 ms oleg128-client.virtnet: 1 5602 1.02 MB/sec warmup 45 sec latency 17.258 ms oleg128-client.virtnet: 1 5995 1.03 MB/sec warmup 46 sec latency 15.629 ms oleg128-client.virtnet: 1 6550 1.12 MB/sec warmup 47 sec latency 16.709 ms oleg128-client.virtnet: 1 7190 1.20 MB/sec warmup 48 sec latency 16.462 ms oleg128-client.virtnet: 1 7464 1.21 MB/sec warmup 49 sec latency 17.609 ms oleg128-client.virtnet: 1 7764 1.19 MB/sec warmup 50 sec latency 19.382 ms oleg128-client.virtnet: 1 8046 1.18 MB/sec warmup 51 sec latency 18.912 ms oleg128-client.virtnet: 1 8405 1.19 MB/sec warmup 52 sec latency 17.259 ms oleg128-client.virtnet: 1 8792 1.22 MB/sec warmup 53 sec latency 17.529 ms oleg128-client.virtnet: 1 9162 1.20 MB/sec warmup 54 sec latency 16.811 ms oleg128-client.virtnet: 1 9546 1.21 MB/sec warmup 55 sec latency 15.977 ms oleg128-client.virtnet: 1 10088 1.28 MB/sec warmup 56 sec latency 16.575 ms oleg128-client.virtnet: 1 10709 1.35 MB/sec warmup 57 sec latency 17.237 ms oleg128-client.virtnet: 1 10976 1.34 MB/sec warmup 58 sec latency 17.551 ms oleg128-client.virtnet: 1 11273 1.33 MB/sec warmup 59 sec latency 17.246 ms oleg128-client.virtnet: 1 11900 1.32 MB/sec execute 1 sec latency 19.950 ms oleg128-client.virtnet: 1 12295 2.21 MB/sec execute 2 sec latency 17.850 ms oleg128-client.virtnet: 1 12315 1.48 MB/sec execute 3 sec latency 914.654 ms oleg128-client.virtnet: 1 12315 1.11 MB/sec execute 4 sec latency 1914.918 ms oleg128-client.virtnet: 1 12315 0.89 MB/sec execute 5 sec latency 2915.050 ms oleg128-client.virtnet: 1 12315 0.74 MB/sec execute 6 sec latency 3915.224 ms oleg128-client.virtnet: 1 12370 0.64 MB/sec execute 7 sec latency 4737.228 ms oleg128-client.virtnet: 1 12735 0.61 MB/sec execute 8 sec latency 11.729 ms oleg128-client.virtnet: 1 13101 0.71 MB/sec execute 9 sec latency 14.978 ms oleg128-client.virtnet: 1 13634 1.18 MB/sec execute 10 sec latency 18.168 ms oleg128-client.virtnet: 1 14222 1.52 MB/sec execute 11 sec latency 17.938 ms oleg128-client.virtnet: 1 14496 1.50 MB/sec execute 12 sec latency 17.887 ms oleg128-client.virtnet: 1 14727 1.41 MB/sec execute 13 sec latency 19.728 ms oleg128-client.virtnet: 1 14996 1.34 MB/sec execute 14 sec latency 229.562 ms oleg128-client.virtnet: 1 15175 1.28 MB/sec execute 15 sec latency 612.907 ms oleg128-client.virtnet: 1 15589 1.46 MB/sec execute 16 sec latency 17.405 ms oleg128-client.virtnet: 1 15890 1.39 MB/sec execute 17 sec latency 17.153 ms oleg128-client.virtnet: 1 15912 1.31 MB/sec execute 18 sec latency 924.909 ms oleg128-client.virtnet: 1 15912 1.24 MB/sec execute Failover mds1 to oleg128-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2204 1285484 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37320 3567012 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12084 3593984 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49404 7160996 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg128-server Stopping /mnt/lustre-mds2 (opts:) on oleg128-server reboot facets: mds2 Failover mds2 to oleg128-server mount facets: mds2 19 sec latency 1925.160 ms oleg128-client.virtnet: 1 15912 1.18 MB/sec execute 20 sec latency 2925.394 ms oleg128-client.virtnet: 1 15912 1.12 MB/sec execute 21 sec latency 3925.608 ms oleg128-client.virtnet: 1 15912 1.07 MB/sec execute 22 sec latency 4925.788 ms oleg128-client.virtnet: 1 15912 1.03 MB/sec execute 23 sec latency 5925.961 ms oleg128-client.virtnet: 1 15912 0.98 MB/sec execute 24 sec latency 6926.166 ms oleg128-client.virtnet: 1 15912 0.94 MB/sec execute 25 sec latency 7926.336 ms oleg128-client.virtnet: 1 15912 0.91 MB/sec execute 26 sec latency 8926.537 ms oleg128-client.virtnet: 1 15912 0.87 MB/sec execute 27 sec latency 9926.722 ms oleg128-client.virtnet: 1 15912 0.84 MB/sec execute 28 sec latency 10926.941 ms oleg128-client.virtnet: 1 15912 0.81 MB/sec execute 29 sec latency 11927.105 ms oleg128-client.virtnet: 1 15912 0.79 MB/sec execute 30 sec latency 12927.283 ms oleg128-client.virtnet: 1 15912 0.76 MB/sec execute 31 sec latency 13927.504 ms oleg128-client.virtnet: 1 15912 0.74 MB/sec execute 32 sec latency 14927.688 ms oleg128-client.virtnet: 1 15912 0.71 MB/sec execute 33 sec latency 15927.935 ms oleg128-client.virtnet: 1 15912 0.69 MB/sec execute 34 sec latency 16928.092 ms oleg128-client.virtnet: 1 15912 0.67 MB/sec execute 35 sec latency 17928.293 ms oleg128-client.virtnet: 1 15912 0.66 MB/sec execute 36 sec latency 18928.523 ms oleg128-client.virtnet: 1 15912 0.64 MB/sec execute 37 sec latency 19928.677 ms oleg128-client.virtnet: 1 15912 0.62 MB/sec execute 38 sec latency 20928.866 ms oleg128-client.virtnet: 1 15912 0.60 MB/sec execute 39 sec latency 21929.076 ms oleg128-client.virtnet: 1 15912 0.59 MB/sec execute 40 sec latency 22929.287 ms oleg128-client.virtnet: 1 16166 0.58 MB/sec execute 41 sec latency 23191.897 ms oleg128-client.virtnet: 1 16554 0.60 MB/sec execute 42 sec latency 15.320 ms oleg128-client.virtnet: 1 16972 0.68 MB/sec execute 43 sec latency 15.709 ms oleg128-client.virtnet: 1 17571 0.75 MB/sec execute 44 sec latency 18.232 ms oleg128-client.virtnet: 1 17975 0.82 MB/sec execute 45 sec latency 17.492 ms oleg128-client.virtnet: 1 18193 0.81 MB/sec execute 46 sec latency 20.408 ms oleg128-client.virtnet: 1 18517 0.80 MB/sec execute 47 sec latency 19.436 ms oleg128-client.virtnet: 1 18829 0.82 MB/sec execute 48 sec latency 17.709 ms oleg128-client.virtnet: 1 19255 0.87 MB/sec execute 49 sec latency 17.572 ms oleg128-client.virtnet: 1 19581 0.86 MB/sec execute 50 sec latency 17.411 ms oleg128-client.virtnet: 1 19990 0.85 MB/sec execute 51 sec latency 10.584 ms oleg128-client.virtnet: 1 20369 0.87 MB/sec execute 52 sec latency 15.623 ms oleg128-client.virtnet: 1 21050 0.98 MB/sec execute 53 sec latency 17.216 ms oleg128-client.virtnet: 1 21537 1.04 MB/sec execute 54 sec latency 16.655 ms oleg128-client.virtnet: 1 21764 1.03 MB/sec execute 55 sec latency 18.339 ms oleg128-client.virtnet: 1 22073 1.02 MB/sec execute 56 sec latency 18.539 ms oleg128-client.virtnet: 1 22384 1.03 MB/sec execute 57 sec latency 18.111 ms oleg128-client.virtnet: 1 22784 1.06 MB/sec execute 58 sec latency 17.686 ms oleg128-client.virtnet: 1 23113 1.05 MB/sec execute 59 sec latency 17.600 ms oleg128-client.virtnet: 1 23491 1.04 MB/sec execute 60 sec latency 10.793 ms oleg128-client.virtnet: 1 23883 1.06 MB/sec execute 61 sec latency 15.474 ms oleg128-client.virtnet: 1 24520 1.13 MB/sec execute 62 sec latency 17.562 ms oleg128-client.virtnet: 1 25008 1.18 MB/sec execute 63 sec latency 16.152 ms oleg1Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2228 1285460 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37476 3566612 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11876 3594152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49352 7160764 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg128-server Stopping /mnt/lustre-mds1 (opts:) on oleg128-server reboot facets: mds1 Failover mds1 to oleg128-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 28-client.virtnet: 1 25313 1.18 MB/sec execute 64 sec latency 17.259 ms oleg128-client.virtnet: 1 25655 1.17 MB/sec execute 65 sec latency 17.157 ms oleg128-client.virtnet: 1 26001 1.18 MB/sec execute 66 sec latency 16.987 ms oleg128-client.virtnet: 1 26396 1.21 MB/sec execute 67 sec latency 16.692 ms oleg128-client.virtnet: 1 26436 1.19 MB/sec execute 68 sec latency 922.879 ms oleg128-client.virtnet: 1 26436 1.17 MB/sec execute 69 sec latency 1923.085 ms oleg128-client.virtnet: 1 26436 1.16 MB/sec execute 70 sec latency 2923.292 ms oleg128-client.virtnet: 1 26680 1.14 MB/sec execute 71 sec latency 3152.018 ms oleg128-client.virtnet: 1 27077 1.13 MB/sec execute 72 sec latency 10.175 ms oleg128-client.virtnet: 1 27467 1.15 MB/sec execute 73 sec latency 16.028 ms oleg128-client.virtnet: 1 28141 1.22 MB/sec execute 74 sec latency 17.308 ms oleg128-client.virtnet: 1 28575 1.25 MB/sec execute 75 sec latency 13.284 ms oleg128-client.virtnet: 1 28828 1.25 MB/sec execute 76 sec latency 18.935 ms oleg128-client.virtnet: 1 29140 1.23 MB/sec execute 77 sec latency 18.209 ms oleg128-client.virtnet: 1 29243 1.22 MB/sec execute 78 sec latency 649.869 ms oleg128-client.virtnet: 1 29584 1.23 MB/sec execute 79 sec latency 17.381 ms oleg128-client.virtnet: 1 29985 1.25 MB/sec execute 80 sec latency 17.239 ms oleg128-client.virtnet: 1 30103 1.24 MB/sec execute 81 sec latency 617.935 ms oleg128-client.virtnet: 1 30103 1.22 MB/sec execute 82 sec latency 1618.126 ms oleg128-client.virtnet: 1 30103 1.21 MB/sec execute 83 sec latency 2618.310 ms oleg128-client.virtnet: 1 30103 1.19 MB/sec execute 84 sec latency 3618.491 ms oleg128-client.virtnet: 1 30103 1.18 MB/sec execute 85 sec latency 4618.694 ms oleg128-client.virtnet: 1 30103 1.16 MB/sec execute 86 sec latency 5618.882 ms oleg128-client.virtnet: 1 30103 1.15 MB/sec execute 87 sec latency 6619.047 ms oleg128-client.virtnet: 1 30103 1.14 MB/sec execute 88 sec latency 7619.253 ms oleg128-client.virtnet: 1 30103 1.12 MB/sec execute 89 sec latency 8619.433 ms oleg128-client.virtnet: 1 30103 1.11 MB/sec execute 90 sec latency 9619.661 ms oleg128-client.virtnet: 1 30103 1.10 MB/sec execute 91 sec latency 10619.859 ms oleg128-client.virtnet: 1 30103 1.09 MB/sec execute 92 sec latency 11620.052 ms oleg128-client.virtnet: 1 30103 1.08 MB/sec execute 93 sec latency 12620.262 ms oleg128-client.virtnet: 1 30103 1.06 MB/sec execute 94 sec latency 13620.452 ms oleg128-client.virtnet: 1 30103 1.05 MB/sec execute 95 sec latency 14620.685 ms oleg128-client.virtnet: 1 30103 1.04 MB/sec execute 96 sec latency 15620.901 ms oleg128-client.virtnet: 1 30103 1.03 MB/sec execute 97 sec latency 16621.114 ms oleg128-client.virtnet: 1 30103 1.02 MB/sec execute 98 sec latency 17621.299 ms oleg128-client.virtnet: 1 30103 1.01 MB/sec execute 99 sec latency 18621.494 ms oleg128-client.virtnet: 1 30103 1.00 MB/sec execute 100 sec latency 19621.670 ms oleg128-client.virtnet: 1 30103 0.99 MB/sec execute 101 sec latency 20621.887 ms oleg128-client.virtnet: 1 30103 0.98 MB/sec execute 102 sec latency 21622.113 ms oleg128-client.virtnet: 1 30194 0.97 MB/sec execute 103 sec latency 22284.718 ms oleg128-client.virtnet: 1 30579 0.97 MB/sec execute 104 sec latency 11.843 ms oleg128-client.virtnet: 1 30915 0.98 MB/sec execute 105 sec latency 20.262 ms oleg128-client.virtnet: 1 31591 1.02 MB/sec execute 106 sec latency 17.377 ms oleg128-client.virtnet: 1 32070 1.05 MB/sec execute 107 sec latency 17.819 ms oleg128-client.virtnet: 1 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2204 1285484 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37496 3566628 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11872 3594196 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49368 7160824 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg128-server Stopping /mnt/lustre-mds2 (opts:) on oleg128-server reboot facets: mds2 Failover mds2 to oleg128-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2228 1285460 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37460 3568812 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11752 3594304 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49212 7163116 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg128-server Stopping /mnt/lustre-mds1 (opts:) on oleg128-server reboot facets: mds1 32329 1.06 MB/sec execute 108 sec latency 17.687 ms oleg128-client.virtnet: 1 32625 1.05 MB/sec execute 109 sec latency 17.601 ms oleg128-client.virtnet: 1 32908 1.04 MB/sec execute 110 sec latency 17.377 ms oleg128-client.virtnet: 1 33233 1.05 MB/sec execute 111 sec latency 17.615 ms oleg128-client.virtnet: 1 33626 1.06 MB/sec execute 112 sec latency 17.271 ms oleg128-client.virtnet: 1 33986 1.06 MB/sec execute 113 sec latency 16.766 ms oleg128-client.virtnet: 1 34382 1.06 MB/sec execute 114 sec latency 15.105 ms oleg128-client.virtnet: 1 34929 1.10 MB/sec execute 115 sec latency 17.317 ms oleg128-client.virtnet: 1 35522 1.13 MB/sec execute 116 sec latency 17.444 ms oleg128-client.virtnet: 1 35811 1.13 MB/sec execute 117 sec latency 17.437 ms oleg128-client.virtnet: 1 36056 1.13 MB/sec execute 118 sec latency 18.006 ms oleg128-client.virtnet: 1 36394 1.12 MB/sec execute 119 sec latency 18.710 ms oleg128-client.virtnet: 1 36715 1.13 MB/sec execute 120 sec latency 17.613 ms oleg128-client.virtnet: 1 37105 1.14 MB/sec execute 121 sec latency 17.156 ms oleg128-client.virtnet: 1 37445 1.14 MB/sec execute 122 sec latency 17.878 ms oleg128-client.virtnet: 1 37867 1.14 MB/sec execute 123 sec latency 12.517 ms oleg128-client.virtnet: 1 38385 1.16 MB/sec execute 124 sec latency 15.447 ms oleg128-client.virtnet: 1 39012 1.20 MB/sec execute 125 sec latency 17.460 ms oleg128-client.virtnet: 1 39323 1.21 MB/sec execute 126 sec latency 17.152 ms oleg128-client.virtnet: 1 39560 1.20 MB/sec execute 127 sec latency 17.924 ms oleg128-client.virtnet: 1 39929 1.19 MB/sec execute 128 sec latency 17.113 ms oleg128-client.virtnet: 1 39989 1.19 MB/sec execute 129 sec latency 740.879 ms oleg128-client.virtnet: 1 39989 1.18 MB/sec execute 130 sec latency 1741.113 ms oleg128-client.virtnet: 1 39989 1.17 MB/sec execute 131 sec latency 2741.327 ms oleg128-client.virtnet: 1 39989 1.16 MB/sec execute 132 sec latency 3741.485 ms oleg128-client.virtnet: 1 40120 1.16 MB/sec execute 133 sec latency 4416.515 ms oleg128-client.virtnet: 1 40525 1.17 MB/sec execute 134 sec latency 17.318 ms oleg128-client.virtnet: 1 40843 1.17 MB/sec execute 135 sec latency 18.300 ms oleg128-client.virtnet: 1 41227 1.16 MB/sec execute 136 sec latency 14.542 ms oleg128-client.virtnet: 1 41611 1.17 MB/sec execute 137 sec latency 15.864 ms oleg128-client.virtnet: 1 42232 1.20 MB/sec execute 138 sec latency 17.331 ms oleg128-client.virtnet: 1 42720 1.22 MB/sec execute 139 sec latency 17.697 ms oleg128-client.virtnet: 1 42981 1.22 MB/sec execute 140 sec latency 17.827 ms oleg128-client.virtnet: 1 43276 1.22 MB/sec execute 141 sec latency 18.490 ms oleg128-client.virtnet: 1 43570 1.21 MB/sec execute 142 sec latency 18.541 ms oleg128-client.virtnet: 1 43829 1.21 MB/sec execute 143 sec latency 263.814 ms oleg128-client.virtnet: 1 43829 1.20 MB/sec execute 144 sec latency 1264.003 ms oleg128-client.virtnet: 1 43829 1.20 MB/sec execute 145 sec latency 2264.240 ms oleg128-client.virtnet: 1 43829 1.19 MB/sec execute 146 sec latency 3264.432 ms oleg128-client.virtnet: 1 43829 1.18 MB/sec execute 147 sec latency 4264.647 ms oleg128-client.virtnet: 1 43829 1.17 MB/sec execute 148 sec latency 5264.830 ms oleg128-client.virtnet: 1 43829 1.16 MB/sec execute 149 sec latency 6265.029 ms oleg128-client.virtnet: 1 43829 1.16 MB/sec execute 150 sec latency 7265.264 ms oleg128-client.virtnet: 1 43829 1.15 MB/sec execute 151 sec latency 8265.451 ms oleg128-client.virtnet: 1 43829 1.14 MB/sec execute 152 sec latency 9265.638 ms olegFailover mds1 to oleg128-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37604 3568800 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11776 3592332 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49380 7161132 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg128-server Stopping /mnt/lustre-mds2 (opts:) on oleg128-server reboot facets: mds2 Failover mds2 to oleg128-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0001 128-client.virtnet: 1 43829 1.13 MB/sec execute 153 sec latency 10265.831 ms oleg128-client.virtnet: 1 43829 1.13 MB/sec execute 154 sec latency 11265.961 ms oleg128-client.virtnet: 1 43829 1.12 MB/sec execute 155 sec latency 12266.173 ms oleg128-client.virtnet: 1 43829 1.11 MB/sec execute 156 sec latency 13266.362 ms oleg128-client.virtnet: 1 43829 1.10 MB/sec execute 157 sec latency 14266.554 ms oleg128-client.virtnet: 1 43829 1.10 MB/sec execute 158 sec latency 15266.760 ms oleg128-client.virtnet: 1 43829 1.09 MB/sec execute 159 sec latency 16266.973 ms oleg128-client.virtnet: 1 43829 1.08 MB/sec execute 160 sec latency 17267.213 ms oleg128-client.virtnet: 1 43829 1.08 MB/sec execute 161 sec latency 18267.416 ms oleg128-client.virtnet: 1 43829 1.07 MB/sec execute 162 sec latency 19267.645 ms oleg128-client.virtnet: 1 43829 1.06 MB/sec execute 163 sec latency 20267.860 ms oleg128-client.virtnet: 1 43829 1.06 MB/sec execute 164 sec latency 21268.016 ms oleg128-client.virtnet: 1 43829 1.05 MB/sec execute 165 sec latency 22268.209 ms oleg128-client.virtnet: 1 43829 1.04 MB/sec execute 166 sec latency 23268.401 ms oleg128-client.virtnet: 1 44093 1.06 MB/sec execute 167 sec latency 23719.829 ms oleg128-client.virtnet: 1 44411 1.05 MB/sec execute 168 sec latency 16.189 ms oleg128-client.virtnet: 1 44844 1.05 MB/sec execute 169 sec latency 11.346 ms oleg128-client.virtnet: 1 45205 1.06 MB/sec execute 170 sec latency 15.198 ms oleg128-client.virtnet: 1 45937 1.09 MB/sec execute 171 sec latency 16.866 ms oleg128-client.virtnet: 1 46356 1.11 MB/sec execute 172 sec latency 14.879 ms oleg128-client.virtnet: 1 46593 1.10 MB/sec execute 173 sec latency 16.446 ms oleg128-client.virtnet: 1 46877 1.10 MB/sec execute 174 sec latency 165.173 ms oleg128-client.virtnet: 1 47219 1.10 MB/sec execute 175 sec latency 17.216 ms oleg128-client.virtnet: 1 47666 1.11 MB/sec execute 176 sec latency 15.344 ms oleg128-client.virtnet: 1 48029 1.11 MB/sec execute 177 sec latency 15.940 ms oleg128-client.virtnet: 1 48515 1.11 MB/sec execute 178 sec latency 11.352 ms oleg128-client.virtnet: 1 49060 1.13 MB/sec execute 179 sec latency 14.175 ms oleg128-client.virtnet: 1 49713 1.15 MB/sec execute 180 sec latency 15.999 ms oleg128-client.virtnet: 1 50051 1.16 MB/sec execute 181 sec latency 14.154 ms oleg128-client.virtnet: 1 50410 1.15 MB/sec execute 182 sec latency 15.234 ms oleg128-client.virtnet: 1 50775 1.16 MB/sec execute 183 sec latency 14.857 ms oleg128-client.virtnet: 1 51256 1.17 MB/sec execute 184 sec latency 15.590 ms oleg128-client.virtnet: 1 51675 1.16 MB/sec execute 185 sec latency 16.928 ms oleg128-client.virtnet: 1 52086 1.16 MB/sec execute 186 sec latency 14.957 ms oleg128-client.virtnet: 1 52605 1.19 MB/sec execute 187 sec latency 15.611 ms oleg128-client.virtnet: 1 53235 1.21 MB/sec execute 188 sec latency 17.131 ms oleg128-client.virtnet: 1 53539 1.21 MB/sec execute 189 sec latency 16.357 ms oleg128-client.virtnet: 1 53819 1.20 MB/sec execute 190 sec latency 16.859 ms oleg128-client.virtnet: 1 54198 1.20 MB/sec execute 191 sec latency 15.991 ms oleg128-client.virtnet: 1 54513 1.20 MB/sec execute 192 sec latency 155.280 ms oleg128-client.virtnet: 1 54513 1.20 MB/sec execute 193 sec latency 1155.471 ms oleg128-client.virtnet: 1 54513 1.19 MB/sec execute 194 sec latency 2155.644 ms oleg128-client.virtnet: 1 54513 1.18 MB/sec execute 195 sec latency 3155.808 ms oleg128-client.virtnet: 1 54724 1.19 MB/sec execute 196 sec latency 3783.902 ms oleg128-client.virtnet: 1 55067 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2232 1285456 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37408 3568852 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11936 3592032 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49344 7160884 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg128-server Stopping /mnt/lustre-mds1 (opts:) on oleg128-server reboot facets: mds1 Failover mds1 to oleg128-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37304 3566984 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12080 3594004 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49384 7160988 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg128-server Stopping /mnt/lustre-mds2 (opts:) on oleg128-server reboot facets: mds2 1.19 MB/sec execute 197 sec latency 16.173 ms oleg128-client.virtnet: 1 55583 1.19 MB/sec execute 198 sec latency 10.403 ms oleg128-client.virtnet: 1 56109 1.21 MB/sec execute 199 sec latency 14.516 ms oleg128-client.virtnet: 1 56750 1.23 MB/sec execute 200 sec latency 15.733 ms oleg128-client.virtnet: 1 57083 1.23 MB/sec execute 201 sec latency 16.777 ms oleg128-client.virtnet: 1 57400 1.23 MB/sec execute 202 sec latency 16.774 ms oleg128-client.virtnet: 1 57551 1.22 MB/sec execute 203 sec latency 611.882 ms oleg128-client.virtnet: 1 57920 1.22 MB/sec execute 204 sec latency 16.576 ms oleg128-client.virtnet: 1 58354 1.23 MB/sec execute 205 sec latency 18.574 ms oleg128-client.virtnet: 1 58437 1.23 MB/sec execute 206 sec latency 771.239 ms oleg128-client.virtnet: 1 58437 1.22 MB/sec execute 207 sec latency 1771.407 ms oleg128-client.virtnet: 1 58437 1.22 MB/sec execute 208 sec latency 2771.572 ms oleg128-client.virtnet: 1 58437 1.21 MB/sec execute 209 sec latency 3771.750 ms oleg128-client.virtnet: 1 58437 1.20 MB/sec execute 210 sec latency 4771.917 ms oleg128-client.virtnet: 1 58437 1.20 MB/sec execute 211 sec latency 5772.071 ms oleg128-client.virtnet: 1 58437 1.19 MB/sec execute 212 sec latency 6772.259 ms oleg128-client.virtnet: 1 58437 1.19 MB/sec execute 213 sec latency 7772.430 ms oleg128-client.virtnet: 1 58437 1.18 MB/sec execute 214 sec latency 8772.650 ms oleg128-client.virtnet: 1 58437 1.18 MB/sec execute 215 sec latency 9772.837 ms oleg128-client.virtnet: 1 58437 1.17 MB/sec execute 216 sec latency 10773.012 ms oleg128-client.virtnet: 1 58437 1.17 MB/sec execute 217 sec latency 11773.187 ms oleg128-client.virtnet: 1 58437 1.16 MB/sec execute 218 sec latency 12773.330 ms oleg128-client.virtnet: 1 58437 1.16 MB/sec execute 219 sec latency 13773.531 ms oleg128-client.virtnet: 1 58437 1.15 MB/sec execute 220 sec latency 14773.763 ms oleg128-client.virtnet: 1 58437 1.14 MB/sec execute 221 sec latency 15774.034 ms oleg128-client.virtnet: 1 58437 1.14 MB/sec execute 222 sec latency 16774.263 ms oleg128-client.virtnet: 1 58437 1.13 MB/sec execute 223 sec latency 17774.456 ms oleg128-client.virtnet: 1 58437 1.13 MB/sec execute 224 sec latency 18774.638 ms oleg128-client.virtnet: 1 58437 1.12 MB/sec execute 225 sec latency 19774.836 ms oleg128-client.virtnet: 1 58437 1.12 MB/sec execute 226 sec latency 20775.089 ms oleg128-client.virtnet: 1 58437 1.11 MB/sec execute 227 sec latency 21775.291 ms oleg128-client.virtnet: 1 58437 1.11 MB/sec execute 228 sec latency 22775.484 ms oleg128-client.virtnet: 1 58437 1.10 MB/sec execute 229 sec latency 23775.683 ms oleg128-client.virtnet: 1 58631 1.10 MB/sec execute 230 sec latency 24108.095 ms oleg128-client.virtnet: 1 59072 1.10 MB/sec execute 231 sec latency 12.955 ms oleg128-client.virtnet: 1 59495 1.12 MB/sec execute 232 sec latency 15.562 ms oleg128-client.virtnet: 1 60128 1.13 MB/sec execute 233 sec latency 17.125 ms oleg128-client.virtnet: 1 60541 1.14 MB/sec execute 234 sec latency 15.877 ms oleg128-client.virtnet: 1 60766 1.14 MB/sec execute 235 sec latency 17.688 ms oleg128-client.virtnet: 1 61087 1.13 MB/sec execute 236 sec latency 21.216 ms oleg128-client.virtnet: 1 61419 1.13 MB/sec execute 237 sec latency 17.545 ms oleg128-client.virtnet: 1 61822 1.14 MB/sec execute 238 sec latency 17.127 ms oleg128-client.virtnet: 1 62143 1.14 MB/sec execute 239 sec latency 17.594 ms oleg128-client.virtnet: 1 62622 1.14 MB/sec execute 240 sec latency 11.450 ms oleg128-client.virtnet: 1 62951 1.14 MB/sec execute 241 Failover mds2 to oleg128-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec sec latency 15.524 ms oleg128-client.virtnet: 1 63666 1.16 MB/sec execute 242 sec latency 17.074 ms oleg128-client.virtnet: 1 64074 1.18 MB/sec execute 243 sec latency 16.766 ms oleg128-client.virtnet: 1 64305 1.17 MB/sec execute 244 sec latency 17.830 ms oleg128-client.virtnet: 1 64648 1.17 MB/sec execute 245 sec latency 19.611 ms oleg128-client.virtnet: 1 64962 1.17 MB/sec execute 246 sec latency 18.163 ms oleg128-client.virtnet: 1 65373 1.18 MB/sec execute 247 sec latency 17.337 ms oleg128-client.virtnet: 1 65696 1.17 MB/sec execute 248 sec latency 16.942 ms oleg128-client.virtnet: 1 66098 1.17 MB/sec execute 249 sec latency 10.496 ms oleg128-client.virtnet: 1 66472 1.18 MB/sec execute 250 sec latency 15.695 ms oleg128-client.virtnet: 1 67207 1.20 MB/sec execute 251 sec latency 17.079 ms oleg128-client.virtnet: 1 67634 1.21 MB/sec execute 252 sec latency 16.434 ms oleg128-client.virtnet: 1 67863 1.21 MB/sec execute 253 sec latency 16.811 ms oleg128-client.virtnet: 1 68210 1.20 MB/sec execute 254 sec latency 16.972 ms oleg128-client.virtnet: 1 68564 1.20 MB/sec execute 255 sec latency 15.406 ms oleg128-client.virtnet: 1 68703 1.20 MB/sec execute 256 sec latency 585.157 ms oleg128-client.virtnet: 1 68703 1.20 MB/sec execute 257 sec latency 1585.346 ms oleg128-client.virtnet: 1 68703 1.19 MB/sec execute 258 sec latency 2585.527 ms oleg128-client.virtnet: 1 68703 1.19 MB/sec execute 259 sec latency 3585.689 ms oleg128-client.virtnet: 1 69029 1.19 MB/sec execute 260 sec latency 3896.328 ms oleg128-client.virtnet: 1 69395 1.19 MB/sec execute 261 sec latency 17.202 ms oleg128-client.virtnet: 1 69807 1.19 MB/sec execute 262 sec latency 12.137 ms oleg128-client.virtnet: 1 70315 1.21 MB/sec execute 263 sec latency 15.423 ms oleg128-client.virtnet: 1 70930 1.22 MB/sec execute 264 sec latency 16.625 ms oleg128-client.virtnet: 1 71224 1.22 MB/sec execute 265 sec latency 18.336 ms oleg128-client.virtnet: 1 71451 1.22 MB/sec execute 266 sec latency 17.158 ms oleg128-client.virtnet: 1 71782 1.22 MB/sec execute 267 sec latency 22.814 ms oleg128-client.virtnet: 1 72126 1.22 MB/sec execute 268 sec latency 16.581 ms oleg128-client.virtnet: 1 72527 1.22 MB/sec execute 269 sec latency 16.943 ms oleg128-client.virtnet: 1 72835 1.22 MB/sec execute 270 sec latency 17.139 ms oleg128-client.virtnet: 1 73261 1.22 MB/sec execute 271 sec latency 11.049 ms oleg128-client.virtnet: 1 73618 1.22 MB/sec execute 272 sec latency 15.589 ms oleg128-client.virtnet: 1 74346 1.24 MB/sec execute 273 sec latency 16.859 ms oleg128-client.virtnet: 1 74764 1.25 MB/sec execute 274 sec latency 16.196 ms oleg128-client.virtnet: 1 74994 1.25 MB/sec execute 275 sec latency 17.530 ms oleg128-client.virtnet: 1 75333 1.25 MB/sec execute 276 sec latency 20.108 ms oleg128-client.virtnet: 1 75683 1.25 MB/sec execute 277 sec latency 16.810 ms oleg128-client.virtnet: 1 76074 1.25 MB/sec execute 278 sec latency 16.878 ms oleg128-client.virtnet: 1 76389 1.25 MB/sec execute 279 sec latency 17.831 ms oleg128-client.virtnet: 1 76818 1.25 MB/sec execute 280 sec latency 11.305 ms oleg128-client.virtnet: 1 77177 1.25 MB/sec execute 281 sec latency 14.912 ms oleg128-client.virtnet: 1 77893 1.27 MB/sec execute 282 sec latency 17.180 ms oleg128-client.virtnet: 1 78310 1.28 MB/sec execute 283 sec latency 16.605 ms oleg128-client.virtnet: 1 78539 1.28 MB/sec execute 284 sec latency 18.241 ms oleg128-client.virtnet: 1 78874 1.27 MB/sec execute 285 sec latency 17.659 ms oleg128-client.virtnet: 1 79197 1.27 MB/sec execute 286 sec latency 16.509 ms oleg128-client.virtnet: 1 79597 1.28 MB/sec execute 287 sec latency 17.879 ms oleg128-client.virtnet: 1 79921 1.28 MB/sec execute 288 sec latency 17.906 ms oleg128-client.virtnet: 1 80340 1.27 MB/sec execute 289 sec latency 10.577 ms oleg128-client.virtnet: 1 80694 1.28 MB/sec execute 290 sec latency 15.683 ms oleg128-client.virtnet: 1 81429 1.30 MB/sec execute 291 sec latency 16.827 ms oleg128-client.virtnet: 1 81854 1.31 MB/sec execute 292 sec latency 15.615 ms oleg128-client.virtnet: 1 82081 1.30 MB/sec execute 293 sec latency 18.063 ms oleg128-client.virtnet: 1 82426 1.30 MB/sec execute 294 sec latency 17.536 ms oleg128-client.virtnet: 1 82782 1.30 MB/sec execute 295 sec latency 12.808 ms oleg128-client.virtnet: 1 83182 1.31 MB/sec execute 296 sec latency 17.362 ms oleg128-client.virtnet: 1 83498 1.30 MB/sec execute 297 sec latency 17.789 ms oleg128-client.virtnet: 1 83938 1.30 MB/sec execute 298 sec latency 11.179 ms oleg128-client.virtnet: 1 84394 1.31 MB/sec execute 299 sec latency 15.336 ms oleg128-client.virtnet: 1 cleanup 300 sec oleg128-client.virtnet: 0 cleanup 300 sec oleg128-client.virtnet: oleg128-client.virtnet: Operation Count AvgLat MaxLat oleg128-client.virtnet: ---------------------------------------- oleg128-client.virtnet: NTCreateX 12654 12.608 24108.078 oleg128-client.virtnet: Close 9328 1.775 6.622 oleg128-client.virtnet: Rename 539 12.154 20.245 oleg128-client.virtnet: Unlink 2524 11.611 22284.698 oleg128-client.virtnet: Qpathinfo 11509 4.823 23719.818 oleg128-client.virtnet: Qfileinfo 2022 0.176 3.125 oleg128-client.virtnet: Qfsinfo 2098 0.288 6.631 oleg128-client.virtnet: Sfileinfo 1037 5.501 11.678 oleg128-client.virtnet: Find 4463 1.256 3151.996 oleg128-client.virtnet: WriteX 6285 1.523 13.817 oleg128-client.virtnet: ReadX 20002 0.017 2.462 oleg128-client.virtnet: LockX 42 1.850 2.122 oleg128-client.virtnet: UnlockX 42 1.912 2.266 oleg128-client.virtnet: Flush 883 10.823 649.850 oleg128-client.virtnet: oleg128-client.virtnet: Throughput 1.31321 MB/sec 1 clients 1 procs max_latency=24108.095 ms oleg128-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg128-client.virtnet at Thu Aug 1 00:16:30 EDT 2024 with return code 0 oleg128-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg128-client.virtnet oleg128-client.virtnet: /mnt/lustre/d70b.replay-single/oleg128-client.virtnet /mnt/lustre/d70b.replay-single/oleg128-client.virtnet oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg128-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg128-client.virtnet: removed directory: 'clients/client0' oleg128-client.virtnet: removed directory: 'clients' oleg128-client.virtnet: removed 'client.txt' oleg128-client.virtnet: /mnt/lustre/d70b.replay-single/oleg128-client.virtnet oleg128-client.virtnet: dbench successfully finished