== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 09:43:39 (1747403019) Starting client oleg423-client.virtnet: -o user_xattr,flock oleg423-server@tcp:/lustre /mnt/lustre Started clients oleg423-client.virtnet: 192.168.204.123@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg423-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=20381 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1285396 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1808 1285880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26024 3556276 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6420 3593312 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 32444 7149588 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server reboot facets: mds1 Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2248 1285440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1764 1285924 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26592 3569540 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6024 3590676 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 32616 7160216 1% /mnt/lustre test_70b fail mds2 2 times oleg423-client.virtnet: looking for dbench program oleg423-client.virtnet: /usr/bin/dbench oleg423-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg423-client.virtnet oleg423-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg423-client.virtnet' oleg423-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg423-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg423-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg423-client.virtnet at Fri May 16 09:43:40 EDT 2025 oleg423-client.virtnet: waiting for dbench pid 20406 oleg423-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg423-client.virtnet: oleg423-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg423-client.virtnet: failed to create barrier semaphore oleg423-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg423-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg423-client.virtnet: releasing clients oleg423-client.virtnet: 1 421 14.86 MB/sec warmup 1 sec latency 17.177 ms oleg423-client.virtnet: 1 1100 11.47 MB/sec warmup 2 sec latency 10.198 ms oleg423-client.virtnet: 1 1868 8.79 MB/sec warmup 3 sec latency 10.198 ms oleg423-client.virtnet: 1 2576 7.24 MB/sec warmup 4 sec latency 10.409 ms oleg423-client.virtnet: 1 3516 7.67 MB/sec warmup 5 sec latency 12.481 ms oleg423-client.virtnet: 1 3947 6.68 MB/sec warmup 6 sec latency 219.754 ms oleg423-client.virtnet: 1 3947 5.72 MB/sec warmup 7 sec latency 1219.993 ms oleg423-client.virtnet: 1 3947 5.01 MB/sec warmup 8 sec latency 2220.403 ms oleg423-client.virtnet: 1 3947 4.45 MB/sec warmup 9 sec latency 3220.731 ms oleg423-client.virtnet: 1 3947 4.01 MB/sec warmup 10 sec latency 4221.079 ms oleg423-client.virtnet: 1 3947 3.64 MB/sec warmup 11 sec latency 5221.384 ms oleg423-client.virtnet: 1 3947 3.34 MB/sec warmup 12 sec latency 6221.664 ms oleg423-client.virtnet: 1 3947 3.08 MB/sec warmup 13 sec latency 7221.883 ms oleg423-client.virtnet: 1 3947 2.86 MB/sec warmup 14 sec latency 8222.043 ms oleg423-client.virtnet: 1 3947 2.67 MB/sec warmup 15 sec latency 9222.264 ms oleg423-client.virtnet: 1 3947 2.50 MB/sec warmup 16 sec latency 10222.543 ms oleg423-client.virtnet: 1 3947 2.36 MB/sec warmup 17 sec latency 11222.698 ms oleg423-client.virtnet: 1 3947 2.23 MB/sec warmup 18 sec latency 12222.865 ms oleg423-client.virtnet: 1 3947 2.11 MB/sec warmup 19 sec latency 13223.067 ms oleg423-client.virtnet: 1 3947 2.00 MB/sec warmup 20 sec latency 14223.368 ms oleg423-client.virtnet: 1 3947 1.91 MB/sec warmup 21 sec latency 15223.694 ms oleg423-client.virtnet: 1 3947 1.82 MB/sec warmup 22 sec latency 16223.820 ms oleg423-client.virtnet: 1 3947 1.74 MB/sec warmup 23 sec latency 17223.999 ms oleg423-client.virtnet: 1 3947 1.67 MB/sec warmup 24 sec latency 18224.237 ms oleg423-client.virtnet: 1 3947 1.60 MB/sec warmup 25 sec latency 19224.610 ms oleg423-client.virtnet: 1 4302 1.55 MB/sec warmup 26 sec latency 19524.241 ms oleg423-client.virtnet: 1 4841 1.56 MB/sec warmup 27 sec latency 12.751 ms oleg423-client.virtnet: 1 5432 1.62 MB/sec warmup 28 sec latency 12.314 ms oleg423-client.virtnet: 1 6026 1.65 MB/sec warmup 29 sec latency 14.088 ms oleg423-client.virtnet: 1 7116 1.92 MB/sec warmup 30 sec latency 12.843 ms oleg423-client.virtnet: 1 7459 1.91 MB/sec warmup 31 sec latency 16.880 ms oleg423-client.virtnet: 1 7937 1.87 MB/sec warmup 32 sec latency 13.919 ms oleg423-client.virtnet: 1 8613 1.95 MB/sec warmup 33 sec latencFailing mds2 on oleg423-server Stopping /mnt/lustre-mds2 (opts:) on oleg423-server reboot facets: mds2 Failover mds2 to oleg423-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1285448 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36436 3569744 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12792 3593076 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49228 7162820 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server reboot facets: mds1 Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 y 11.931 ms oleg423-client.virtnet: 1 9268 1.91 MB/sec warmup 34 sec latency 10.221 ms oleg423-client.virtnet: 1 10268 2.07 MB/sec warmup 35 sec latency 9.005 ms oleg423-client.virtnet: 1 11022 2.17 MB/sec warmup 36 sec latency 8.890 ms oleg423-client.virtnet: 1 11625 2.14 MB/sec warmup 37 sec latency 14.204 ms oleg423-client.virtnet: 1 12298 2.20 MB/sec warmup 38 sec latency 10.249 ms oleg423-client.virtnet: 1 12988 2.18 MB/sec warmup 39 sec latency 13.270 ms oleg423-client.virtnet: 1 13662 2.27 MB/sec warmup 40 sec latency 12.180 ms oleg423-client.virtnet: 1 14329 2.34 MB/sec warmup 41 sec latency 12.657 ms oleg423-client.virtnet: 1 14602 2.32 MB/sec warmup 42 sec latency 18.124 ms oleg423-client.virtnet: 1 14938 2.27 MB/sec warmup 43 sec latency 20.942 ms oleg423-client.virtnet: 1 15281 2.26 MB/sec warmup 44 sec latency 19.332 ms oleg423-client.virtnet: 1 15761 2.28 MB/sec warmup 45 sec latency 16.822 ms oleg423-client.virtnet: 1 16336 2.24 MB/sec warmup 46 sec latency 11.873 ms oleg423-client.virtnet: 1 16758 2.24 MB/sec warmup 47 sec latency 15.224 ms oleg423-client.virtnet: 1 17818 2.40 MB/sec warmup 48 sec latency 9.872 ms oleg423-client.virtnet: 1 18254 2.38 MB/sec warmup 49 sec latency 19.691 ms oleg423-client.virtnet: 1 18254 2.33 MB/sec warmup 50 sec latency 1019.852 ms oleg423-client.virtnet: 1 18254 2.29 MB/sec warmup 51 sec latency 2020.060 ms oleg423-client.virtnet: 1 18254 2.24 MB/sec warmup 52 sec latency 3020.260 ms oleg423-client.virtnet: 1 18486 2.21 MB/sec warmup 53 sec latency 3717.014 ms oleg423-client.virtnet: 1 19208 2.25 MB/sec warmup 54 sec latency 10.140 ms oleg423-client.virtnet: 1 19918 2.22 MB/sec warmup 55 sec latency 8.404 ms oleg423-client.virtnet: 1 20698 2.31 MB/sec warmup 56 sec latency 10.043 ms oleg423-client.virtnet: 1 21485 2.36 MB/sec warmup 57 sec latency 16.648 ms oleg423-client.virtnet: 1 22057 2.35 MB/sec warmup 58 sec latency 10.796 ms oleg423-client.virtnet: 1 22479 2.33 MB/sec warmup 59 sec latency 384.657 ms oleg423-client.virtnet: 1 23654 1.46 MB/sec execute 1 sec latency 508.935 ms oleg423-client.virtnet: 1 23654 0.73 MB/sec execute 2 sec latency 1509.312 ms oleg423-client.virtnet: 1 23654 0.49 MB/sec execute 3 sec latency 2509.671 ms oleg423-client.virtnet: 1 23654 0.36 MB/sec execute 4 sec latency 3509.928 ms oleg423-client.virtnet: 1 23654 0.29 MB/sec execute 5 sec latency 4510.236 ms oleg423-client.virtnet: 1 23654 0.24 MB/sec execute 6 sec latency 5510.473 ms oleg423-client.virtnet: 1 23654 0.21 MB/sec execute 7 sec latency 6510.749 ms oleg423-client.virtnet: 1 23654 0.18 MB/sec execute 8 sec latency 7510.983 ms oleg423-client.virtnet: 1 23654 0.16 MB/sec execute 9 sec latency 8511.238 ms oleg423-client.virtnet: 1 23654 0.15 MB/sec execute 10 sec latency 9511.473 ms oleg423-client.virtnet: 1 23654 0.13 MB/sec execute 11 sec latency 10511.745 ms oleg423-client.virtnet: 1 23654 0.12 MB/sec execute 12 sec latency 11511.964 ms oleg423-client.virtnet: 1 23654 0.11 MB/sec execute 13 sec latency 12512.147 ms oleg423-client.virtnet: 1 23654 0.10 MB/sec execute 14 sec latency 13512.336 ms oleg423-client.virtnet: 1 23654 0.10 MB/sec execute 15 sec latency 14512.606 ms oleg423-client.virtnet: 1 23654 0.09 MB/sec execute 16 sec latency 15512.796 ms oleg423-client.virtnet: 1 23654 0.09 MB/sec execute 17 sec latency 16513.042 ms oleg423-client.virtnet: 1 23654 0.08 MB/sec execute 18 sec latency 17513.250 ms oleg423-client.virtnet: 1 23654 0.08 MB/sec exoleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2204 1285484 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36628 3570008 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12792 3592188 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49420 7162196 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg423-server Stopping /mnt/lustre-mds2 (opts:) on oleg423-server reboot facets: mds2 Failover mds2 to oleg423-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36456 3569960 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13084 3592204 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49540 7162164 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server reboot facets: mds1 ecute 19 sec latency 18513.407 ms oleg423-client.virtnet: 1 23654 0.07 MB/sec execute 20 sec latency 19513.557 ms oleg423-client.virtnet: 1 23654 0.07 MB/sec execute 21 sec latency 20513.702 ms oleg423-client.virtnet: 1 24240 0.32 MB/sec execute 22 sec latency 20607.129 ms oleg423-client.virtnet: 1 25003 0.54 MB/sec execute 23 sec latency 15.819 ms oleg423-client.virtnet: 1 25282 0.57 MB/sec execute 24 sec latency 18.500 ms oleg423-client.virtnet: 1 25648 0.56 MB/sec execute 25 sec latency 16.804 ms oleg423-client.virtnet: 1 26062 0.61 MB/sec execute 26 sec latency 17.090 ms oleg423-client.virtnet: 1 26568 0.70 MB/sec execute 27 sec latency 13.993 ms oleg423-client.virtnet: 1 27103 0.70 MB/sec execute 28 sec latency 10.645 ms oleg423-client.virtnet: 1 27478 0.75 MB/sec execute 29 sec latency 16.071 ms oleg423-client.virtnet: 1 28268 0.94 MB/sec execute 30 sec latency 15.646 ms oleg423-client.virtnet: 1 28705 1.05 MB/sec execute 31 sec latency 18.252 ms oleg423-client.virtnet: 1 29009 1.03 MB/sec execute 32 sec latency 16.046 ms oleg423-client.virtnet: 1 29323 1.02 MB/sec execute 33 sec latency 24.489 ms oleg423-client.virtnet: 1 29675 1.03 MB/sec execute 34 sec latency 17.632 ms oleg423-client.virtnet: 1 30152 1.09 MB/sec execute 35 sec latency 25.444 ms oleg423-client.virtnet: 1 30622 1.07 MB/sec execute 36 sec latency 11.679 ms oleg423-client.virtnet: 1 30999 1.10 MB/sec execute 37 sec latency 16.177 ms oleg423-client.virtnet: 1 31963 1.32 MB/sec execute 38 sec latency 16.512 ms oleg423-client.virtnet: 1 32616 1.33 MB/sec execute 39 sec latency 8.160 ms oleg423-client.virtnet: 1 33381 1.42 MB/sec execute 40 sec latency 8.555 ms oleg423-client.virtnet: 1 34127 1.41 MB/sec execute 41 sec latency 8.525 ms oleg423-client.virtnet: 1 35171 1.55 MB/sec execute 42 sec latency 10.076 ms oleg423-client.virtnet: 1 35937 1.65 MB/sec execute 43 sec latency 8.443 ms oleg423-client.virtnet: 1 36635 1.66 MB/sec execute 44 sec latency 8.290 ms oleg423-client.virtnet: 1 37262 1.70 MB/sec execute 45 sec latency 16.464 ms oleg423-client.virtnet: 1 37582 1.67 MB/sec execute 46 sec latency 497.750 ms oleg423-client.virtnet: 1 37582 1.63 MB/sec execute 47 sec latency 1497.937 ms oleg423-client.virtnet: 1 37582 1.60 MB/sec execute 48 sec latency 2498.157 ms oleg423-client.virtnet: 1 37878 1.59 MB/sec execute 49 sec latency 3184.943 ms oleg423-client.virtnet: 1 38882 1.71 MB/sec execute 50 sec latency 11.586 ms oleg423-client.virtnet: 1 39513 1.77 MB/sec execute 51 sec latency 14.879 ms oleg423-client.virtnet: 1 40199 1.77 MB/sec execute 52 sec latency 14.015 ms oleg423-client.virtnet: 1 40836 1.80 MB/sec execute 53 sec latency 12.018 ms oleg423-client.virtnet: 1 41536 1.81 MB/sec execute 54 sec latency 101.400 ms oleg423-client.virtnet: 1 42426 1.90 MB/sec execute 55 sec latency 380.462 ms oleg423-client.virtnet: 1 43088 1.95 MB/sec execute 56 sec latency 9.285 ms oleg423-client.virtnet: 1 43340 1.92 MB/sec execute 57 sec latency 657.768 ms oleg423-client.virtnet: 1 43340 1.89 MB/sec execute 58 sec latency 1657.930 ms oleg423-client.virtnet: 1 43340 1.86 MB/sec execute 59 sec latency 2658.092 ms oleg423-client.virtnet: 1 43340 1.83 MB/sec execute 60 sec latency 3658.218 ms oleg423-client.virtnet: 1 43340 1.80 MB/sec execute 61 sec latency 4658.405 ms oleg423-client.virtnet: 1 43340 1.77 MB/sec execute 62 sec latency 5658.545 ms oleg423-client.virtnet: 1 43340 1.74 MB/sec execute 63 sec latency 6658.692 ms oleg423-client.virtnet: 1 Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36436 3567748 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12792 3593000 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49228 7160748 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg423-server Stopping /mnt/lustre-mds2 (opts:) on oleg423-server reboot facets: mds2 Failover mds2 to oleg423-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 43340 1.71 MB/sec execute 64 sec latency 7658.840 ms oleg423-client.virtnet: 1 43340 1.69 MB/sec execute 65 sec latency 8659.050 ms oleg423-client.virtnet: 1 43340 1.66 MB/sec execute 66 sec latency 9659.290 ms oleg423-client.virtnet: 1 43340 1.64 MB/sec execute 67 sec latency 10659.415 ms oleg423-client.virtnet: 1 43340 1.61 MB/sec execute 68 sec latency 11659.515 ms oleg423-client.virtnet: 1 43340 1.59 MB/sec execute 69 sec latency 12659.664 ms oleg423-client.virtnet: 1 43340 1.57 MB/sec execute 70 sec latency 13659.824 ms oleg423-client.virtnet: 1 43340 1.54 MB/sec execute 71 sec latency 14660.017 ms oleg423-client.virtnet: 1 43340 1.52 MB/sec execute 72 sec latency 15660.150 ms oleg423-client.virtnet: 1 43340 1.50 MB/sec execute 73 sec latency 16660.276 ms oleg423-client.virtnet: 1 43340 1.48 MB/sec execute 74 sec latency 17660.511 ms oleg423-client.virtnet: 1 43340 1.46 MB/sec execute 75 sec latency 18660.997 ms oleg423-client.virtnet: 1 43340 1.44 MB/sec execute 76 sec latency 19661.274 ms oleg423-client.virtnet: 1 43340 1.42 MB/sec execute 77 sec latency 20661.424 ms oleg423-client.virtnet: 1 43865 1.43 MB/sec execute 78 sec latency 20836.097 ms oleg423-client.virtnet: 1 44611 1.45 MB/sec execute 79 sec latency 11.172 ms oleg423-client.virtnet: 1 45572 1.52 MB/sec execute 80 sec latency 8.423 ms oleg423-client.virtnet: 1 46478 1.58 MB/sec execute 81 sec latency 7.985 ms oleg423-client.virtnet: 1 47082 1.58 MB/sec execute 82 sec latency 9.984 ms oleg423-client.virtnet: 1 47840 1.61 MB/sec execute 83 sec latency 10.122 ms oleg423-client.virtnet: 1 48630 1.62 MB/sec execute 84 sec latency 7.784 ms oleg423-client.virtnet: 1 49679 1.72 MB/sec execute 85 sec latency 11.358 ms oleg423-client.virtnet: 1 50094 1.72 MB/sec execute 86 sec latency 17.375 ms oleg423-client.virtnet: 1 50503 1.70 MB/sec execute 87 sec latency 13.396 ms oleg423-client.virtnet: 1 51167 1.73 MB/sec execute 88 sec latency 15.806 ms oleg423-client.virtnet: 1 51563 1.72 MB/sec execute 89 sec latency 17.859 ms oleg423-client.virtnet: 1 52029 1.72 MB/sec execute 90 sec latency 12.000 ms oleg423-client.virtnet: 1 52588 1.75 MB/sec execute 91 sec latency 15.645 ms oleg423-client.virtnet: 1 53219 1.79 MB/sec execute 92 sec latency 18.004 ms oleg423-client.virtnet: 1 53541 1.79 MB/sec execute 93 sec latency 18.504 ms oleg423-client.virtnet: 1 53839 1.77 MB/sec execute 94 sec latency 17.276 ms oleg423-client.virtnet: 1 54170 1.76 MB/sec execute 95 sec latency 20.322 ms oleg423-client.virtnet: 1 54632 1.79 MB/sec execute 96 sec latency 16.171 ms oleg423-client.virtnet: 1 54955 1.77 MB/sec execute 97 sec latency 17.302 ms oleg423-client.virtnet: 1 55362 1.76 MB/sec execute 98 sec latency 10.752 ms oleg423-client.virtnet: 1 55782 1.76 MB/sec execute 99 sec latency 14.981 ms oleg423-client.virtnet: 1 56447 1.80 MB/sec execute 100 sec latency 15.645 ms oleg423-client.virtnet: 1 56842 1.82 MB/sec execute 101 sec latency 364.740 ms oleg423-client.virtnet: 1 56842 1.81 MB/sec execute 102 sec latency 1365.010 ms oleg423-client.virtnet: 1 56842 1.79 MB/sec execute 103 sec latency 2365.318 ms oleg423-client.virtnet: 1 56842 1.77 MB/sec execute 104 sec latency 3365.635 ms oleg423-client.virtnet: 1 56963 1.76 MB/sec execute 105 sec latency 3937.931 ms oleg423-client.virtnet: 1 57218 1.75 MB/sec execute 106 sec latency 18.413 ms oleg423-client.virtnet: 1 57694 1.74 MB/sec execute 107 sec latency 14.511 ms oleg423-client.virtnet: 1 58233 1.77 MB/sec execute 108 sec latUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36504 3570004 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12776 3593356 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49280 7163360 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server reboot facets: mds1 Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1285480 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36356 3569828 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13016 3590916 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49372 7160744 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg423-server Stopping /mnt/lustre-mds2 (opts:) on oleg423-server reboot facets: mds2 ency 15.461 ms oleg423-client.virtnet: 1 58611 1.75 MB/sec execute 109 sec latency 17.494 ms oleg423-client.virtnet: 1 59076 1.75 MB/sec execute 110 sec latency 11.791 ms oleg423-client.virtnet: 1 59693 1.79 MB/sec execute 111 sec latency 15.214 ms oleg423-client.virtnet: 1 59988 1.78 MB/sec execute 112 sec latency 492.415 ms oleg423-client.virtnet: 1 60464 1.80 MB/sec execute 113 sec latency 12.861 ms oleg423-client.virtnet: 1 60731 1.80 MB/sec execute 114 sec latency 19.552 ms oleg423-client.virtnet: 1 60782 1.78 MB/sec execute 115 sec latency 759.506 ms oleg423-client.virtnet: 1 60782 1.77 MB/sec execute 116 sec latency 1759.729 ms oleg423-client.virtnet: 1 60782 1.75 MB/sec execute 117 sec latency 2760.032 ms oleg423-client.virtnet: 1 60782 1.74 MB/sec execute 118 sec latency 3760.301 ms oleg423-client.virtnet: 1 60782 1.72 MB/sec execute 119 sec latency 4760.608 ms oleg423-client.virtnet: 1 60782 1.71 MB/sec execute 120 sec latency 5760.945 ms oleg423-client.virtnet: 1 60782 1.69 MB/sec execute 121 sec latency 6761.249 ms oleg423-client.virtnet: 1 60782 1.68 MB/sec execute 122 sec latency 7761.561 ms oleg423-client.virtnet: 1 60782 1.67 MB/sec execute 123 sec latency 8761.878 ms oleg423-client.virtnet: 1 60782 1.65 MB/sec execute 124 sec latency 9762.183 ms oleg423-client.virtnet: 1 60782 1.64 MB/sec execute 125 sec latency 10762.417 ms oleg423-client.virtnet: 1 60782 1.63 MB/sec execute 126 sec latency 11762.659 ms oleg423-client.virtnet: 1 60782 1.61 MB/sec execute 127 sec latency 12762.865 ms oleg423-client.virtnet: 1 60782 1.60 MB/sec execute 128 sec latency 13763.059 ms oleg423-client.virtnet: 1 60782 1.59 MB/sec execute 129 sec latency 14763.371 ms oleg423-client.virtnet: 1 60782 1.58 MB/sec execute 130 sec latency 15763.827 ms oleg423-client.virtnet: 1 60782 1.56 MB/sec execute 131 sec latency 16764.148 ms oleg423-client.virtnet: 1 60782 1.55 MB/sec execute 132 sec latency 17764.441 ms oleg423-client.virtnet: 1 60782 1.54 MB/sec execute 133 sec latency 18764.894 ms oleg423-client.virtnet: 1 60782 1.53 MB/sec execute 134 sec latency 19765.151 ms oleg423-client.virtnet: 1 60782 1.52 MB/sec execute 135 sec latency 20765.399 ms oleg423-client.virtnet: 1 60782 1.51 MB/sec execute 136 sec latency 21765.694 ms oleg423-client.virtnet: 1 60782 1.50 MB/sec execute 137 sec latency 22765.933 ms oleg423-client.virtnet: 1 60783 1.48 MB/sec execute 138 sec latency 23765.842 ms oleg423-client.virtnet: 1 61375 1.49 MB/sec execute 139 sec latency 13.201 ms oleg423-client.virtnet: 1 62008 1.50 MB/sec execute 140 sec latency 14.716 ms oleg423-client.virtnet: 1 62587 1.49 MB/sec execute 141 sec latency 16.541 ms oleg423-client.virtnet: 1 63510 1.54 MB/sec execute 142 sec latency 11.008 ms oleg423-client.virtnet: 1 64126 1.56 MB/sec execute 143 sec latency 16.781 ms oleg423-client.virtnet: 1 64562 1.56 MB/sec execute 144 sec latency 13.466 ms oleg423-client.virtnet: 1 65120 1.56 MB/sec execute 145 sec latency 10.621 ms oleg423-client.virtnet: 1 65634 1.57 MB/sec execute 146 sec latency 17.270 ms oleg423-client.virtnet: 1 66123 1.56 MB/sec execute 147 sec latency 10.336 ms oleg423-client.virtnet: 1 66644 1.59 MB/sec execute 148 sec latency 15.566 ms oleg423-client.virtnet: 1 67491 1.62 MB/sec execute 149 sec latency 16.017 ms oleg423-client.virtnet: 1 67836 1.62 MB/sec execute 150 sec latency 17.559 ms oleg423-client.virtnet: 1 68439 1.62 MB/sec execute 151 sec latency 10.221 ms oleg423-client.virtnet: 1 68913 1.63 MB/sec execute 152 sec latency 17.302 ms oleg423-client.Failover mds2 to oleg423-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2236 1285452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36268 3570128 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13012 3592924 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49280 7163052 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server reboot facets: mds1 Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 virtnet: 1 69450 1.63 MB/sec execute 153 sec latency 15.135 ms oleg423-client.virtnet: 1 70002 1.63 MB/sec execute 154 sec latency 13.199 ms oleg423-client.virtnet: 1 71000 1.68 MB/sec execute 155 sec latency 15.507 ms oleg423-client.virtnet: 1 71392 1.68 MB/sec execute 156 sec latency 17.733 ms oleg423-client.virtnet: 1 71837 1.67 MB/sec execute 157 sec latency 19.515 ms oleg423-client.virtnet: 1 72369 1.69 MB/sec execute 158 sec latency 16.406 ms oleg423-client.virtnet: 1 72854 1.68 MB/sec execute 159 sec latency 14.451 ms oleg423-client.virtnet: 1 73592 1.69 MB/sec execute 160 sec latency 8.528 ms oleg423-client.virtnet: 1 74749 1.75 MB/sec execute 161 sec latency 8.432 ms oleg423-client.virtnet: 1 74951 1.74 MB/sec execute 162 sec latency 550.091 ms oleg423-client.virtnet: 1 74951 1.73 MB/sec execute 163 sec latency 1550.360 ms oleg423-client.virtnet: 1 74951 1.72 MB/sec execute 164 sec latency 2550.641 ms oleg423-client.virtnet: 1 74951 1.71 MB/sec execute 165 sec latency 3550.981 ms oleg423-client.virtnet: 1 75293 1.70 MB/sec execute 166 sec latency 3839.075 ms oleg423-client.virtnet: 1 75986 1.72 MB/sec execute 167 sec latency 11.452 ms oleg423-client.virtnet: 1 76494 1.71 MB/sec execute 168 sec latency 12.966 ms oleg423-client.virtnet: 1 77000 1.71 MB/sec execute 169 sec latency 13.229 ms oleg423-client.virtnet: 1 77722 1.73 MB/sec execute 170 sec latency 15.283 ms oleg423-client.virtnet: 1 78295 1.76 MB/sec execute 171 sec latency 13.520 ms oleg423-client.virtnet: 1 78579 1.75 MB/sec execute 172 sec latency 16.846 ms oleg423-client.virtnet: 1 78948 1.74 MB/sec execute 173 sec latency 18.244 ms oleg423-client.virtnet: 1 79473 1.76 MB/sec execute 174 sec latency 16.537 ms oleg423-client.virtnet: 1 79514 1.75 MB/sec execute 175 sec latency 895.330 ms oleg423-client.virtnet: 1 79514 1.74 MB/sec execute 176 sec latency 1895.600 ms oleg423-client.virtnet: 1 79514 1.73 MB/sec execute 177 sec latency 2895.965 ms oleg423-client.virtnet: 1 79514 1.72 MB/sec execute 178 sec latency 3896.247 ms oleg423-client.virtnet: 1 79514 1.71 MB/sec execute 179 sec latency 4896.536 ms oleg423-client.virtnet: 1 79514 1.70 MB/sec execute 180 sec latency 5896.957 ms oleg423-client.virtnet: 1 79514 1.69 MB/sec execute 181 sec latency 6897.266 ms oleg423-client.virtnet: 1 79514 1.68 MB/sec execute 182 sec latency 7897.521 ms oleg423-client.virtnet: 1 79514 1.67 MB/sec execute 183 sec latency 8897.869 ms oleg423-client.virtnet: 1 79514 1.66 MB/sec execute 184 sec latency 9898.113 ms oleg423-client.virtnet: 1 79514 1.65 MB/sec execute 185 sec latency 10898.319 ms oleg423-client.virtnet: 1 79514 1.64 MB/sec execute 186 sec latency 11898.535 ms oleg423-client.virtnet: 1 79514 1.63 MB/sec execute 187 sec latency 12898.789 ms oleg423-client.virtnet: 1 79514 1.63 MB/sec execute 188 sec latency 13899.021 ms oleg423-client.virtnet: 1 79514 1.62 MB/sec execute 189 sec latency 14899.296 ms oleg423-client.virtnet: 1 79514 1.61 MB/sec execute 190 sec latency 15899.567 ms oleg423-client.virtnet: 1 79514 1.60 MB/sec execute 191 sec latency 16899.950 ms oleg423-client.virtnet: 1 79514 1.59 MB/sec execute 192 sec latency 17900.218 ms oleg423-client.virtnet: 1 79514 1.58 MB/sec execute 193 sec latency 18900.498 ms oleg423-client.virtnet: 1 79514 1.58 MB/sec execute 194 sec latency 19900.787 ms oleg423-client.virtnet: 1 79514 1.57 MB/sec execute 195 sec latency 20901.025 ms oleg423-client.virtnet: 1 79514 1.56 MB/sec execute 196 sec latency 21901.310 ms oleg423-client.virtnet: 1 79514 1.oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2212 1285476 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36292 3567964 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13108 3592948 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49400 7160912 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg423-server Stopping /mnt/lustre-mds2 (opts:) on oleg423-server reboot facets: mds2 Failover mds2 to oleg423-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1285444 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36664 3569844 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12812 3593216 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49476 7163060 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server reboot facets: mds1 55 MB/sec execute 197 sec latency 22901.650 ms oleg423-client.virtnet: 1 79514 1.54 MB/sec execute 198 sec latency 23902.002 ms oleg423-client.virtnet: 1 79514 1.54 MB/sec execute 199 sec latency 24902.360 ms oleg423-client.virtnet: 1 79514 1.53 MB/sec execute 200 sec latency 25902.735 ms oleg423-client.virtnet: 1 79514 1.52 MB/sec execute 201 sec latency 26902.853 ms oleg423-client.virtnet: 1 79514 1.51 MB/sec execute 202 sec latency 27903.070 ms oleg423-client.virtnet: 1 79514 1.51 MB/sec execute 203 sec latency 28903.316 ms oleg423-client.virtnet: 1 79514 1.50 MB/sec execute 204 sec latency 29903.531 ms oleg423-client.virtnet: 1 79566 1.49 MB/sec execute 205 sec latency 30800.180 ms oleg423-client.virtnet: 1 80270 1.49 MB/sec execute 206 sec latency 8.468 ms oleg423-client.virtnet: 1 81073 1.51 MB/sec execute 207 sec latency 15.988 ms oleg423-client.virtnet: 1 81882 1.54 MB/sec execute 208 sec latency 16.130 ms oleg423-client.virtnet: 1 82136 1.53 MB/sec execute 209 sec latency 18.205 ms oleg423-client.virtnet: 1 82775 1.53 MB/sec execute 210 sec latency 27.031 ms oleg423-client.virtnet: 1 83434 1.54 MB/sec execute 211 sec latency 11.084 ms oleg423-client.virtnet: 1 84104 1.54 MB/sec execute 212 sec latency 10.065 ms oleg423-client.virtnet: 1 85257 1.58 MB/sec execute 213 sec latency 9.209 ms oleg423-client.virtnet: 1 85631 1.58 MB/sec execute 214 sec latency 15.189 ms oleg423-client.virtnet: 1 86048 1.58 MB/sec execute 215 sec latency 17.812 ms oleg423-client.virtnet: 1 86794 1.59 MB/sec execute 216 sec latency 8.909 ms oleg423-client.virtnet: 1 87475 1.59 MB/sec execute 217 sec latency 9.103 ms oleg423-client.virtnet: 1 88073 1.61 MB/sec execute 218 sec latency 16.250 ms oleg423-client.virtnet: 1 88735 1.63 MB/sec execute 219 sec latency 17.738 ms oleg423-client.virtnet: 1 89085 1.63 MB/sec execute 220 sec latency 17.362 ms oleg423-client.virtnet: 1 89557 1.62 MB/sec execute 221 sec latency 16.270 ms oleg423-client.virtnet: 1 90296 1.64 MB/sec execute 222 sec latency 11.502 ms oleg423-client.virtnet: 1 91072 1.64 MB/sec execute 223 sec latency 9.409 ms oleg423-client.virtnet: 1 92131 1.66 MB/sec execute 224 sec latency 8.680 ms oleg423-client.virtnet: 1 92735 1.68 MB/sec execute 225 sec latency 11.061 ms oleg423-client.virtnet: 1 93380 1.68 MB/sec execute 226 sec latency 10.273 ms oleg423-client.virtnet: 1 94080 1.68 MB/sec execute 227 sec latency 10.985 ms oleg423-client.virtnet: 1 94682 1.68 MB/sec execute 228 sec latency 278.378 ms oleg423-client.virtnet: 1 94682 1.68 MB/sec execute 229 sec latency 1278.574 ms oleg423-client.virtnet: 1 94682 1.67 MB/sec execute 230 sec latency 2278.777 ms oleg423-client.virtnet: 1 94682 1.66 MB/sec execute 231 sec latency 3278.952 ms oleg423-client.virtnet: 1 95292 1.68 MB/sec execute 232 sec latency 3529.716 ms oleg423-client.virtnet: 1 96161 1.70 MB/sec execute 233 sec latency 9.270 ms oleg423-client.virtnet: 1 96748 1.70 MB/sec execute 234 sec latency 13.013 ms oleg423-client.virtnet: 1 97310 1.71 MB/sec execute 235 sec latency 15.772 ms oleg423-client.virtnet: 1 97830 1.70 MB/sec execute 236 sec latency 10.317 ms oleg423-client.virtnet: 1 98295 1.70 MB/sec execute 237 sec latency 57.998 ms oleg423-client.virtnet: 1 98696 1.71 MB/sec execute 238 sec latency 523.315 ms oleg423-client.virtnet: 1 99306 1.73 MB/sec execute 239 sec latency 18.225 ms oleg423-client.virtnet: 1 99625 1.73 MB/sec execute 240 sec latency 17.717 ms oleg423-client.virtnet: 1 99662 1.72 MB/sec execute 241 sec latency 815.303 ms oleg423-client.Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec virtnet: 1 99662 1.71 MB/sec execute 242 sec latency 1815.494 ms oleg423-client.virtnet: 1 99662 1.71 MB/sec execute 243 sec latency 2815.736 ms oleg423-client.virtnet: 1 99662 1.70 MB/sec execute 244 sec latency 3815.995 ms oleg423-client.virtnet: 1 99662 1.69 MB/sec execute 245 sec latency 4816.306 ms oleg423-client.virtnet: 1 99662 1.69 MB/sec execute 246 sec latency 5816.551 ms oleg423-client.virtnet: 1 99662 1.68 MB/sec execute 247 sec latency 6816.817 ms oleg423-client.virtnet: 1 99662 1.67 MB/sec execute 248 sec latency 7817.157 ms oleg423-client.virtnet: 1 99662 1.67 MB/sec execute 249 sec latency 8817.426 ms oleg423-client.virtnet: 1 99662 1.66 MB/sec execute 250 sec latency 9817.670 ms oleg423-client.virtnet: 1 99662 1.65 MB/sec execute 251 sec latency 10817.810 ms oleg423-client.virtnet: 1 99662 1.65 MB/sec execute 252 sec latency 11817.998 ms oleg423-client.virtnet: 1 99662 1.64 MB/sec execute 253 sec latency 12818.262 ms oleg423-client.virtnet: 1 99662 1.63 MB/sec execute 254 sec latency 13818.432 ms oleg423-client.virtnet: 1 99662 1.63 MB/sec execute 255 sec latency 14818.601 ms oleg423-client.virtnet: 1 99662 1.62 MB/sec execute 256 sec latency 15818.803 ms oleg423-client.virtnet: 1 99662 1.61 MB/sec execute 257 sec latency 16819.125 ms oleg423-client.virtnet: 1 99662 1.61 MB/sec execute 258 sec latency 17819.308 ms oleg423-client.virtnet: 1 99662 1.60 MB/sec execute 259 sec latency 18819.502 ms oleg423-client.virtnet: 1 99662 1.60 MB/sec execute 260 sec latency 19819.678 ms oleg423-client.virtnet: 1 99662 1.59 MB/sec execute 261 sec latency 20819.926 ms oleg423-client.virtnet: 1 99662 1.58 MB/sec execute 262 sec latency 21820.134 ms oleg423-client.virtnet: 1 99662 1.58 MB/sec execute 263 sec latency 22820.361 ms oleg423-client.virtnet: 1 99662 1.57 MB/sec execute 264 sec latency 23820.633 ms oleg423-client.virtnet: 1 99662 1.57 MB/sec execute 265 sec latency 24820.855 ms oleg423-client.virtnet: 1 99662 1.56 MB/sec execute 266 sec latency 25821.035 ms oleg423-client.virtnet: 1 99662 1.55 MB/sec execute 267 sec latency 26821.301 ms oleg423-client.virtnet: 1 99662 1.55 MB/sec execute 268 sec latency 27821.531 ms oleg423-client.virtnet: 1 99662 1.54 MB/sec execute 269 sec latency 28821.812 ms oleg423-client.virtnet: 1 99662 1.54 MB/sec execute 270 sec latency 29822.056 ms oleg423-client.virtnet: 1 99723 1.53 MB/sec execute 271 sec latency 30602.451 ms oleg423-client.virtnet: 1 100045 1.53 MB/sec execute 272 sec latency 17.687 ms oleg423-client.virtnet: 1 100420 1.53 MB/sec execute 273 sec latency 16.898 ms oleg423-client.virtnet: 1 100867 1.53 MB/sec execute 274 sec latency 17.087 ms oleg423-client.virtnet: 1 101181 1.53 MB/sec execute 275 sec latency 17.034 ms oleg423-client.virtnet: 1 101620 1.53 MB/sec execute 276 sec latency 10.935 ms oleg423-client.virtnet: 1 102228 1.54 MB/sec execute 277 sec latency 14.344 ms oleg423-client.virtnet: 1 102925 1.56 MB/sec execute 278 sec latency 17.507 ms oleg423-client.virtnet: 1 103343 1.56 MB/sec execute 279 sec latency 13.444 ms oleg423-client.virtnet: 1 104003 1.56 MB/sec execute 280 sec latency 9.302 ms oleg423-client.virtnet: 1 104730 1.56 MB/sec execute 281 sec latency 9.414 ms oleg423-client.virtnet: 1 105385 1.57 MB/sec execute 282 sec latency 9.109 ms oleg423-client.virtnet: 1 106450 1.60 MB/sec execute 283 sec latency 10.954 ms oleg423-client.virtnet: 1 106874 1.60 MB/sec execute 284 sec latency 14.773 ms oleg423-client.virtnet: 1 107378 1.59 MB/sec execute 285 sec latency 13.705 ms oleg423-client.virtnet: 1 108041 1.60 MB/sec execute 286 sec latency 12.478 ms oleg423-client.virtnet: 1 108662 1.60 MB/sec execute 287 sec latency 16.077 ms oleg423-client.virtnet: 1 109473 1.62 MB/sec execute 288 sec latency 11.689 ms oleg423-client.virtnet: 1 110156 1.63 MB/sec execute 289 sec latency 11.908 ms oleg423-client.virtnet: 1 110543 1.63 MB/sec execute 290 sec latency 15.692 ms oleg423-client.virtnet: 1 111140 1.63 MB/sec execute 291 sec latency 15.479 ms oleg423-client.virtnet: 1 111791 1.64 MB/sec execute 292 sec latency 11.309 ms oleg423-client.virtnet: 1 112202 1.63 MB/sec execute 293 sec latency 11.157 ms oleg423-client.virtnet: 1 112596 1.63 MB/sec execute 294 sec latency 16.967 ms oleg423-client.virtnet: 1 113361 1.65 MB/sec execute 295 sec latency 17.164 ms oleg423-client.virtnet: 1 113796 1.66 MB/sec execute 296 sec latency 18.059 ms oleg423-client.virtnet: 1 114043 1.66 MB/sec execute 297 sec latency 16.892 ms oleg423-client.virtnet: 1 114424 1.65 MB/sec execute 298 sec latency 27.854 ms oleg423-client.virtnet: 1 114766 1.65 MB/sec execute 299 sec latency 17.253 ms oleg423-client.virtnet: 1 cleanup 300 sec oleg423-client.virtnet: 0 cleanup 300 sec oleg423-client.virtnet: oleg423-client.virtnet: Operation Count AvgLat MaxLat oleg423-client.virtnet: ---------------------------------------- oleg423-client.virtnet: NTCreateX 15884 9.200 30800.167 oleg423-client.virtnet: Close 11661 5.539 30602.410 oleg423-client.virtnet: Rename 675 8.242 19.049 oleg423-client.virtnet: Unlink 3221 2.002 5.815 oleg423-client.virtnet: Qpathinfo 14423 1.926 3937.900 oleg423-client.virtnet: Qfileinfo 2511 0.134 2.416 oleg423-client.virtnet: Qfsinfo 2652 0.205 4.119 oleg423-client.virtnet: Sfileinfo 1300 19.626 20836.078 oleg423-client.virtnet: Find 5580 1.043 3529.705 oleg423-client.virtnet: WriteX 7869 1.060 11.912 oleg423-client.virtnet: ReadX 24985 0.012 0.172 oleg423-client.virtnet: LockX 52 1.347 6.914 oleg423-client.virtnet: UnlockX 52 1.238 2.026 oleg423-client.virtnet: Flush 1118 7.059 523.291 oleg423-client.virtnet: oleg423-client.virtnet: Throughput 1.65165 MB/sec 1 clients 1 procs max_latency=30800.180 ms oleg423-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg423-client.virtnet at Fri May 16 09:49:41 EDT 2025 with return code 0 oleg423-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg423-client.virtnet oleg423-client.virtnet: /mnt/lustre/d70b.replay-single/oleg423-client.virtnet /mnt/lustre/d70b.replay-single/oleg423-client.virtnet oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg423-client.virtnet: removed directory: 'clients/client0' oleg423-client.virtnet: removed directory: 'clients' oleg423-client.virtnet: removed 'client.txt' oleg423-client.virtnet: /mnt/lustre/d70b.replay-single/oleg423-client.virtnet oleg423-client.virtnet: dbench successfully finished