== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 17:04:15 (1717016655) Starting client oleg416-client.virtnet: -o user_xattr,flock oleg416-server@tcp:/lustre /mnt/lustre Started clients oleg416-client.virtnet: 192.168.204.116@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg416-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=12045 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3556144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7161596 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 17:04:25 (1717016665) shut down Failover mds1 to oleg416-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 17:04:42 (1717016682) targets are mounted 17:04:42 (1717016682) facet_failover done oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2172 1285516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 27440 3568916 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1880 3593056 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 29320 7161972 1% /mnt/lustre oleg416-client.virtnet: looking for dbench program oleg416-client.virtnet: /usr/bin/dbench oleg416-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg416-client.virtnet oleg416-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg416-client.virtnet' oleg416-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg416-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg416-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg416-client.virtnet at Wed May 29 17:04:17 EDT 2024 oleg416-client.virtnet: waiting for dbench pid 12070 oleg416-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg416-client.virtnet: oleg416-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg416-client.virtnet: failed to create barrier semaphore oleg416-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg416-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg416-client.virtnet: releasing clients oleg416-client.virtnet: 1 199 7.82 MB/sec warmup 1 sec latency 51.489 ms oleg416-client.virtnet: 1 405 7.16 MB/sec warmup 2 sec latency 37.717 ms oleg416-client.virtnet: 1 669 7.05 MB/sec warmup 3 sec latency 33.754 ms oleg416-client.virtnet: 1 902 5.42 MB/sec warmup 4 sec latency 35.418 ms oleg416-client.virtnet: 1 1161 4.60 MB/sec warmup 5 sec latency 26.464 ms oleg416-client.virtnet: 1 1503 4.34 MB/sec warmup 6 sec latency 23.005 ms oleg416-client.virtnet: 1 1911 3.78 MB/sec warmup 7 sec latency 25.720 ms oleg416-client.virtnet: 1 2418 3.51 MB/sec warmup 8 sec latency 26.006 ms oleg416-client.virtnet: 1 2931 3.72 MB/sec warmup 9 sec latency 29.431 ms oleg416-client.virtnet: 1 3562 3.84 MB/sec warmup 10 sec latency 22.073 ms oleg416-client.virtnet: 1 3923 3.64 MB/sec warmup 11 sec latency 20.801 ms oleg416-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 347.799 ms oleg416-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 1348.115 ms oleg416-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 2348.326 ms oleg416-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 3348.537 ms oleg416-client.virtnet: 1 4083 2.50 MB/sec warmup 16 sec latency 4348.805 ms oleg416-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 5349.308 ms oleg416-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 6349.531 ms oleg416-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 7350.544 ms oleg416-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 8351.309 ms oleg416-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 9351.697 ms oleg416-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 10352.124 ms oleg416-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 11352.374 ms oleg416-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 12355.413 ms oleg416-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 13355.607 ms oleg416-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 14356.237 ms oleg416-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 15356.664 ms oleg416-client.virtnet: 1 4123 1.43 MB/sec warmup 28 sec latency 16139.032 ms oleg416-client.virtnet: 1 4362 1.40 MB/sec warmup 29 sec latency 61.389 ms oleg416-client.virtnet: 1 4609 1.40 MB/sec warmup 30 sec latency 37.107 ms oleg416-client.virtnet: 1 4967 1.46 MB/sec warmup 31 sec latency 34.128 ms oleg416-client.virtnet: 1 5410 1.42 MB/sec warmup 32 sec latency 26.369 ms oleg416-client.virtnet: 1 5937 1.43 MB/sec warmup 33 sec latency 30.649 ms test_70b fail mds2 2 times Failing mds2 on oleg416-server Stopping /mnt/lustre-mds2 (opts:) on oleg416-server 17:04:57 (1717016697) shut down Failover mds2 to oleg416-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0001 17:05:14 (1717016714) targets are mounted 17:05:14 (1717016714) facet_failover done oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3570560 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13088 3593640 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7164200 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 17:05:29 (1717016729) shut down oleg416-client.virtnet: 1 6125 1.41 MB/sec warmup 34 sec latency 36.303 ms oleg416-client.virtnet: 1 6477 1.50 MB/sec warmup 35 sec latency 91.942 ms oleg416-client.virtnet: 1 6580 1.46 MB/sec warmup 36 sec latency 502.552 ms oleg416-client.virtnet: 1 7163 1.56 MB/sec warmup 37 sec latency 32.495 ms oleg416-client.virtnet: 1 7360 1.55 MB/sec warmup 38 sec latency 33.723 ms oleg416-client.virtnet: 1 7425 1.51 MB/sec warmup 39 sec latency 345.742 ms oleg416-client.virtnet: 1 7425 1.47 MB/sec warmup 40 sec latency 1346.456 ms oleg416-client.virtnet: 1 7425 1.44 MB/sec warmup 41 sec latency 2347.056 ms oleg416-client.virtnet: 1 7425 1.40 MB/sec warmup 42 sec latency 3347.219 ms oleg416-client.virtnet: 1 7425 1.37 MB/sec warmup 43 sec latency 4349.672 ms oleg416-client.virtnet: 1 7425 1.34 MB/sec warmup 44 sec latency 5350.607 ms oleg416-client.virtnet: 1 7425 1.31 MB/sec warmup 45 sec latency 6350.892 ms oleg416-client.virtnet: 1 7425 1.28 MB/sec warmup 46 sec latency 7352.112 ms oleg416-client.virtnet: 1 7425 1.25 MB/sec warmup 47 sec latency 8352.287 ms oleg416-client.virtnet: 1 7425 1.23 MB/sec warmup 48 sec latency 9352.655 ms oleg416-client.virtnet: 1 7425 1.20 MB/sec warmup 49 sec latency 10352.868 ms oleg416-client.virtnet: 1 7425 1.18 MB/sec warmup 50 sec latency 11353.055 ms oleg416-client.virtnet: 1 7425 1.15 MB/sec warmup 51 sec latency 12356.778 ms oleg416-client.virtnet: 1 7425 1.13 MB/sec warmup 52 sec latency 13358.107 ms oleg416-client.virtnet: 1 7425 1.11 MB/sec warmup 53 sec latency 14358.221 ms oleg416-client.virtnet: 1 7425 1.09 MB/sec warmup 54 sec latency 15360.046 ms oleg416-client.virtnet: 1 7425 1.07 MB/sec warmup 55 sec latency 16361.228 ms oleg416-client.virtnet: 1 7425 1.05 MB/sec warmup 56 sec latency 17361.543 ms oleg416-client.virtnet: 1 7425 1.03 MB/sec warmup 57 sec latency 18362.039 ms oleg416-client.virtnet: 1 7425 1.01 MB/sec warmup 58 sec latency 19362.628 ms oleg416-client.virtnet: 1 7425 1.00 MB/sec warmup 59 sec latency 20362.793 ms oleg416-client.virtnet: 1 7544 0.30 MB/sec execute 1 sec latency 21342.810 ms oleg416-client.virtnet: 1 7707 0.16 MB/sec execute 2 sec latency 30.100 ms oleg416-client.virtnet: 1 8004 0.35 MB/sec execute 3 sec latency 32.208 ms oleg416-client.virtnet: 1 8280 0.59 MB/sec execute 4 sec latency 31.538 ms oleg416-client.virtnet: 1 8676 1.09 MB/sec execute 5 sec latency 27.746 ms oleg416-client.virtnet: 1 9165 1.00 MB/sec execute 6 sec latency 29.372 ms oleg416-client.virtnet: 1 9510 1.05 MB/sec execute 7 sec latency 50.271 ms oleg416-client.virtnet: 1 9710 1.05 MB/sec execute 8 sec latency 39.876 ms oleg416-client.virtnet: 1 10268 1.50 MB/sec execute 9 sec latency 25.932 ms oleg416-client.virtnet: 1 10630 1.77 MB/sec execute 10 sec latency 40.553 ms oleg416-client.virtnet: 1 10891 1.74 MB/sec execute 11 sec latency 35.611 ms oleg416-client.virtnet: 1 11029 1.62 MB/sec execute 12 sec latency 39.739 ms oleg416-client.virtnet: 1 11237 1.49 MB/sec execute 13 sec latency 32.005 ms oleg416-client.virtnet: 1 11533 1.44 MB/sec execute 14 sec latency 34.730 ms oleg416-client.virtnet: 1 11823 1.43 MB/sec execute 15 sec latency 34.321 ms oleg416-client.virtnet: 1 12183 1.53 MB/sec execute 16 sec latency 23.877 ms oleg416-client.virtnet: 1 12759 1.48 MB/sec execute 17 sec latency 30.433 ms oleg416-client.virtnet: 1 13133 1.51 MB/sec execute 18 sec latency 29.579 ms oleg416-client.virtnet: 1 13663 1.68 MB/sec executFailover mds1 to oleg416-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 17:05:45 (1717016745) targets are mounted 17:05:45 (1717016745) facet_failover done oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1285568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36412 3570432 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13096 3593640 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49508 7164072 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg416-server Stopping /mnt/lustre-mds2 (opts:) on oleg416-server 17:06:00 (1717016760) shut down Failover mds2 to oleg416-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0001 17:06:17 (1717016777) targets are mounted 17:06:17 (1717016777) facet_failover done oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec e 19 sec latency 23.098 ms oleg416-client.virtnet: 1 14270 1.85 MB/sec execute 20 sec latency 25.369 ms oleg416-client.virtnet: 1 14507 1.82 MB/sec execute 21 sec latency 24.066 ms oleg416-client.virtnet: 1 14683 1.75 MB/sec execute 22 sec latency 31.977 ms oleg416-client.virtnet: 1 14937 1.69 MB/sec execute 23 sec latency 43.053 ms oleg416-client.virtnet: 1 15249 1.69 MB/sec execute 24 sec latency 24.650 ms oleg416-client.virtnet: 1 15608 1.74 MB/sec execute 25 sec latency 26.184 ms oleg416-client.virtnet: 1 16039 1.69 MB/sec execute 26 sec latency 41.677 ms oleg416-client.virtnet: 1 16593 1.69 MB/sec execute 27 sec latency 29.690 ms oleg416-client.virtnet: 1 16963 1.77 MB/sec execute 28 sec latency 26.651 ms oleg416-client.virtnet: 1 17551 1.83 MB/sec execute 29 sec latency 25.701 ms oleg416-client.virtnet: 1 17968 1.91 MB/sec execute 30 sec latency 27.557 ms oleg416-client.virtnet: 1 18153 1.86 MB/sec execute 31 sec latency 24.370 ms oleg416-client.virtnet: 1 18412 1.80 MB/sec execute 32 sec latency 22.870 ms oleg416-client.virtnet: 1 18656 1.77 MB/sec execute 33 sec latency 31.250 ms oleg416-client.virtnet: 1 18965 1.76 MB/sec execute 34 sec latency 34.175 ms oleg416-client.virtnet: 1 19396 1.80 MB/sec execute 35 sec latency 29.856 ms oleg416-client.virtnet: 1 20080 1.79 MB/sec execute 36 sec latency 32.887 ms oleg416-client.virtnet: 1 20275 1.77 MB/sec execute 37 sec latency 35.850 ms oleg416-client.virtnet: 1 20702 1.85 MB/sec execute 38 sec latency 99.407 ms oleg416-client.virtnet: 1 20804 1.80 MB/sec execute 39 sec latency 622.954 ms oleg416-client.virtnet: 1 21397 1.88 MB/sec execute 40 sec latency 28.092 ms oleg416-client.virtnet: 1 21574 1.87 MB/sec execute 41 sec latency 51.296 ms oleg416-client.virtnet: 1 21689 1.83 MB/sec execute 42 sec latency 164.148 ms oleg416-client.virtnet: 1 21689 1.79 MB/sec execute 43 sec latency 1164.675 ms oleg416-client.virtnet: 1 21689 1.75 MB/sec execute 44 sec latency 2166.572 ms oleg416-client.virtnet: 1 21689 1.71 MB/sec execute 45 sec latency 3166.764 ms oleg416-client.virtnet: 1 21689 1.67 MB/sec execute 46 sec latency 4167.139 ms oleg416-client.virtnet: 1 21689 1.63 MB/sec execute 47 sec latency 5168.242 ms oleg416-client.virtnet: 1 21689 1.60 MB/sec execute 48 sec latency 6168.993 ms oleg416-client.virtnet: 1 21689 1.57 MB/sec execute 49 sec latency 7169.239 ms oleg416-client.virtnet: 1 21689 1.54 MB/sec execute 50 sec latency 8170.633 ms oleg416-client.virtnet: 1 21689 1.51 MB/sec execute 51 sec latency 9170.951 ms oleg416-client.virtnet: 1 21689 1.48 MB/sec execute 52 sec latency 10171.304 ms oleg416-client.virtnet: 1 21689 1.45 MB/sec execute 53 sec latency 11171.949 ms oleg416-client.virtnet: 1 21689 1.42 MB/sec execute 54 sec latency 12174.952 ms oleg416-client.virtnet: 1 21689 1.40 MB/sec execute 55 sec latency 13175.607 ms oleg416-client.virtnet: 1 21689 1.37 MB/sec execute 56 sec latency 14176.154 ms oleg416-client.virtnet: 1 21689 1.35 MB/sec execute 57 sec latency 15176.383 ms oleg416-client.virtnet: 1 21689 1.32 MB/sec execute 58 sec latency 16177.383 ms oleg416-client.virtnet: 1 21689 1.30 MB/sec execute 59 sec latency 17178.677 ms oleg416-client.virtnet: 1 21689 1.28 MB/sec execute 60 sec latency 18178.855 ms oleg416-client.virtnet: 1 21689 1.26 MB/sec execute 61 sec latency 19178.997 ms oleg416-client.virtnet: 1 21689 1.24 MB/sec execute 62 sec latency 20179.585 ms oleg416-client.virtnet: 1 21694 1.22 MB/sec execute 63 sec latency 21163.433 ms oleg416UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36400 3569708 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13324 3592756 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49724 7162464 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 17:06:31 (1717016791) shut down Failover mds1 to oleg416-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 17:06:48 (1717016808) targets are mounted 17:06:48 (1717016808) facet_failover done oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1285568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36456 3570108 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13152 3592824 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49608 7162932 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg416-server Stopping /mnt/lustre-mds2 (opts:) on oleg416-server 17:07:01 (1717016821) shut down -client.virtnet: 1 21879 1.20 MB/sec execute 64 sec latency 21.522 ms oleg416-client.virtnet: 1 22130 1.19 MB/sec execute 65 sec latency 37.992 ms oleg416-client.virtnet: 1 22504 1.19 MB/sec execute 66 sec latency 29.413 ms oleg416-client.virtnet: 1 22929 1.22 MB/sec execute 67 sec latency 30.690 ms oleg416-client.virtnet: 1 23493 1.21 MB/sec execute 68 sec latency 40.281 ms oleg416-client.virtnet: 1 23754 1.22 MB/sec execute 69 sec latency 63.702 ms oleg416-client.virtnet: 1 24089 1.25 MB/sec execute 70 sec latency 40.043 ms oleg416-client.virtnet: 1 24569 1.28 MB/sec execute 71 sec latency 34.143 ms oleg416-client.virtnet: 1 25017 1.31 MB/sec execute 72 sec latency 33.533 ms oleg416-client.virtnet: 1 25176 1.31 MB/sec execute 73 sec latency 48.925 ms oleg416-client.virtnet: 1 25297 1.30 MB/sec execute 74 sec latency 40.821 ms oleg416-client.virtnet: 1 25541 1.28 MB/sec execute 75 sec latency 33.117 ms oleg416-client.virtnet: 1 25790 1.27 MB/sec execute 76 sec latency 41.601 ms oleg416-client.virtnet: 1 26074 1.27 MB/sec execute 77 sec latency 31.709 ms oleg416-client.virtnet: 1 26513 1.30 MB/sec execute 78 sec latency 34.459 ms oleg416-client.virtnet: 1 27016 1.29 MB/sec execute 79 sec latency 43.314 ms oleg416-client.virtnet: 1 27310 1.29 MB/sec execute 80 sec latency 42.811 ms oleg416-client.virtnet: 1 27775 1.33 MB/sec execute 81 sec latency 25.681 ms oleg416-client.virtnet: 1 28213 1.35 MB/sec execute 82 sec latency 39.837 ms oleg416-client.virtnet: 1 28635 1.38 MB/sec execute 83 sec latency 31.115 ms oleg416-client.virtnet: 1 28798 1.37 MB/sec execute 84 sec latency 44.243 ms oleg416-client.virtnet: 1 28950 1.35 MB/sec execute 85 sec latency 37.821 ms oleg416-client.virtnet: 1 29197 1.34 MB/sec execute 86 sec latency 50.794 ms oleg416-client.virtnet: 1 29397 1.33 MB/sec execute 87 sec latency 59.496 ms oleg416-client.virtnet: 1 29668 1.33 MB/sec execute 88 sec latency 33.436 ms oleg416-client.virtnet: 1 30109 1.35 MB/sec execute 89 sec latency 28.077 ms oleg416-client.virtnet: 1 30799 1.36 MB/sec execute 90 sec latency 22.357 ms oleg416-client.virtnet: 1 31194 1.38 MB/sec execute 91 sec latency 28.855 ms oleg416-client.virtnet: 1 31788 1.41 MB/sec execute 92 sec latency 21.737 ms oleg416-client.virtnet: 1 32245 1.44 MB/sec execute 93 sec latency 22.789 ms oleg416-client.virtnet: 1 32437 1.43 MB/sec execute 94 sec latency 20.895 ms oleg416-client.virtnet: 1 32867 1.42 MB/sec execute 95 sec latency 25.965 ms oleg416-client.virtnet: 1 33366 1.45 MB/sec execute 96 sec latency 20.364 ms oleg416-client.virtnet: 1 33815 1.44 MB/sec execute 97 sec latency 29.452 ms oleg416-client.virtnet: 1 34421 1.44 MB/sec execute 98 sec latency 20.802 ms oleg416-client.virtnet: 1 34945 1.48 MB/sec execute 99 sec latency 33.912 ms oleg416-client.virtnet: 1 35159 1.47 MB/sec execute 100 sec latency 644.180 ms oleg416-client.virtnet: 1 35641 1.50 MB/sec execute 101 sec latency 751.762 ms oleg416-client.virtnet: 1 35866 1.50 MB/sec execute 102 sec latency 30.523 ms oleg416-client.virtnet: 1 35993 1.49 MB/sec execute 103 sec latency 34.881 ms oleg416-client.virtnet: 1 35994 1.47 MB/sec execute 104 sec latency 987.508 ms oleg416-client.virtnet: 1 35994 1.46 MB/sec execute 105 sec latency 1987.662 ms oleg416-client.virtnet: 1 35994 1.45 MB/sec execute 106 sec latency 2987.820 ms oleg416-client.virtnet: 1 35994 1.43 MB/sec execute 107 sec latency 3989.530 ms oleg416-client.virtnet: 1 35994 1.42 MB/sec execute 108 sec latency 4989.722 Failover mds2 to oleg416-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0001 17:07:17 (1717016837) targets are mounted 17:07:17 (1717016837) facet_failover done oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36408 3570132 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13092 3592960 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7163092 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 17:07:31 (1717016851) shut down Failover mds1 to oleg416-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 17:07:48 (1717016868) targets are mounted 17:07:48 (1717016868) facet_failover done ms oleg416-client.virtnet: 1 35994 1.41 MB/sec execute 109 sec latency 5989.929 ms oleg416-client.virtnet: 1 35994 1.39 MB/sec execute 110 sec latency 6990.131 ms oleg416-client.virtnet: 1 35994 1.38 MB/sec execute 111 sec latency 7990.347 ms oleg416-client.virtnet: 1 35994 1.37 MB/sec execute 112 sec latency 8990.568 ms oleg416-client.virtnet: 1 35994 1.36 MB/sec execute 113 sec latency 9990.763 ms oleg416-client.virtnet: 1 35994 1.34 MB/sec execute 114 sec latency 10990.979 ms oleg416-client.virtnet: 1 35994 1.33 MB/sec execute 115 sec latency 11992.930 ms oleg416-client.virtnet: 1 35994 1.32 MB/sec execute 116 sec latency 12996.822 ms oleg416-client.virtnet: 1 35994 1.31 MB/sec execute 117 sec latency 13997.924 ms oleg416-client.virtnet: 1 35994 1.30 MB/sec execute 118 sec latency 14998.105 ms oleg416-client.virtnet: 1 35994 1.29 MB/sec execute 119 sec latency 15998.847 ms oleg416-client.virtnet: 1 35994 1.28 MB/sec execute 120 sec latency 16999.014 ms oleg416-client.virtnet: 1 35994 1.27 MB/sec execute 121 sec latency 17999.204 ms oleg416-client.virtnet: 1 35994 1.26 MB/sec execute 122 sec latency 19001.506 ms oleg416-client.virtnet: 1 35994 1.25 MB/sec execute 123 sec latency 20001.815 ms oleg416-client.virtnet: 1 36071 1.24 MB/sec execute 124 sec latency 20684.326 ms oleg416-client.virtnet: 1 36315 1.23 MB/sec execute 125 sec latency 35.105 ms oleg416-client.virtnet: 1 36584 1.23 MB/sec execute 126 sec latency 72.387 ms oleg416-client.virtnet: 1 36983 1.25 MB/sec execute 127 sec latency 28.927 ms oleg416-client.virtnet: 1 37584 1.24 MB/sec execute 128 sec latency 30.893 ms oleg416-client.virtnet: 1 37989 1.25 MB/sec execute 129 sec latency 21.714 ms oleg416-client.virtnet: 1 38534 1.28 MB/sec execute 130 sec latency 24.737 ms oleg416-client.virtnet: 1 39192 1.31 MB/sec execute 131 sec latency 19.760 ms oleg416-client.virtnet: 1 39434 1.31 MB/sec execute 132 sec latency 25.728 ms oleg416-client.virtnet: 1 39720 1.30 MB/sec execute 133 sec latency 23.209 ms oleg416-client.virtnet: 1 39985 1.29 MB/sec execute 134 sec latency 31.173 ms oleg416-client.virtnet: 1 40441 1.31 MB/sec execute 135 sec latency 25.009 ms oleg416-client.virtnet: 1 40867 1.31 MB/sec execute 136 sec latency 25.063 ms oleg416-client.virtnet: 1 41485 1.31 MB/sec execute 137 sec latency 22.237 ms oleg416-client.virtnet: 1 41988 1.34 MB/sec execute 138 sec latency 25.568 ms oleg416-client.virtnet: 1 42661 1.37 MB/sec execute 139 sec latency 21.614 ms oleg416-client.virtnet: 1 42962 1.37 MB/sec execute 140 sec latency 22.131 ms oleg416-client.virtnet: 1 43288 1.36 MB/sec execute 141 sec latency 23.972 ms oleg416-client.virtnet: 1 43776 1.36 MB/sec execute 142 sec latency 18.364 ms oleg416-client.virtnet: 1 44361 1.38 MB/sec execute 143 sec latency 24.080 ms oleg416-client.virtnet: 1 45015 1.38 MB/sec execute 144 sec latency 18.727 ms oleg416-client.virtnet: 1 45379 1.40 MB/sec execute 145 sec latency 29.588 ms oleg416-client.virtnet: 1 45856 1.41 MB/sec execute 146 sec latency 36.273 ms oleg416-client.virtnet: 1 46266 1.42 MB/sec execute 147 sec latency 30.110 ms oleg416-client.virtnet: 1 46445 1.42 MB/sec execute 148 sec latency 52.254 ms oleg416-client.virtnet: 1 46594 1.41 MB/sec execute 149 sec latency 33.031 ms oleg416-client.virtnet: 1 46869 1.41 MB/sec execute 150 sec latency 47.784 ms oleg416-client.virtnet: 1 47126 1.40 MB/sec execute 151 sec latency 41.187 ms oleg416-client.virtnet: 1 47354 1.40 MB/sec execute 152 sec latency 60.283 ms oleg416-client.virtnet: 1 47739 1.4oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1285564 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36504 3569808 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13088 3591588 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49592 7161396 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg416-server Stopping /mnt/lustre-mds2 (opts:) on oleg416-server 17:08:03 (1717016883) shut down Failover mds2 to oleg416-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0001 17:08:20 (1717016900) targets are mounted 17:08:20 (1717016900) facet_failover done oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36408 3569360 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13108 3592684 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49516 7162044 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 MB/sec execute 153 sec latency 39.687 ms oleg416-client.virtnet: 1 48071 1.40 MB/sec execute 154 sec latency 49.791 ms oleg416-client.virtnet: 1 48528 1.40 MB/sec execute 155 sec latency 46.305 ms oleg416-client.virtnet: 1 48750 1.40 MB/sec execute 156 sec latency 34.259 ms oleg416-client.virtnet: 1 49169 1.42 MB/sec execute 157 sec latency 34.665 ms oleg416-client.virtnet: 1 49767 1.44 MB/sec execute 158 sec latency 24.380 ms oleg416-client.virtnet: 1 49951 1.44 MB/sec execute 159 sec latency 58.311 ms oleg416-client.virtnet: 1 50075 1.44 MB/sec execute 160 sec latency 34.538 ms oleg416-client.virtnet: 1 50165 1.43 MB/sec execute 161 sec latency 46.489 ms oleg416-client.virtnet: 1 50379 1.42 MB/sec execute 162 sec latency 61.258 ms oleg416-client.virtnet: 1 50618 1.41 MB/sec execute 163 sec latency 34.768 ms oleg416-client.virtnet: 1 50844 1.41 MB/sec execute 164 sec latency 38.219 ms oleg416-client.virtnet: 1 50958 1.41 MB/sec execute 165 sec latency 385.700 ms oleg416-client.virtnet: 1 50958 1.40 MB/sec execute 166 sec latency 1386.363 ms oleg416-client.virtnet: 1 50958 1.39 MB/sec execute 167 sec latency 2387.040 ms oleg416-client.virtnet: 1 50958 1.38 MB/sec execute 168 sec latency 3387.241 ms oleg416-client.virtnet: 1 50958 1.37 MB/sec execute 169 sec latency 4388.371 ms oleg416-client.virtnet: 1 50958 1.36 MB/sec execute 170 sec latency 5388.792 ms oleg416-client.virtnet: 1 50958 1.36 MB/sec execute 171 sec latency 6390.051 ms oleg416-client.virtnet: 1 50958 1.35 MB/sec execute 172 sec latency 7410.537 ms oleg416-client.virtnet: 1 50958 1.34 MB/sec execute 173 sec latency 8412.786 ms oleg416-client.virtnet: 1 50958 1.33 MB/sec execute 174 sec latency 9414.570 ms oleg416-client.virtnet: 1 50958 1.32 MB/sec execute 175 sec latency 10414.773 ms oleg416-client.virtnet: 1 50958 1.32 MB/sec execute 176 sec latency 11415.990 ms oleg416-client.virtnet: 1 50958 1.31 MB/sec execute 177 sec latency 12416.108 ms oleg416-client.virtnet: 1 50958 1.30 MB/sec execute 178 sec latency 13416.290 ms oleg416-client.virtnet: 1 50958 1.29 MB/sec execute 179 sec latency 14416.412 ms oleg416-client.virtnet: 1 50958 1.29 MB/sec execute 180 sec latency 15417.088 ms oleg416-client.virtnet: 1 50958 1.28 MB/sec execute 181 sec latency 16419.197 ms oleg416-client.virtnet: 1 50958 1.27 MB/sec execute 182 sec latency 17419.334 ms oleg416-client.virtnet: 1 50958 1.27 MB/sec execute 183 sec latency 18419.488 ms oleg416-client.virtnet: 1 50958 1.26 MB/sec execute 184 sec latency 19419.898 ms oleg416-client.virtnet: 1 50958 1.25 MB/sec execute 185 sec latency 20420.529 ms oleg416-client.virtnet: 1 50958 1.25 MB/sec execute 186 sec latency 21420.901 ms oleg416-client.virtnet: 1 51143 1.26 MB/sec execute 187 sec latency 22058.093 ms oleg416-client.virtnet: 1 51501 1.25 MB/sec execute 188 sec latency 40.304 ms oleg416-client.virtnet: 1 52059 1.25 MB/sec execute 189 sec latency 23.613 ms oleg416-client.virtnet: 1 52292 1.25 MB/sec execute 190 sec latency 38.609 ms oleg416-client.virtnet: 1 52856 1.27 MB/sec execute 191 sec latency 33.935 ms oleg416-client.virtnet: 1 53214 1.29 MB/sec execute 192 sec latency 40.815 ms oleg416-client.virtnet: 1 53483 1.29 MB/sec execute 193 sec latency 35.624 ms oleg416-client.virtnet: 1 53645 1.28 MB/sec execute 194 sec latency 28.451 ms oleg416-client.virtnet: 1 53922 1.28 MB/sec execute 195 sec latency 35.827 ms oleg416-client.virtnet: 1 54176 1.27 MB/sec execute 196 sec latency 27.983 ms oleg416-client.virtnet: 1 54465 1.27 MB/sec execute 197 sec latency17:08:35 (1717016915) shut down Failover mds1 to oleg416-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 17:08:52 (1717016932) targets are mounted 17:08:52 (1717016932) facet_failover done oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1285568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36684 3569844 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13016 3593152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49700 7162996 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg416-server Stopping /mnt/lustre-mds2 (opts:) on oleg416-server 17:09:06 (1717016946) shut down Failover mds2 to oleg416-server mount facets: mds2 31.047 ms oleg416-client.virtnet: 1 54857 1.28 MB/sec execute 198 sec latency 45.642 ms oleg416-client.virtnet: 1 55425 1.28 MB/sec execute 199 sec latency 34.411 ms oleg416-client.virtnet: 1 55706 1.28 MB/sec execute 200 sec latency 30.229 ms oleg416-client.virtnet: 1 56214 1.30 MB/sec execute 201 sec latency 26.685 ms oleg416-client.virtnet: 1 56787 1.32 MB/sec execute 202 sec latency 24.785 ms oleg416-client.virtnet: 1 57064 1.32 MB/sec execute 203 sec latency 27.628 ms oleg416-client.virtnet: 1 57223 1.31 MB/sec execute 204 sec latency 31.522 ms oleg416-client.virtnet: 1 57505 1.31 MB/sec execute 205 sec latency 37.876 ms oleg416-client.virtnet: 1 57783 1.30 MB/sec execute 206 sec latency 31.561 ms oleg416-client.virtnet: 1 58233 1.32 MB/sec execute 207 sec latency 26.410 ms oleg416-client.virtnet: 1 58847 1.31 MB/sec execute 208 sec latency 25.293 ms oleg416-client.virtnet: 1 59209 1.32 MB/sec execute 209 sec latency 46.301 ms oleg416-client.virtnet: 1 59547 1.33 MB/sec execute 210 sec latency 35.971 ms oleg416-client.virtnet: 1 60115 1.34 MB/sec execute 211 sec latency 24.159 ms oleg416-client.virtnet: 1 60463 1.35 MB/sec execute 212 sec latency 33.537 ms oleg416-client.virtnet: 1 60653 1.35 MB/sec execute 213 sec latency 43.151 ms oleg416-client.virtnet: 1 60796 1.34 MB/sec execute 214 sec latency 29.049 ms oleg416-client.virtnet: 1 61061 1.34 MB/sec execute 215 sec latency 36.156 ms oleg416-client.virtnet: 1 61324 1.33 MB/sec execute 216 sec latency 50.260 ms oleg416-client.virtnet: 1 61579 1.33 MB/sec execute 217 sec latency 31.322 ms oleg416-client.virtnet: 1 61938 1.34 MB/sec execute 218 sec latency 57.850 ms oleg416-client.virtnet: 1 62285 1.34 MB/sec execute 219 sec latency 43.891 ms oleg416-client.virtnet: 1 62732 1.34 MB/sec execute 220 sec latency 25.476 ms oleg416-client.virtnet: 1 62941 1.33 MB/sec execute 221 sec latency 46.086 ms oleg416-client.virtnet: 1 63516 1.35 MB/sec execute 222 sec latency 33.839 ms oleg416-client.virtnet: 1 63884 1.37 MB/sec execute 223 sec latency 50.988 ms oleg416-client.virtnet: 1 64137 1.36 MB/sec execute 224 sec latency 36.726 ms oleg416-client.virtnet: 1 64282 1.36 MB/sec execute 225 sec latency 34.035 ms oleg416-client.virtnet: 1 64472 1.35 MB/sec execute 226 sec latency 35.371 ms oleg416-client.virtnet: 1 64770 1.35 MB/sec execute 227 sec latency 58.154 ms oleg416-client.virtnet: 1 65040 1.35 MB/sec execute 228 sec latency 31.573 ms oleg416-client.virtnet: 1 65054 1.35 MB/sec execute 229 sec latency 916.492 ms oleg416-client.virtnet: 1 65054 1.34 MB/sec execute 230 sec latency 1917.138 ms oleg416-client.virtnet: 1 65054 1.33 MB/sec execute 231 sec latency 2918.353 ms oleg416-client.virtnet: 1 65054 1.33 MB/sec execute 232 sec latency 3918.797 ms oleg416-client.virtnet: 1 65054 1.32 MB/sec execute 233 sec latency 4918.956 ms oleg416-client.virtnet: 1 65054 1.32 MB/sec execute 234 sec latency 5920.517 ms oleg416-client.virtnet: 1 65054 1.31 MB/sec execute 235 sec latency 6921.986 ms oleg416-client.virtnet: 1 65054 1.31 MB/sec execute 236 sec latency 7923.331 ms oleg416-client.virtnet: 1 65054 1.30 MB/sec execute 237 sec latency 8923.507 ms oleg416-client.virtnet: 1 65054 1.29 MB/sec execute 238 sec latency 9923.670 ms oleg416-client.virtnet: 1 65054 1.29 MB/sec execute 239 sec latency 10924.080 ms oleg416-client.virtnet: 1 65054 1.28 MB/sec execute 240 sec latency 11924.269 ms oleg416-client.virtnet: 1 65054 1.28 MB/sec execute 241 sec latency 12924.474 ms oleg416-client.virtnet: 1 65054 1.27 MB/seStarting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0001 17:09:24 (1717016964) targets are mounted 17:09:24 (1717016964) facet_failover done oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c execute 242 sec latency 13924.782 ms oleg416-client.virtnet: 1 65054 1.27 MB/sec execute 243 sec latency 14925.964 ms oleg416-client.virtnet: 1 65054 1.26 MB/sec execute 244 sec latency 15926.635 ms oleg416-client.virtnet: 1 65054 1.26 MB/sec execute 245 sec latency 16927.089 ms oleg416-client.virtnet: 1 65054 1.25 MB/sec execute 246 sec latency 17927.337 ms oleg416-client.virtnet: 1 65054 1.25 MB/sec execute 247 sec latency 18928.215 ms oleg416-client.virtnet: 1 65054 1.24 MB/sec execute 248 sec latency 19928.446 ms oleg416-client.virtnet: 1 65054 1.24 MB/sec execute 249 sec latency 20928.644 ms oleg416-client.virtnet: 1 65056 1.23 MB/sec execute 250 sec latency 21908.465 ms oleg416-client.virtnet: 1 65369 1.24 MB/sec execute 251 sec latency 28.779 ms oleg416-client.virtnet: 1 65713 1.24 MB/sec execute 252 sec latency 51.131 ms oleg416-client.virtnet: 1 66217 1.24 MB/sec execute 253 sec latency 28.494 ms oleg416-client.virtnet: 1 66374 1.24 MB/sec execute 254 sec latency 40.242 ms oleg416-client.virtnet: 1 66847 1.25 MB/sec execute 255 sec latency 34.525 ms oleg416-client.virtnet: 1 67381 1.26 MB/sec execute 256 sec latency 30.721 ms oleg416-client.virtnet: 1 67669 1.26 MB/sec execute 257 sec latency 26.952 ms oleg416-client.virtnet: 1 67832 1.26 MB/sec execute 258 sec latency 26.852 ms oleg416-client.virtnet: 1 68023 1.26 MB/sec execute 259 sec latency 36.179 ms oleg416-client.virtnet: 1 68305 1.25 MB/sec execute 260 sec latency 44.874 ms oleg416-client.virtnet: 1 68620 1.25 MB/sec execute 261 sec latency 27.097 ms oleg416-client.virtnet: 1 69069 1.26 MB/sec execute 262 sec latency 28.044 ms oleg416-client.virtnet: 1 69719 1.26 MB/sec execute 263 sec latency 40.733 ms oleg416-client.virtnet: 1 69917 1.26 MB/sec execute 264 sec latency 32.536 ms oleg416-client.virtnet: 1 70393 1.27 MB/sec execute 265 sec latency 34.299 ms oleg416-client.virtnet: 1 70908 1.29 MB/sec execute 266 sec latency 32.226 ms oleg416-client.virtnet: 1 71214 1.29 MB/sec execute 267 sec latency 32.927 ms oleg416-client.virtnet: 1 71368 1.28 MB/sec execute 268 sec latency 32.174 ms oleg416-client.virtnet: 1 71582 1.28 MB/sec execute 269 sec latency 27.909 ms oleg416-client.virtnet: 1 71875 1.28 MB/sec execute 270 sec latency 42.671 ms oleg416-client.virtnet: 1 72178 1.28 MB/sec execute 271 sec latency 35.945 ms oleg416-client.virtnet: 1 72624 1.29 MB/sec execute 272 sec latency 24.375 ms oleg416-client.virtnet: 1 73343 1.29 MB/sec execute 273 sec latency 26.033 ms oleg416-client.virtnet: 1 73567 1.29 MB/sec execute 274 sec latency 27.910 ms oleg416-client.virtnet: 1 74011 1.30 MB/sec execute 275 sec latency 30.725 ms oleg416-client.virtnet: 1 74614 1.31 MB/sec execute 276 sec latency 23.714 ms oleg416-client.virtnet: 1 74831 1.31 MB/sec execute 277 sec latency 34.165 ms oleg416-client.virtnet: 1 74997 1.31 MB/sec execute 278 sec latency 28.979 ms oleg416-client.virtnet: 1 75266 1.30 MB/sec execute 279 sec latency 39.840 ms oleg416-client.virtnet: 1 75524 1.30 MB/sec execute 280 sec latency 40.009 ms oleg416-client.virtnet: 1 75813 1.30 MB/sec execute 281 sec latency 23.459 ms oleg416-client.virtnet: 1 76352 1.31 MB/sec execute 282 sec latency 26.842 ms oleg416-client.virtnet: 1 76947 1.31 MB/sec execute 283 sec latency 29.195 ms oleg416-client.virtnet: 1 77339 1.32 MB/sec execute 284 sec latency 28.305 ms oleg416-client.virtnet: 1 77893 1.33 MB/sec execute 285 sec latency 33.715 ms oleg416-client.virtnet: 1 78307 1.34 MB/sec execute 286 sec latency 30.491 ms oleg416-client.virtnet: 1 78470 1.33 MB/sec execute 287 sec latency 52.425 ms oleg416-client.virtnet: 1 78678 1.33 MB/sec execute 288 sec latency 26.504 ms oleg416-client.virtnet: 1 78962 1.33 MB/sec execute 289 sec latency 34.410 ms oleg416-client.virtnet: 1 79221 1.33 MB/sec execute 290 sec latency 40.471 ms oleg416-client.virtnet: 1 79580 1.33 MB/sec execute 291 sec latency 31.082 ms oleg416-client.virtnet: 1 79981 1.33 MB/sec execute 292 sec latency 29.700 ms oleg416-client.virtnet: 1 80484 1.33 MB/sec execute 293 sec latency 41.441 ms oleg416-client.virtnet: 1 80852 1.34 MB/sec execute 294 sec latency 28.005 ms oleg416-client.virtnet: 1 81280 1.34 MB/sec execute 295 sec latency 37.932 ms oleg416-client.virtnet: 1 81769 1.35 MB/sec execute 296 sec latency 34.709 ms oleg416-client.virtnet: 1 81974 1.35 MB/sec execute 297 sec latency 36.051 ms oleg416-client.virtnet: 1 82135 1.35 MB/sec execute 298 sec latency 28.473 ms oleg416-client.virtnet: 1 82426 1.34 MB/sec execute 299 sec latency 51.287 ms oleg416-client.virtnet: 1 cleanup 300 sec oleg416-client.virtnet: 0 cleanup 301 sec oleg416-client.virtnet: oleg416-client.virtnet: Operation Count AvgLat MaxLat oleg416-client.virtnet: ---------------------------------------- oleg416-client.virtnet: NTCreateX 13059 9.902 21163.418 oleg416-client.virtnet: Close 9569 1.004 32.662 oleg416-client.virtnet: Rename 550 18.426 60.252 oleg416-client.virtnet: Unlink 2661 6.007 28.032 oleg416-client.virtnet: Qpathinfo 11798 5.194 21908.450 oleg416-client.virtnet: Qfileinfo 2048 11.024 22058.081 oleg416-client.virtnet: Qfsinfo 2171 0.462 23.809 oleg416-client.virtnet: Sfileinfo 1068 13.003 52.592 oleg416-client.virtnet: Find 4564 0.821 35.349 oleg416-client.virtnet: WriteX 6435 2.216 58.136 oleg416-client.virtnet: ReadX 20335 0.026 3.650 oleg416-client.virtnet: LockX 42 3.260 9.259 oleg416-client.virtnet: UnlockX 42 2.778 6.950 oleg416-client.virtnet: Flush 927 18.001 751.749 oleg416-client.virtnet: oleg416-client.virtnet: Throughput 1.34475 MB/sec 1 clients 1 procs max_latency=22058.093 ms oleg416-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg416-client.virtnet at Wed May 29 17:10:18 EDT 2024 with return code 0 oleg416-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg416-client.virtnet oleg416-client.virtnet: /mnt/lustre/d70b.replay-single/oleg416-client.virtnet /mnt/lustre/d70b.replay-single/oleg416-client.virtnet oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg416-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg416-client.virtnet: removed directory: 'clients/client0' oleg416-client.virtnet: removed directory: 'clients' oleg416-client.virtnet: removed 'client.txt' oleg416-client.virtnet: /mnt/lustre/d70b.replay-single/oleg416-client.virtnet oleg416-client.virtnet: dbench successfully finished