== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 05:32:16 (1731753136) Starting client oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre Started clients oleg118-client.virtnet: 192.168.201.118@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,csumfix,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg118-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13023 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2496 1285192 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1744 3593608 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26148 3556152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27892 7149760 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 05:32:23 (1731753143) shut down Failover mds1 to oleg118-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 05:32:37 (1731753157) targets are mounted 05:32:37 (1731753157) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1816 3594000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26412 3568812 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28228 7162812 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg118-server Stopping /mnt/lustre-mds2 (opts:) on oleg118-server 05:32:49 (1731753169) shut down oleg118-client.virtnet: looking for dbench program oleg118-client.virtnet: /usr/bin/dbench oleg118-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg118-client.virtnet oleg118-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg118-client.virtnet' oleg118-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg118-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg118-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg118-client.virtnet at Sat Nov 16 05:32:17 EST 2024 oleg118-client.virtnet: waiting for dbench pid 13047 oleg118-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg118-client.virtnet: oleg118-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg118-client.virtnet: failed to create barrier semaphore oleg118-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg118-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg118-client.virtnet: releasing clients oleg118-client.virtnet: 1 640 21.06 MB/sec warmup 1 sec latency 12.579 ms oleg118-client.virtnet: 1 1236 11.54 MB/sec warmup 2 sec latency 8.973 ms oleg118-client.virtnet: 1 1986 8.82 MB/sec warmup 3 sec latency 9.201 ms oleg118-client.virtnet: 1 2902 8.21 MB/sec warmup 4 sec latency 8.710 ms oleg118-client.virtnet: 1 3867 7.95 MB/sec warmup 5 sec latency 9.998 ms oleg118-client.virtnet: 1 4049 6.68 MB/sec warmup 6 sec latency 629.956 ms oleg118-client.virtnet: 1 4049 5.72 MB/sec warmup 7 sec latency 1630.115 ms oleg118-client.virtnet: 1 4049 5.01 MB/sec warmup 8 sec latency 2630.273 ms oleg118-client.virtnet: 1 4049 4.45 MB/sec warmup 9 sec latency 3630.434 ms oleg118-client.virtnet: 1 4049 4.01 MB/sec warmup 10 sec latency 4630.589 ms oleg118-client.virtnet: 1 4049 3.64 MB/sec warmup 11 sec latency 5630.762 ms oleg118-client.virtnet: 1 4049 3.34 MB/sec warmup 12 sec latency 6630.914 ms oleg118-client.virtnet: 1 4049 3.08 MB/sec warmup 13 sec latency 7631.074 ms oleg118-client.virtnet: 1 4049 2.86 MB/sec warmup 14 sec latency 8631.242 ms oleg118-client.virtnet: 1 4049 2.67 MB/sec warmup 15 sec latency 9631.431 ms oleg118-client.virtnet: 1 4049 2.50 MB/sec warmup 16 sec latency 10631.553 ms oleg118-client.virtnet: 1 4049 2.36 MB/sec warmup 17 sec latency 11631.658 ms oleg118-client.virtnet: 1 4049 2.23 MB/sec warmup 18 sec latency 12631.781 ms oleg118-client.virtnet: 1 4049 2.11 MB/sec warmup 19 sec latency 13631.868 ms oleg118-client.virtnet: 1 4049 2.00 MB/sec warmup 20 sec latency 14632.003 ms oleg118-client.virtnet: 1 4049 1.91 MB/sec warmup 21 sec latency 15632.174 ms oleg118-client.virtnet: 1 4049 1.82 MB/sec warmup 22 sec latency 16632.403 ms oleg118-client.virtnet: 1 4049 1.74 MB/sec warmup 23 sec latency 17632.570 ms oleg118-client.virtnet: 1 4351 1.69 MB/sec warmup 24 sec latency 18090.913 ms oleg118-client.virtnet: 1 5040 1.81 MB/sec warmup 25 sec latency 9.369 ms oleg118-client.virtnet: 1 5660 1.76 MB/sec warmup 26 sec latency 10.047 ms oleg118-client.virtnet: 1 6191 1.80 MB/sec warmup 27 sec latency 14.234 ms oleg118-client.virtnet: 1 7190 2.06 MB/sec warmup 28 sec latency 11.286 ms oleg118-client.virtnet: 1 7603 2.04 MB/sec warmup 29 sec latency 12.462 ms oleg118-client.virtnet: 1 8070 2.00 MB/sec warmup 30 sec latency 15.811 ms oleg118-client.virtnet: 1 8736 2.08 MB/sec warmup 31 sec latency 13.665 ms oleg118-client.virtnet: 1 9331 2.03 MB/sec warmup 32 sec latency 10.260 ms oleg118-client.virtnet: 1 10268 2.19 MB/sec warmup 33 sec latency 10.653 msFailover mds2 to oleg118-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 05:33:03 (1731753183) targets are mounted 05:33:03 (1731753183) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12308 3593628 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37420 3569004 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49728 7162632 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 05:33:14 (1731753194) shut down Failover mds1 to oleg118-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 05:33:29 (1731753209) targets are mounted 05:33:29 (1731753209) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg118-client.virtnet: 1 10870 2.29 MB/sec warmup 34 sec latency 12.415 ms oleg118-client.virtnet: 1 11300 2.24 MB/sec warmup 35 sec latency 12.412 ms oleg118-client.virtnet: 1 11717 2.23 MB/sec warmup 36 sec latency 14.766 ms oleg118-client.virtnet: 1 12219 2.26 MB/sec warmup 37 sec latency 12.601 ms oleg118-client.virtnet: 1 12636 2.21 MB/sec warmup 38 sec latency 12.445 ms oleg118-client.virtnet: 1 13080 2.19 MB/sec warmup 39 sec latency 12.386 ms oleg118-client.virtnet: 1 14036 2.32 MB/sec warmup 40 sec latency 11.631 ms oleg118-client.virtnet: 1 14605 2.38 MB/sec warmup 41 sec latency 9.782 ms oleg118-client.virtnet: 1 15115 2.34 MB/sec warmup 42 sec latency 10.574 ms oleg118-client.virtnet: 1 15699 2.38 MB/sec warmup 43 sec latency 10.935 ms oleg118-client.virtnet: 1 16203 2.34 MB/sec warmup 44 sec latency 12.424 ms oleg118-client.virtnet: 1 16942 2.41 MB/sec warmup 45 sec latency 10.412 ms oleg118-client.virtnet: 1 17777 2.50 MB/sec warmup 46 sec latency 262.995 ms oleg118-client.virtnet: 1 17777 2.44 MB/sec warmup 47 sec latency 1263.168 ms oleg118-client.virtnet: 1 17777 2.39 MB/sec warmup 48 sec latency 2263.339 ms oleg118-client.virtnet: 1 17777 2.34 MB/sec warmup 49 sec latency 3263.467 ms oleg118-client.virtnet: 1 18106 2.33 MB/sec warmup 50 sec latency 3687.552 ms oleg118-client.virtnet: 1 18687 2.30 MB/sec warmup 51 sec latency 11.106 ms oleg118-client.virtnet: 1 19382 2.34 MB/sec warmup 52 sec latency 9.044 ms oleg118-client.virtnet: 1 20059 2.32 MB/sec warmup 53 sec latency 9.445 ms oleg118-client.virtnet: 1 20977 2.41 MB/sec warmup 54 sec latency 114.994 ms oleg118-client.virtnet: 1 21609 2.46 MB/sec warmup 55 sec latency 335.115 ms oleg118-client.virtnet: 1 22152 2.43 MB/sec warmup 56 sec latency 15.761 ms oleg118-client.virtnet: 1 22324 2.40 MB/sec warmup 57 sec latency 678.985 ms oleg118-client.virtnet: 1 22324 2.36 MB/sec warmup 58 sec latency 1679.162 ms oleg118-client.virtnet: 1 22324 2.32 MB/sec warmup 59 sec latency 2679.338 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 1 sec latency 4679.580 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 2 sec latency 5679.766 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 3 sec latency 6679.929 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 4 sec latency 7680.108 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 5 sec latency 8680.290 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 6 sec latency 9680.456 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 7 sec latency 10680.579 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 8 sec latency 11680.720 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 9 sec latency 12680.963 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 10 sec latency 13681.295 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 11 sec latency 14681.480 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 12 sec latency 15681.740 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 13 sec latency 16681.955 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 14 sec latency 17682.134 ms oleg118-client.virtnet: 1 22324 0.00 MB/sec execute 15 sec latency 18682.293 ms oleg118-client.virtnet: 1 22464 0.07 MB/sec execute 16 sec latency 19391.064 ms oleg118-client.virtnet: 1 22900 0.25 MB/sec execute 17 sec latency 13.768 ms oleg118-client.virtnet: 1 23294 0.25 MB/sec execute 18 sec latency 15.366 ms oleg118-client.virtnet: 1 23728 0.32 MB/sec executUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1285276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12124 3593772 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37540 3566816 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49664 7160588 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg118-server Stopping /mnt/lustre-mds2 (opts:) on oleg118-server 05:33:42 (1731753222) shut down Failover mds2 to oleg118-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 05:33:56 (1731753236) targets are mounted 05:33:56 (1731753236) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12052 3593856 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37656 3566268 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49708 7160124 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 05:34:10 (1731753250) shut down Failover mds1 to oleg118-server mount facets: mds1 e 19 sec latency 16.064 ms oleg118-client.virtnet: 1 24347 0.58 MB/sec execute 20 sec latency 15.884 ms oleg118-client.virtnet: 1 25104 0.85 MB/sec execute 21 sec latency 13.450 ms oleg118-client.virtnet: 1 25401 0.83 MB/sec execute 22 sec latency 14.662 ms oleg118-client.virtnet: 1 25799 0.83 MB/sec execute 23 sec latency 15.685 ms oleg118-client.virtnet: 1 26335 0.97 MB/sec execute 24 sec latency 13.846 ms oleg118-client.virtnet: 1 26715 0.94 MB/sec execute 25 sec latency 16.131 ms oleg118-client.virtnet: 1 27150 0.94 MB/sec execute 26 sec latency 10.638 ms oleg118-client.virtnet: 1 27769 1.11 MB/sec execute 27 sec latency 14.176 ms oleg118-client.virtnet: 1 28454 1.27 MB/sec execute 28 sec latency 13.922 ms oleg118-client.virtnet: 1 28821 1.29 MB/sec execute 29 sec latency 12.168 ms oleg118-client.virtnet: 1 29208 1.26 MB/sec execute 30 sec latency 13.809 ms oleg118-client.virtnet: 1 29584 1.27 MB/sec execute 31 sec latency 14.993 ms oleg118-client.virtnet: 1 30009 1.33 MB/sec execute 32 sec latency 14.596 ms oleg118-client.virtnet: 1 30402 1.30 MB/sec execute 33 sec latency 15.548 ms oleg118-client.virtnet: 1 31072 1.36 MB/sec execute 34 sec latency 10.798 ms oleg118-client.virtnet: 1 32025 1.57 MB/sec execute 35 sec latency 13.124 ms oleg118-client.virtnet: 1 32381 1.57 MB/sec execute 36 sec latency 11.869 ms oleg118-client.virtnet: 1 32770 1.54 MB/sec execute 37 sec latency 14.061 ms oleg118-client.virtnet: 1 33163 1.54 MB/sec execute 38 sec latency 13.822 ms oleg118-client.virtnet: 1 33650 1.58 MB/sec execute 39 sec latency 11.542 ms oleg118-client.virtnet: 1 33681 1.54 MB/sec execute 40 sec latency 907.447 ms oleg118-client.virtnet: 1 33681 1.50 MB/sec execute 41 sec latency 1907.759 ms oleg118-client.virtnet: 1 33681 1.47 MB/sec execute 42 sec latency 2908.079 ms oleg118-client.virtnet: 1 34035 1.44 MB/sec execute 43 sec latency 3146.935 ms oleg118-client.virtnet: 1 34560 1.46 MB/sec execute 44 sec latency 12.576 ms oleg118-client.virtnet: 1 35238 1.57 MB/sec execute 45 sec latency 18.579 ms oleg118-client.virtnet: 1 35673 1.61 MB/sec execute 46 sec latency 13.741 ms oleg118-client.virtnet: 1 35930 1.61 MB/sec execute 47 sec latency 16.333 ms oleg118-client.virtnet: 1 36230 1.58 MB/sec execute 48 sec latency 18.024 ms oleg118-client.virtnet: 1 36388 1.56 MB/sec execute 49 sec latency 478.797 ms oleg118-client.virtnet: 1 36672 1.55 MB/sec execute 50 sec latency 19.733 ms oleg118-client.virtnet: 1 37031 1.58 MB/sec execute 51 sec latency 17.707 ms oleg118-client.virtnet: 1 37174 1.55 MB/sec execute 52 sec latency 512.460 ms oleg118-client.virtnet: 1 37174 1.52 MB/sec execute 53 sec latency 1512.608 ms oleg118-client.virtnet: 1 37174 1.50 MB/sec execute 54 sec latency 2512.848 ms oleg118-client.virtnet: 1 37174 1.47 MB/sec execute 55 sec latency 3513.026 ms oleg118-client.virtnet: 1 37174 1.44 MB/sec execute 56 sec latency 4513.255 ms oleg118-client.virtnet: 1 37174 1.42 MB/sec execute 57 sec latency 5513.517 ms oleg118-client.virtnet: 1 37174 1.39 MB/sec execute 58 sec latency 6513.767 ms oleg118-client.virtnet: 1 37174 1.37 MB/sec execute 59 sec latency 7514.012 ms oleg118-client.virtnet: 1 37174 1.35 MB/sec execute 60 sec latency 8514.294 ms oleg118-client.virtnet: 1 37174 1.32 MB/sec execute 61 sec latency 9514.605 ms oleg118-client.virtnet: 1 37174 1.30 MB/sec execute 62 sec latency 10514.873 ms oleg118-client.virtnet: 1 37174 1.28 MB/sec execute 63 sec latency 11515.054 ms oleg118-client.virtnet: 1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 05:34:25 (1731753265) targets are mounted 05:34:25 (1731753265) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11920 3594036 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37592 3567956 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49512 7161992 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg118-server Stopping /mnt/lustre-mds2 (opts:) on oleg118-server 05:34:38 (1731753278) shut down Failover mds2 to oleg118-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 05:34:53 (1731753293) targets are mounted 05:34:53 (1731753293) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11920 3594988 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37580 3569184 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7164172 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 37174 1.26 MB/sec execute 64 sec latency 12515.263 ms oleg118-client.virtnet: 1 37174 1.24 MB/sec execute 65 sec latency 13515.565 ms oleg118-client.virtnet: 1 37174 1.22 MB/sec execute 66 sec latency 14515.778 ms oleg118-client.virtnet: 1 37174 1.21 MB/sec execute 67 sec latency 15515.956 ms oleg118-client.virtnet: 1 37174 1.19 MB/sec execute 68 sec latency 16516.306 ms oleg118-client.virtnet: 1 37174 1.17 MB/sec execute 69 sec latency 17516.512 ms oleg118-client.virtnet: 1 37174 1.15 MB/sec execute 70 sec latency 18516.714 ms oleg118-client.virtnet: 1 37174 1.14 MB/sec execute 71 sec latency 19516.871 ms oleg118-client.virtnet: 1 37369 1.12 MB/sec execute 72 sec latency 19656.746 ms oleg118-client.virtnet: 1 37736 1.11 MB/sec execute 73 sec latency 13.419 ms oleg118-client.virtnet: 1 38093 1.13 MB/sec execute 74 sec latency 17.549 ms oleg118-client.virtnet: 1 38870 1.20 MB/sec execute 75 sec latency 17.408 ms oleg118-client.virtnet: 1 39439 1.25 MB/sec execute 76 sec latency 10.993 ms oleg118-client.virtnet: 1 39845 1.23 MB/sec execute 77 sec latency 11.803 ms oleg118-client.virtnet: 1 40255 1.24 MB/sec execute 78 sec latency 14.500 ms oleg118-client.virtnet: 1 40626 1.26 MB/sec execute 79 sec latency 17.487 ms oleg118-client.virtnet: 1 40942 1.25 MB/sec execute 80 sec latency 19.925 ms oleg118-client.virtnet: 1 41396 1.25 MB/sec execute 81 sec latency 12.541 ms oleg118-client.virtnet: 1 41773 1.29 MB/sec execute 82 sec latency 18.207 ms oleg118-client.virtnet: 1 42400 1.31 MB/sec execute 83 sec latency 18.160 ms oleg118-client.virtnet: 1 42772 1.34 MB/sec execute 84 sec latency 19.899 ms oleg118-client.virtnet: 1 43026 1.34 MB/sec execute 85 sec latency 16.885 ms oleg118-client.virtnet: 1 43385 1.33 MB/sec execute 86 sec latency 17.973 ms oleg118-client.virtnet: 1 43706 1.33 MB/sec execute 87 sec latency 28.587 ms oleg118-client.virtnet: 1 44096 1.35 MB/sec execute 88 sec latency 24.845 ms oleg118-client.virtnet: 1 44388 1.34 MB/sec execute 89 sec latency 19.397 ms oleg118-client.virtnet: 1 44970 1.34 MB/sec execute 90 sec latency 11.366 ms oleg118-client.virtnet: 1 45588 1.39 MB/sec execute 91 sec latency 17.798 ms oleg118-client.virtnet: 1 46223 1.43 MB/sec execute 92 sec latency 14.137 ms oleg118-client.virtnet: 1 46491 1.43 MB/sec execute 93 sec latency 16.453 ms oleg118-client.virtnet: 1 46948 1.42 MB/sec execute 94 sec latency 12.393 ms oleg118-client.virtnet: 1 47197 1.42 MB/sec execute 95 sec latency 274.789 ms oleg118-client.virtnet: 1 47197 1.40 MB/sec execute 96 sec latency 1275.029 ms oleg118-client.virtnet: 1 47197 1.39 MB/sec execute 97 sec latency 2275.230 ms oleg118-client.virtnet: 1 47197 1.38 MB/sec execute 98 sec latency 3275.552 ms oleg118-client.virtnet: 1 47219 1.36 MB/sec execute 99 sec latency 4212.088 ms oleg118-client.virtnet: 1 47621 1.38 MB/sec execute 100 sec latency 17.009 ms oleg118-client.virtnet: 1 47923 1.37 MB/sec execute 101 sec latency 22.946 ms oleg118-client.virtnet: 1 48248 1.36 MB/sec execute 102 sec latency 14.034 ms oleg118-client.virtnet: 1 48590 1.36 MB/sec execute 103 sec latency 17.174 ms oleg118-client.virtnet: 1 49156 1.40 MB/sec execute 104 sec latency 17.205 ms oleg118-client.virtnet: 1 49766 1.43 MB/sec execute 105 sec latency 17.394 ms oleg118-client.virtnet: 1 50036 1.43 MB/sec execute 106 sec latency 16.433 ms oleg118-client.virtnet: 1 50313 1.42 MB/sec execute 107 sec latency 16.206 ms oleg118-client.virtnet: 1 50640 1.41 MB/sec execute 108 sec latency 19.8805:35:06 (1731753306) shut down Failover mds1 to oleg118-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:22 (1731753322) targets are mounted 05:35:22 (1731753322) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11920 3594300 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37576 3568908 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7163208 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg118-server Stopping /mnt/lustre-mds2 (opts:) on oleg118-server 05:35:35 (1731753335) shut down Failover mds2 to oleg118-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 05:35:50 (1731753350) targets are mounted 05:35:50 (1731753350) facet_failover done 2 ms oleg118-client.virtnet: 1 50682 1.40 MB/sec execute 109 sec latency 835.826 ms oleg118-client.virtnet: 1 50682 1.39 MB/sec execute 110 sec latency 1836.033 ms oleg118-client.virtnet: 1 50682 1.38 MB/sec execute 111 sec latency 2836.287 ms oleg118-client.virtnet: 1 50682 1.37 MB/sec execute 112 sec latency 3836.522 ms oleg118-client.virtnet: 1 50682 1.35 MB/sec execute 113 sec latency 4836.835 ms oleg118-client.virtnet: 1 50682 1.34 MB/sec execute 114 sec latency 5837.117 ms oleg118-client.virtnet: 1 50682 1.33 MB/sec execute 115 sec latency 6837.302 ms oleg118-client.virtnet: 1 50682 1.32 MB/sec execute 116 sec latency 7837.547 ms oleg118-client.virtnet: 1 50682 1.31 MB/sec execute 117 sec latency 8837.873 ms oleg118-client.virtnet: 1 50682 1.30 MB/sec execute 118 sec latency 9838.137 ms oleg118-client.virtnet: 1 50682 1.29 MB/sec execute 119 sec latency 10838.285 ms oleg118-client.virtnet: 1 50682 1.27 MB/sec execute 120 sec latency 11838.509 ms oleg118-client.virtnet: 1 50682 1.26 MB/sec execute 121 sec latency 12838.762 ms oleg118-client.virtnet: 1 50682 1.25 MB/sec execute 122 sec latency 13839.009 ms oleg118-client.virtnet: 1 50682 1.24 MB/sec execute 123 sec latency 14839.201 ms oleg118-client.virtnet: 1 50682 1.23 MB/sec execute 124 sec latency 15839.376 ms oleg118-client.virtnet: 1 50682 1.22 MB/sec execute 125 sec latency 16839.578 ms oleg118-client.virtnet: 1 50682 1.21 MB/sec execute 126 sec latency 17839.862 ms oleg118-client.virtnet: 1 50682 1.20 MB/sec execute 127 sec latency 18840.084 ms oleg118-client.virtnet: 1 50682 1.19 MB/sec execute 128 sec latency 19840.260 ms oleg118-client.virtnet: 1 51125 1.22 MB/sec execute 129 sec latency 19900.022 ms oleg118-client.virtnet: 1 51427 1.21 MB/sec execute 130 sec latency 19.788 ms oleg118-client.virtnet: 1 51763 1.20 MB/sec execute 131 sec latency 13.108 ms oleg118-client.virtnet: 1 52256 1.21 MB/sec execute 132 sec latency 11.828 ms oleg118-client.virtnet: 1 53040 1.25 MB/sec execute 133 sec latency 16.282 ms oleg118-client.virtnet: 1 53454 1.27 MB/sec execute 134 sec latency 14.488 ms oleg118-client.virtnet: 1 53751 1.27 MB/sec execute 135 sec latency 15.939 ms oleg118-client.virtnet: 1 54072 1.26 MB/sec execute 136 sec latency 20.525 ms oleg118-client.virtnet: 1 54393 1.26 MB/sec execute 137 sec latency 17.179 ms oleg118-client.virtnet: 1 54752 1.28 MB/sec execute 138 sec latency 16.473 ms oleg118-client.virtnet: 1 55056 1.27 MB/sec execute 139 sec latency 17.534 ms oleg118-client.virtnet: 1 55414 1.26 MB/sec execute 140 sec latency 12.472 ms oleg118-client.virtnet: 1 55954 1.29 MB/sec execute 141 sec latency 18.270 ms oleg118-client.virtnet: 1 56765 1.33 MB/sec execute 142 sec latency 15.299 ms oleg118-client.virtnet: 1 57084 1.33 MB/sec execute 143 sec latency 16.046 ms oleg118-client.virtnet: 1 57400 1.32 MB/sec execute 144 sec latency 15.521 ms oleg118-client.virtnet: 1 57742 1.32 MB/sec execute 145 sec latency 17.697 ms oleg118-client.virtnet: 1 58215 1.34 MB/sec execute 146 sec latency 15.383 ms oleg118-client.virtnet: 1 58499 1.33 MB/sec execute 147 sec latency 19.343 ms oleg118-client.virtnet: 1 58812 1.32 MB/sec execute 148 sec latency 12.145 ms oleg118-client.virtnet: 1 59234 1.32 MB/sec execute 149 sec latency 12.422 ms oleg118-client.virtnet: 1 59945 1.36 MB/sec execute 150 sec latency 17.866 ms oleg118-client.virtnet: 1 60444 1.38 MB/sec execute 151 sec latency 16.857 ms oleg118-client.virtnet: 1 60823 1.38 MB/sec execute 152 sec latency 18.356 ms oleg118-client.virtnet: 1 61oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12000 3594008 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37588 3568136 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49588 7162144 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 05:36:03 (1731753363) shut down Failover mds1 to oleg118-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 05:36:17 (1731753377) targets are mounted 05:36:17 (1731753377) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1285272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12036 3594132 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37580 3566680 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160812 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg118-server Stopping /mnt/lustre-mds2 (opts:) on oleg118-server 05:36:30 (1731753390) shut down 457 1.38 MB/sec execute 153 sec latency 10.285 ms oleg118-client.virtnet: 1 61463 1.37 MB/sec execute 154 sec latency 990.704 ms oleg118-client.virtnet: 1 61463 1.36 MB/sec execute 155 sec latency 1990.922 ms oleg118-client.virtnet: 1 61463 1.35 MB/sec execute 156 sec latency 2991.175 ms oleg118-client.virtnet: 1 61764 1.37 MB/sec execute 157 sec latency 3403.860 ms oleg118-client.virtnet: 1 62058 1.36 MB/sec execute 158 sec latency 14.847 ms oleg118-client.virtnet: 1 62421 1.35 MB/sec execute 159 sec latency 12.186 ms oleg118-client.virtnet: 1 62778 1.35 MB/sec execute 160 sec latency 18.217 ms oleg118-client.virtnet: 1 63684 1.39 MB/sec execute 161 sec latency 17.753 ms oleg118-client.virtnet: 1 64246 1.41 MB/sec execute 162 sec latency 12.969 ms oleg118-client.virtnet: 1 64560 1.40 MB/sec execute 163 sec latency 28.073 ms oleg118-client.virtnet: 1 65050 1.41 MB/sec execute 164 sec latency 15.903 ms oleg118-client.virtnet: 1 65454 1.42 MB/sec execute 165 sec latency 347.711 ms oleg118-client.virtnet: 1 65454 1.41 MB/sec execute 166 sec latency 1347.906 ms oleg118-client.virtnet: 1 65454 1.40 MB/sec execute 167 sec latency 2348.134 ms oleg118-client.virtnet: 1 65454 1.39 MB/sec execute 168 sec latency 3348.394 ms oleg118-client.virtnet: 1 65454 1.38 MB/sec execute 169 sec latency 4348.623 ms oleg118-client.virtnet: 1 65454 1.37 MB/sec execute 170 sec latency 5348.972 ms oleg118-client.virtnet: 1 65454 1.37 MB/sec execute 171 sec latency 6349.262 ms oleg118-client.virtnet: 1 65454 1.36 MB/sec execute 172 sec latency 7349.493 ms oleg118-client.virtnet: 1 65454 1.35 MB/sec execute 173 sec latency 8349.736 ms oleg118-client.virtnet: 1 65454 1.34 MB/sec execute 174 sec latency 9350.012 ms oleg118-client.virtnet: 1 65454 1.34 MB/sec execute 175 sec latency 10350.220 ms oleg118-client.virtnet: 1 65454 1.33 MB/sec execute 176 sec latency 11350.402 ms oleg118-client.virtnet: 1 65454 1.32 MB/sec execute 177 sec latency 12350.586 ms oleg118-client.virtnet: 1 65454 1.31 MB/sec execute 178 sec latency 13350.923 ms oleg118-client.virtnet: 1 65454 1.31 MB/sec execute 179 sec latency 14351.173 ms oleg118-client.virtnet: 1 65454 1.30 MB/sec execute 180 sec latency 15351.389 ms oleg118-client.virtnet: 1 65454 1.29 MB/sec execute 181 sec latency 16351.621 ms oleg118-client.virtnet: 1 65454 1.28 MB/sec execute 182 sec latency 17352.053 ms oleg118-client.virtnet: 1 65454 1.28 MB/sec execute 183 sec latency 18352.283 ms oleg118-client.virtnet: 1 65492 1.27 MB/sec execute 184 sec latency 19213.124 ms oleg118-client.virtnet: 1 66027 1.27 MB/sec execute 185 sec latency 15.403 ms oleg118-client.virtnet: 1 66590 1.29 MB/sec execute 186 sec latency 15.702 ms oleg118-client.virtnet: 1 67242 1.30 MB/sec execute 187 sec latency 18.580 ms oleg118-client.virtnet: 1 67697 1.32 MB/sec execute 188 sec latency 15.767 ms oleg118-client.virtnet: 1 67936 1.31 MB/sec execute 189 sec latency 15.888 ms oleg118-client.virtnet: 1 68389 1.31 MB/sec execute 190 sec latency 12.149 ms oleg118-client.virtnet: 1 68932 1.32 MB/sec execute 191 sec latency 12.339 ms oleg118-client.virtnet: 1 69250 1.32 MB/sec execute 192 sec latency 18.493 ms oleg118-client.virtnet: 1 69658 1.31 MB/sec execute 193 sec latency 11.946 ms oleg118-client.virtnet: 1 70015 1.32 MB/sec execute 194 sec latency 18.099 ms oleg118-client.virtnet: 1 70689 1.34 MB/sec execute 195 sec latency 18.885 ms oleg118-client.virtnet: 1 71111 1.35 MB/sec execute 196 sec latency 13.059 ms oleg118-client.virtnet: 1 71384 1.35 MB/sec execute 197 secFailover mds2 to oleg118-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 05:36:46 (1731753406) targets are mounted 05:36:46 (1731753406) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11928 3593252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37584 3568284 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49512 7161536 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 05:36:58 (1731753418) shut down Failover mds1 to oleg118-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 05:37:13 (1731753433) targets are mounted 05:37:13 (1731753433) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec latency 16.653 ms oleg118-client.virtnet: 1 71682 1.35 MB/sec execute 198 sec latency 20.466 ms oleg118-client.virtnet: 1 71941 1.34 MB/sec execute 199 sec latency 19.042 ms oleg118-client.virtnet: 1 72373 1.36 MB/sec execute 200 sec latency 17.906 ms oleg118-client.virtnet: 1 72655 1.35 MB/sec execute 201 sec latency 16.026 ms oleg118-client.virtnet: 1 72966 1.35 MB/sec execute 202 sec latency 18.991 ms oleg118-client.virtnet: 1 73331 1.35 MB/sec execute 203 sec latency 12.013 ms oleg118-client.virtnet: 1 73762 1.36 MB/sec execute 204 sec latency 17.797 ms oleg118-client.virtnet: 1 74507 1.39 MB/sec execute 205 sec latency 17.527 ms oleg118-client.virtnet: 1 74856 1.39 MB/sec execute 206 sec latency 13.798 ms oleg118-client.virtnet: 1 75192 1.38 MB/sec execute 207 sec latency 15.711 ms oleg118-client.virtnet: 1 75432 1.38 MB/sec execute 208 sec latency 215.800 ms oleg118-client.virtnet: 1 75432 1.37 MB/sec execute 209 sec latency 1216.011 ms oleg118-client.virtnet: 1 75432 1.36 MB/sec execute 210 sec latency 2216.235 ms oleg118-client.virtnet: 1 75432 1.36 MB/sec execute 211 sec latency 3216.454 ms oleg118-client.virtnet: 1 75581 1.36 MB/sec execute 212 sec latency 3949.166 ms oleg118-client.virtnet: 1 76007 1.37 MB/sec execute 213 sec latency 16.793 ms oleg118-client.virtnet: 1 76370 1.36 MB/sec execute 214 sec latency 19.930 ms oleg118-client.virtnet: 1 76714 1.36 MB/sec execute 215 sec latency 12.609 ms oleg118-client.virtnet: 1 77140 1.36 MB/sec execute 216 sec latency 14.370 ms oleg118-client.virtnet: 1 77893 1.38 MB/sec execute 217 sec latency 16.811 ms oleg118-client.virtnet: 1 78358 1.40 MB/sec execute 218 sec latency 15.641 ms oleg118-client.virtnet: 1 78598 1.39 MB/sec execute 219 sec latency 15.493 ms oleg118-client.virtnet: 1 78954 1.39 MB/sec execute 220 sec latency 18.425 ms oleg118-client.virtnet: 1 79140 1.39 MB/sec execute 221 sec latency 571.243 ms oleg118-client.virtnet: 1 79140 1.38 MB/sec execute 222 sec latency 1571.518 ms oleg118-client.virtnet: 1 79140 1.38 MB/sec execute 223 sec latency 2571.763 ms oleg118-client.virtnet: 1 79140 1.37 MB/sec execute 224 sec latency 3571.982 ms oleg118-client.virtnet: 1 79140 1.36 MB/sec execute 225 sec latency 4572.172 ms oleg118-client.virtnet: 1 79140 1.36 MB/sec execute 226 sec latency 5572.331 ms oleg118-client.virtnet: 1 79140 1.35 MB/sec execute 227 sec latency 6572.567 ms oleg118-client.virtnet: 1 79140 1.35 MB/sec execute 228 sec latency 7572.770 ms oleg118-client.virtnet: 1 79140 1.34 MB/sec execute 229 sec latency 8572.994 ms oleg118-client.virtnet: 1 79140 1.33 MB/sec execute 230 sec latency 9573.253 ms oleg118-client.virtnet: 1 79140 1.33 MB/sec execute 231 sec latency 10573.514 ms oleg118-client.virtnet: 1 79140 1.32 MB/sec execute 232 sec latency 11573.802 ms oleg118-client.virtnet: 1 79140 1.32 MB/sec execute 233 sec latency 12574.027 ms oleg118-client.virtnet: 1 79140 1.31 MB/sec execute 234 sec latency 13574.130 ms oleg118-client.virtnet: 1 79140 1.31 MB/sec execute 235 sec latency 14574.278 ms oleg118-client.virtnet: 1 79140 1.30 MB/sec execute 236 sec latency 15574.442 ms oleg118-client.virtnet: 1 79140 1.30 MB/sec execute 237 sec latency 16574.649 ms oleg118-client.virtnet: 1 79140 1.29 MB/sec execute 238 sec latency 17574.825 ms oleg118-client.virtnet: 1 79140 1.28 MB/sec execute 239 sec latency 18575.047 ms oleg118-client.virtnet: 1 79511 1.29 MB/sec execute 240 sec latency 18918.900 ms oleg118-client.virtnet: 1 79964 1.29 MB/sec execute 241 sec latency 10.243 ms oleg118-client.virtnet: 1 80441 1.29 MB/sec execute 242 sec latency 11.760 ms oleg118-client.virtnet: 1 80865 1.30 MB/sec execute 243 sec latency 19.705 ms oleg118-client.virtnet: 1 81614 1.32 MB/sec execute 244 sec latency 14.771 ms oleg118-client.virtnet: 1 81932 1.32 MB/sec execute 245 sec latency 14.944 ms oleg118-client.virtnet: 1 82210 1.32 MB/sec execute 246 sec latency 14.127 ms oleg118-client.virtnet: 1 82654 1.32 MB/sec execute 247 sec latency 13.992 ms oleg118-client.virtnet: 1 83151 1.33 MB/sec execute 248 sec latency 17.866 ms oleg118-client.virtnet: 1 83484 1.32 MB/sec execute 249 sec latency 17.486 ms oleg118-client.virtnet: 1 83896 1.32 MB/sec execute 250 sec latency 11.188 ms oleg118-client.virtnet: 1 84392 1.34 MB/sec execute 251 sec latency 17.490 ms oleg118-client.virtnet: 1 85146 1.36 MB/sec execute 252 sec latency 15.486 ms oleg118-client.virtnet: 1 85478 1.36 MB/sec execute 253 sec latency 15.850 ms oleg118-client.virtnet: 1 85852 1.35 MB/sec execute 254 sec latency 15.500 ms oleg118-client.virtnet: 1 86261 1.35 MB/sec execute 255 sec latency 20.741 ms oleg118-client.virtnet: 1 86664 1.36 MB/sec execute 256 sec latency 16.726 ms oleg118-client.virtnet: 1 87127 1.36 MB/sec execute 257 sec latency 12.840 ms oleg118-client.virtnet: 1 87528 1.36 MB/sec execute 258 sec latency 12.931 ms oleg118-client.virtnet: 1 88071 1.37 MB/sec execute 259 sec latency 17.433 ms oleg118-client.virtnet: 1 88706 1.39 MB/sec execute 260 sec latency 17.696 ms oleg118-client.virtnet: 1 89018 1.39 MB/sec execute 261 sec latency 14.566 ms oleg118-client.virtnet: 1 89453 1.38 MB/sec execute 262 sec latency 18.726 ms oleg118-client.virtnet: 1 89786 1.39 MB/sec execute 263 sec latency 36.709 ms oleg118-client.virtnet: 1 90167 1.39 MB/sec execute 264 sec latency 18.286 ms oleg118-client.virtnet: 1 90451 1.39 MB/sec execute 265 sec latency 19.723 ms oleg118-client.virtnet: 1 90868 1.38 MB/sec execute 266 sec latency 11.846 ms oleg118-client.virtnet: 1 91322 1.39 MB/sec execute 267 sec latency 16.132 ms oleg118-client.virtnet: 1 92096 1.41 MB/sec execute 268 sec latency 18.352 ms oleg118-client.virtnet: 1 92509 1.42 MB/sec execute 269 sec latency 15.533 ms oleg118-client.virtnet: 1 92832 1.41 MB/sec execute 270 sec latency 14.722 ms oleg118-client.virtnet: 1 93172 1.41 MB/sec execute 271 sec latency 22.390 ms oleg118-client.virtnet: 1 93539 1.41 MB/sec execute 272 sec latency 16.997 ms oleg118-client.virtnet: 1 93973 1.42 MB/sec execute 273 sec latency 19.075 ms oleg118-client.virtnet: 1 94361 1.41 MB/sec execute 274 sec latency 11.500 ms oleg118-client.virtnet: 1 94853 1.42 MB/sec execute 275 sec latency 11.589 ms oleg118-client.virtnet: 1 95585 1.43 MB/sec execute 276 sec latency 17.828 ms oleg118-client.virtnet: 1 96043 1.44 MB/sec execute 277 sec latency 14.899 ms oleg118-client.virtnet: 1 96292 1.44 MB/sec execute 278 sec latency 15.706 ms oleg118-client.virtnet: 1 96676 1.44 MB/sec execute 279 sec latency 18.767 ms oleg118-client.virtnet: 1 97067 1.44 MB/sec execute 280 sec latency 16.980 ms oleg118-client.virtnet: 1 97662 1.44 MB/sec execute 281 sec latency 9.550 ms oleg118-client.virtnet: 1 98141 1.44 MB/sec execute 282 sec latency 13.273 ms oleg118-client.virtnet: 1 98696 1.46 MB/sec execute 283 sec latency 17.494 ms oleg118-client.virtnet: 1 99396 1.47 MB/sec execute 284 sec latency 14.640 ms oleg118-client.virtnet: 1 99721 1.47 MB/sec execute 285 sec latency 15.836 ms oleg118-client.virtnet: 1 100081 1.47 MB/sec execute 286 sec latency 16.222 ms oleg118-client.virtnet: 1 100411 1.47 MB/sec execute 287 sec latency 21.438 ms oleg118-client.virtnet: 1 100867 1.48 MB/sec execute 288 sec latency 17.142 ms oleg118-client.virtnet: 1 101367 1.47 MB/sec execute 289 sec latency 17.762 ms oleg118-client.virtnet: 1 101791 1.47 MB/sec execute 290 sec latency 18.597 ms oleg118-client.virtnet: 1 102736 1.49 MB/sec execute 291 sec latency 14.241 ms oleg118-client.virtnet: 1 103370 1.50 MB/sec execute 292 sec latency 8.952 ms oleg118-client.virtnet: 1 103794 1.50 MB/sec execute 293 sec latency 17.068 ms oleg118-client.virtnet: 1 104177 1.50 MB/sec execute 294 sec latency 17.453 ms oleg118-client.virtnet: 1 104595 1.51 MB/sec execute 295 sec latency 14.887 ms oleg118-client.virtnet: 1 104983 1.50 MB/sec execute 296 sec latency 13.764 ms oleg118-client.virtnet: 1 105380 1.50 MB/sec execute 297 sec latency 17.597 ms oleg118-client.virtnet: 1 106091 1.52 MB/sec execute 298 sec latency 17.998 ms oleg118-client.virtnet: 1 106594 1.53 MB/sec execute 299 sec latency 15.519 ms oleg118-client.virtnet: 1 cleanup 300 sec oleg118-client.virtnet: 0 cleanup 300 sec oleg118-client.virtnet: oleg118-client.virtnet: Operation Count AvgLat MaxLat oleg118-client.virtnet: ---------------------------------------- oleg118-client.virtnet: NTCreateX 14623 11.995 19656.723 oleg118-client.virtnet: Close 10739 5.072 19899.998 oleg118-client.virtnet: Rename 620 11.311 22.873 oleg118-client.virtnet: Unlink 2936 2.514 8.985 oleg118-client.virtnet: Qpathinfo 13276 2.020 15.601 oleg118-client.virtnet: Qfileinfo 2319 0.189 2.392 oleg118-client.virtnet: Qfsinfo 2428 0.300 7.291 oleg118-client.virtnet: Sfileinfo 1182 9.394 3403.838 oleg118-client.virtnet: Find 5120 0.536 14.264 oleg118-client.virtnet: WriteX 7228 1.179 9.964 oleg118-client.virtnet: ReadX 23012 0.026 5.529 oleg118-client.virtnet: LockX 48 1.456 2.071 oleg118-client.virtnet: UnlockX 48 1.561 2.204 oleg118-client.virtnet: Flush 1008 7.397 478.771 oleg118-client.virtnet: oleg118-client.virtnet: Throughput 1.52717 MB/sec 1 clients 1 procs max_latency=19900.022 ms oleg118-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg118-client.virtnet at Sat Nov 16 05:38:18 EST 2024 with return code 0 oleg118-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg118-client.virtnet oleg118-client.virtnet: /mnt/lustre/d70b.replay-single/oleg118-client.virtnet /mnt/lustre/d70b.replay-single/oleg118-client.virtnet oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg118-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg118-client.virtnet: removed directory: 'clients/client0' oleg118-client.virtnet: removed directory: 'clients' oleg118-client.virtnet: removed 'client.txt' oleg118-client.virtnet: /mnt/lustre/d70b.replay-single/oleg118-client.virtnet oleg118-client.virtnet: dbench successfully finished