== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 06:14:08 (1730805248) Starting client oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre Started clients oleg209-client.virtnet: 192.168.202.109@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg209-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13273 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1744 3593920 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26148 3580872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27892 7174792 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 06:14:15 (1730805255) shut down Failover mds1 to oleg209-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 06:14:30 (1730805270) targets are mounted 06:14:30 (1730805270) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1884 3591176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26592 3568584 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28476 7159760 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg209-server Stopping /mnt/lustre-mds2 (opts:) on oleg209-server oleg209-client.virtnet: looking for dbench program oleg209-client.virtnet: /usr/bin/dbench oleg209-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg209-client.virtnet oleg209-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg209-client.virtnet' oleg209-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg209-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg209-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg209-client.virtnet at Tue Nov 5 06:14:09 EST 2024 oleg209-client.virtnet: waiting for dbench pid 13298 oleg209-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg209-client.virtnet: oleg209-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg209-client.virtnet: failed to create barrier semaphore oleg209-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg209-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg209-client.virtnet: releasing clients oleg209-client.virtnet: 1 382 13.56 MB/sec warmup 1 sec latency 18.229 ms oleg209-client.virtnet: 1 796 10.73 MB/sec warmup 2 sec latency 16.519 ms oleg209-client.virtnet: 1 1418 8.67 MB/sec warmup 3 sec latency 12.544 ms oleg209-client.virtnet: 1 1814 6.58 MB/sec warmup 4 sec latency 259.968 ms oleg209-client.virtnet: 1 2180 5.35 MB/sec warmup 5 sec latency 11.201 ms oleg209-client.virtnet: 1 2917 5.58 MB/sec warmup 6 sec latency 106.706 ms oleg209-client.virtnet: 1 2917 4.78 MB/sec warmup 7 sec latency 1106.913 ms oleg209-client.virtnet: 1 2917 4.18 MB/sec warmup 8 sec latency 2107.203 ms oleg209-client.virtnet: 1 2917 3.72 MB/sec warmup 9 sec latency 3107.482 ms oleg209-client.virtnet: 1 2917 3.35 MB/sec warmup 10 sec latency 4107.704 ms oleg209-client.virtnet: 1 2917 3.04 MB/sec warmup 11 sec latency 5107.923 ms oleg209-client.virtnet: 1 2917 2.79 MB/sec warmup 12 sec latency 6108.116 ms oleg209-client.virtnet: 1 2917 2.57 MB/sec warmup 13 sec latency 7108.285 ms oleg209-client.virtnet: 1 2917 2.39 MB/sec warmup 14 sec latency 8108.548 ms oleg209-client.virtnet: 1 2917 2.23 MB/sec warmup 15 sec latency 9108.760 ms oleg209-client.virtnet: 1 2917 2.09 MB/sec warmup 16 sec latency 10108.959 ms oleg209-client.virtnet: 1 2917 1.97 MB/sec warmup 17 sec latency 11109.147 ms oleg209-client.virtnet: 1 2917 1.86 MB/sec warmup 18 sec latency 12109.273 ms oleg209-client.virtnet: 1 2917 1.76 MB/sec warmup 19 sec latency 13109.481 ms oleg209-client.virtnet: 1 2917 1.67 MB/sec warmup 20 sec latency 14109.654 ms oleg209-client.virtnet: 1 2917 1.59 MB/sec warmup 21 sec latency 15109.862 ms oleg209-client.virtnet: 1 2917 1.52 MB/sec warmup 22 sec latency 16110.107 ms oleg209-client.virtnet: 1 2917 1.46 MB/sec warmup 23 sec latency 17110.376 ms oleg209-client.virtnet: 1 2917 1.39 MB/sec warmup 24 sec latency 18110.615 ms oleg209-client.virtnet: 1 2972 1.34 MB/sec warmup 25 sec latency 19036.157 ms oleg209-client.virtnet: 1 3852 1.53 MB/sec warmup 26 sec latency 9.250 ms oleg209-client.virtnet: 1 4333 1.50 MB/sec warmup 27 sec latency 13.324 ms oleg209-client.virtnet: 1 4797 1.51 MB/sec warmup 28 sec latency 16.773 ms oleg209-client.virtnet: 1 5344 1.57 MB/sec warmup 29 sec latency 13.283 ms oleg209-client.virtnet: 1 5989 1.57 MB/sec warmup 30 sec latency 8.484 ms oleg209-client.virtnet: 1 6914 1.76 MB/sec warmup 31 sec latency 36.020 ms oleg209-client.virtnet: 1 7463 1.85 MB/sec warmup 32 sec latency 13.268 ms oleg209-client.virtnet: 1 7788 1.80 MB/sec warmup 33 sec latency 2306:14:42 (1730805282) shut down Failover mds2 to oleg209-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0001 06:14:57 (1730805297) targets are mounted 06:14:57 (1730805297) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12072 3593920 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37424 3569028 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7162948 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 06:15:10 (1730805310) shut down Failover mds1 to oleg209-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 06:15:26 (1730805326) targets are mounted 06:15:26 (1730805326) facet_failover done .451 ms oleg209-client.virtnet: 1 8245 1.80 MB/sec warmup 34 sec latency 16.377 ms oleg209-client.virtnet: 1 8839 1.84 MB/sec warmup 35 sec latency 10.452 ms oleg209-client.virtnet: 1 9482 1.84 MB/sec warmup 36 sec latency 15.488 ms oleg209-client.virtnet: 1 10619 2.07 MB/sec warmup 37 sec latency 10.332 ms oleg209-client.virtnet: 1 11012 2.06 MB/sec warmup 38 sec latency 15.656 ms oleg209-client.virtnet: 1 11390 2.02 MB/sec warmup 39 sec latency 22.353 ms oleg209-client.virtnet: 1 11687 2.01 MB/sec warmup 40 sec latency 19.107 ms oleg209-client.virtnet: 1 12227 2.04 MB/sec warmup 41 sec latency 13.744 ms oleg209-client.virtnet: 1 12660 2.00 MB/sec warmup 42 sec latency 19.571 ms oleg209-client.virtnet: 1 13096 1.99 MB/sec warmup 43 sec latency 17.144 ms oleg209-client.virtnet: 1 13826 2.08 MB/sec warmup 44 sec latency 17.380 ms oleg209-client.virtnet: 1 14336 2.14 MB/sec warmup 45 sec latency 18.364 ms oleg209-client.virtnet: 1 14661 2.12 MB/sec warmup 46 sec latency 15.741 ms oleg209-client.virtnet: 1 14945 2.08 MB/sec warmup 47 sec latency 335.372 ms oleg209-client.virtnet: 1 14945 2.04 MB/sec warmup 48 sec latency 1335.517 ms oleg209-client.virtnet: 1 14945 2.00 MB/sec warmup 49 sec latency 2335.750 ms oleg209-client.virtnet: 1 14945 1.96 MB/sec warmup 50 sec latency 3336.050 ms oleg209-client.virtnet: 1 14945 1.92 MB/sec warmup 51 sec latency 4336.364 ms oleg209-client.virtnet: 1 15125 1.89 MB/sec warmup 52 sec latency 4769.636 ms oleg209-client.virtnet: 1 15471 1.88 MB/sec warmup 53 sec latency 19.366 ms oleg209-client.virtnet: 1 15835 1.90 MB/sec warmup 54 sec latency 16.780 ms oleg209-client.virtnet: 1 16134 1.87 MB/sec warmup 55 sec latency 19.679 ms oleg209-client.virtnet: 1 16544 1.86 MB/sec warmup 56 sec latency 11.144 ms oleg209-client.virtnet: 1 17070 1.91 MB/sec warmup 57 sec latency 17.647 ms oleg209-client.virtnet: 1 17389 1.91 MB/sec warmup 58 sec latency 459.830 ms oleg209-client.virtnet: 1 17872 1.95 MB/sec warmup 59 sec latency 485.252 ms oleg209-client.virtnet: 1 18332 0.02 MB/sec execute 1 sec latency 288.775 ms oleg209-client.virtnet: 1 18332 0.01 MB/sec execute 2 sec latency 1289.154 ms oleg209-client.virtnet: 1 18332 0.01 MB/sec execute 3 sec latency 2289.341 ms oleg209-client.virtnet: 1 18332 0.01 MB/sec execute 4 sec latency 3289.610 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 5 sec latency 4289.868 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 6 sec latency 5290.128 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 7 sec latency 6290.367 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 8 sec latency 7290.621 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 9 sec latency 8290.899 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 10 sec latency 9291.143 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 11 sec latency 10291.389 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 12 sec latency 11291.638 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 13 sec latency 12291.940 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 14 sec latency 13292.186 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 15 sec latency 14292.401 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 16 sec latency 15292.619 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 17 sec latency 16292.879 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 18 sec latency 17293.085 ms oleg209-client.virtnet: 1 18332 0.00 MB/secoleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12072 3594892 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3569268 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7164160 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg209-server Stopping /mnt/lustre-mds2 (opts:) on oleg209-server 06:15:39 (1730805339) shut down Failover mds2 to oleg209-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0001 06:15:54 (1730805354) targets are mounted 06:15:54 (1730805354) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12072 3593920 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37424 3569084 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7163004 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 06:16:07 (1730805367) shut down execute 19 sec latency 18293.268 ms oleg209-client.virtnet: 1 18332 0.00 MB/sec execute 20 sec latency 19293.477 ms oleg209-client.virtnet: 1 18432 0.01 MB/sec execute 21 sec latency 20009.550 ms oleg209-client.virtnet: 1 18722 0.04 MB/sec execute 22 sec latency 17.257 ms oleg209-client.virtnet: 1 19150 0.22 MB/sec execute 23 sec latency 17.991 ms oleg209-client.virtnet: 1 19446 0.22 MB/sec execute 24 sec latency 16.275 ms oleg209-client.virtnet: 1 19846 0.23 MB/sec execute 25 sec latency 11.577 ms oleg209-client.virtnet: 1 20275 0.30 MB/sec execute 26 sec latency 17.822 ms oleg209-client.virtnet: 1 21136 0.54 MB/sec execute 27 sec latency 13.651 ms oleg209-client.virtnet: 1 21570 0.67 MB/sec execute 28 sec latency 15.943 ms oleg209-client.virtnet: 1 21812 0.66 MB/sec execute 29 sec latency 18.680 ms oleg209-client.virtnet: 1 22134 0.66 MB/sec execute 30 sec latency 27.638 ms oleg209-client.virtnet: 1 22530 0.69 MB/sec execute 31 sec latency 14.356 ms oleg209-client.virtnet: 1 22908 0.76 MB/sec execute 32 sec latency 13.425 ms oleg209-client.virtnet: 1 23173 0.74 MB/sec execute 33 sec latency 19.587 ms oleg209-client.virtnet: 1 23529 0.73 MB/sec execute 34 sec latency 12.040 ms oleg209-client.virtnet: 1 23980 0.81 MB/sec execute 35 sec latency 19.309 ms oleg209-client.virtnet: 1 24817 1.02 MB/sec execute 36 sec latency 17.205 ms oleg209-client.virtnet: 1 25298 1.03 MB/sec execute 37 sec latency 13.093 ms oleg209-client.virtnet: 1 25757 1.03 MB/sec execute 38 sec latency 11.373 ms oleg209-client.virtnet: 1 26111 1.04 MB/sec execute 39 sec latency 15.914 ms oleg209-client.virtnet: 1 26596 1.09 MB/sec execute 40 sec latency 17.482 ms oleg209-client.virtnet: 1 27087 1.08 MB/sec execute 41 sec latency 12.482 ms oleg209-client.virtnet: 1 27631 1.17 MB/sec execute 42 sec latency 16.224 ms oleg209-client.virtnet: 1 28375 1.30 MB/sec execute 43 sec latency 17.610 ms oleg209-client.virtnet: 1 28733 1.30 MB/sec execute 44 sec latency 18.981 ms oleg209-client.virtnet: 1 29036 1.28 MB/sec execute 45 sec latency 264.550 ms oleg209-client.virtnet: 1 29036 1.25 MB/sec execute 46 sec latency 1264.776 ms oleg209-client.virtnet: 1 29036 1.22 MB/sec execute 47 sec latency 2265.037 ms oleg209-client.virtnet: 1 29036 1.20 MB/sec execute 48 sec latency 3265.278 ms oleg209-client.virtnet: 1 29204 1.18 MB/sec execute 49 sec latency 3979.699 ms oleg209-client.virtnet: 1 29651 1.19 MB/sec execute 50 sec latency 14.643 ms oleg209-client.virtnet: 1 30019 1.23 MB/sec execute 51 sec latency 16.361 ms oleg209-client.virtnet: 1 30284 1.21 MB/sec execute 52 sec latency 21.355 ms oleg209-client.virtnet: 1 30693 1.20 MB/sec execute 53 sec latency 11.772 ms oleg209-client.virtnet: 1 31183 1.27 MB/sec execute 54 sec latency 16.907 ms oleg209-client.virtnet: 1 31622 1.29 MB/sec execute 55 sec latency 503.505 ms oleg209-client.virtnet: 1 32187 1.36 MB/sec execute 56 sec latency 15.232 ms oleg209-client.virtnet: 1 32446 1.34 MB/sec execute 57 sec latency 16.019 ms oleg209-client.virtnet: 1 32501 1.32 MB/sec execute 58 sec latency 782.964 ms oleg209-client.virtnet: 1 32501 1.30 MB/sec execute 59 sec latency 1783.287 ms oleg209-client.virtnet: 1 32501 1.28 MB/sec execute 60 sec latency 2783.561 ms oleg209-client.virtnet: 1 32501 1.25 MB/sec execute 61 sec latency 3783.839 ms oleg209-client.virtnet: 1 32501 1.23 MB/sec execute 62 sec latency 4784.055 ms oleg209-client.virtnet: 1 32501 1.22 MB/sec execute 63 sec latency 5784.286 ms oleg209-client.virtnet: Failover mds1 to oleg209-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 06:16:21 (1730805381) targets are mounted 06:16:21 (1730805381) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11912 3592216 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37652 3569060 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7161276 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg209-server Stopping /mnt/lustre-mds2 (opts:) on oleg209-server 06:16:34 (1730805394) shut down Failover mds2 to oleg209-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0001 06:16:49 (1730805409) targets are mounted 06:16:49 (1730805409) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11920 3594212 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37580 3568788 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7163000 1% /mnt/lustre 1 32501 1.20 MB/sec execute 64 sec latency 6784.583 ms oleg209-client.virtnet: 1 32501 1.18 MB/sec execute 65 sec latency 7784.876 ms oleg209-client.virtnet: 1 32501 1.16 MB/sec execute 66 sec latency 8785.179 ms oleg209-client.virtnet: 1 32501 1.14 MB/sec execute 67 sec latency 9785.408 ms oleg209-client.virtnet: 1 32501 1.13 MB/sec execute 68 sec latency 10785.638 ms oleg209-client.virtnet: 1 32501 1.11 MB/sec execute 69 sec latency 11785.822 ms oleg209-client.virtnet: 1 32501 1.09 MB/sec execute 70 sec latency 12785.997 ms oleg209-client.virtnet: 1 32501 1.08 MB/sec execute 71 sec latency 13786.187 ms oleg209-client.virtnet: 1 32501 1.06 MB/sec execute 72 sec latency 14786.330 ms oleg209-client.virtnet: 1 32501 1.05 MB/sec execute 73 sec latency 15786.544 ms oleg209-client.virtnet: 1 32501 1.03 MB/sec execute 74 sec latency 16786.839 ms oleg209-client.virtnet: 1 32501 1.02 MB/sec execute 75 sec latency 17787.112 ms oleg209-client.virtnet: 1 32501 1.01 MB/sec execute 76 sec latency 18787.457 ms oleg209-client.virtnet: 1 32712 1.00 MB/sec execute 77 sec latency 19227.928 ms oleg209-client.virtnet: 1 33132 1.01 MB/sec execute 78 sec latency 16.791 ms oleg209-client.virtnet: 1 33505 1.03 MB/sec execute 79 sec latency 17.022 ms oleg209-client.virtnet: 1 33948 1.03 MB/sec execute 80 sec latency 15.222 ms oleg209-client.virtnet: 1 34439 1.04 MB/sec execute 81 sec latency 12.838 ms oleg209-client.virtnet: 1 35208 1.10 MB/sec execute 82 sec latency 16.227 ms oleg209-client.virtnet: 1 35780 1.15 MB/sec execute 83 sec latency 18.624 ms oleg209-client.virtnet: 1 36161 1.14 MB/sec execute 84 sec latency 13.811 ms oleg209-client.virtnet: 1 36665 1.15 MB/sec execute 85 sec latency 12.681 ms oleg209-client.virtnet: 1 37258 1.17 MB/sec execute 86 sec latency 11.271 ms oleg209-client.virtnet: 1 37876 1.18 MB/sec execute 87 sec latency 8.860 ms oleg209-client.virtnet: 1 38401 1.22 MB/sec execute 88 sec latency 18.134 ms oleg209-client.virtnet: 1 39012 1.27 MB/sec execute 89 sec latency 17.734 ms oleg209-client.virtnet: 1 39314 1.27 MB/sec execute 90 sec latency 15.717 ms oleg209-client.virtnet: 1 39547 1.26 MB/sec execute 91 sec latency 15.787 ms oleg209-client.virtnet: 1 39989 1.26 MB/sec execute 92 sec latency 22.224 ms oleg209-client.virtnet: 1 40481 1.29 MB/sec execute 93 sec latency 17.704 ms oleg209-client.virtnet: 1 40824 1.28 MB/sec execute 94 sec latency 15.790 ms oleg209-client.virtnet: 1 41209 1.27 MB/sec execute 95 sec latency 11.747 ms oleg209-client.virtnet: 1 41548 1.28 MB/sec execute 96 sec latency 17.608 ms oleg209-client.virtnet: 1 42243 1.32 MB/sec execute 97 sec latency 17.395 ms oleg209-client.virtnet: 1 42726 1.35 MB/sec execute 98 sec latency 17.516 ms oleg209-client.virtnet: 1 43070 1.35 MB/sec execute 99 sec latency 15.085 ms oleg209-client.virtnet: 1 43412 1.34 MB/sec execute 100 sec latency 163.526 ms oleg209-client.virtnet: 1 43412 1.33 MB/sec execute 101 sec latency 1163.808 ms oleg209-client.virtnet: 1 43412 1.32 MB/sec execute 102 sec latency 2164.083 ms oleg209-client.virtnet: 1 43412 1.30 MB/sec execute 103 sec latency 3164.498 ms oleg209-client.virtnet: 1 43542 1.30 MB/sec execute 104 sec latency 3842.670 ms oleg209-client.virtnet: 1 44154 1.32 MB/sec execute 105 sec latency 16.249 ms oleg209-client.virtnet: 1 44461 1.31 MB/sec execute 106 sec latency 18.285 ms oleg209-client.virtnet: 1 44852 1.31 MB/sec execute 107 sec latency 11.730 ms oleg209-client.virtnet: 1 45316 1.34 MB/sec execute 108 sectest_70b fail mds1 7 times Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 06:17:01 (1730805421) shut down Failover mds1 to oleg209-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 06:17:17 (1730805437) targets are mounted 06:17:17 (1730805437) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12036 3594028 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37632 3568960 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49668 7162988 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg209-server Stopping /mnt/lustre-mds2 (opts:) on oleg209-server 06:17:29 (1730805449) shut down Failover mds2 to oleg209-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 latency 18.274 ms oleg209-client.virtnet: 1 46088 1.39 MB/sec execute 109 sec latency 18.648 ms oleg209-client.virtnet: 1 46511 1.39 MB/sec execute 110 sec latency 15.124 ms oleg209-client.virtnet: 1 46851 1.38 MB/sec execute 111 sec latency 16.627 ms oleg209-client.virtnet: 1 47145 1.37 MB/sec execute 112 sec latency 93.140 ms oleg209-client.virtnet: 1 47145 1.36 MB/sec execute 113 sec latency 1093.407 ms oleg209-client.virtnet: 1 47145 1.35 MB/sec execute 114 sec latency 2093.739 ms oleg209-client.virtnet: 1 47145 1.34 MB/sec execute 115 sec latency 3093.915 ms oleg209-client.virtnet: 1 47145 1.33 MB/sec execute 116 sec latency 4094.079 ms oleg209-client.virtnet: 1 47145 1.32 MB/sec execute 117 sec latency 5094.363 ms oleg209-client.virtnet: 1 47145 1.30 MB/sec execute 118 sec latency 6094.692 ms oleg209-client.virtnet: 1 47145 1.29 MB/sec execute 119 sec latency 7094.903 ms oleg209-client.virtnet: 1 47145 1.28 MB/sec execute 120 sec latency 8095.152 ms oleg209-client.virtnet: 1 47145 1.27 MB/sec execute 121 sec latency 9095.363 ms oleg209-client.virtnet: 1 47145 1.26 MB/sec execute 122 sec latency 10095.607 ms oleg209-client.virtnet: 1 47145 1.25 MB/sec execute 123 sec latency 11095.825 ms oleg209-client.virtnet: 1 47145 1.24 MB/sec execute 124 sec latency 12096.046 ms oleg209-client.virtnet: 1 47145 1.23 MB/sec execute 125 sec latency 13096.262 ms oleg209-client.virtnet: 1 47145 1.22 MB/sec execute 126 sec latency 14096.513 ms oleg209-client.virtnet: 1 47145 1.21 MB/sec execute 127 sec latency 15096.764 ms oleg209-client.virtnet: 1 47145 1.20 MB/sec execute 128 sec latency 16096.982 ms oleg209-client.virtnet: 1 47145 1.19 MB/sec execute 129 sec latency 17097.160 ms oleg209-client.virtnet: 1 47145 1.18 MB/sec execute 130 sec latency 18097.483 ms oleg209-client.virtnet: 1 47145 1.18 MB/sec execute 131 sec latency 19097.792 ms oleg209-client.virtnet: 1 47317 1.17 MB/sec execute 132 sec latency 19815.478 ms oleg209-client.virtnet: 1 47874 1.19 MB/sec execute 133 sec latency 11.701 ms oleg209-client.virtnet: 1 48524 1.19 MB/sec execute 134 sec latency 7.989 ms oleg209-client.virtnet: 1 49420 1.24 MB/sec execute 135 sec latency 14.631 ms oleg209-client.virtnet: 1 50033 1.27 MB/sec execute 136 sec latency 12.593 ms oleg209-client.virtnet: 1 50519 1.26 MB/sec execute 137 sec latency 10.608 ms oleg209-client.virtnet: 1 50911 1.26 MB/sec execute 138 sec latency 17.699 ms oleg209-client.virtnet: 1 51392 1.28 MB/sec execute 139 sec latency 14.285 ms oleg209-client.virtnet: 1 52051 1.28 MB/sec execute 140 sec latency 9.276 ms oleg209-client.virtnet: 1 52609 1.31 MB/sec execute 141 sec latency 16.594 ms oleg209-client.virtnet: 1 53255 1.34 MB/sec execute 142 sec latency 17.940 ms oleg209-client.virtnet: 1 53538 1.34 MB/sec execute 143 sec latency 15.866 ms oleg209-client.virtnet: 1 53785 1.33 MB/sec execute 144 sec latency 15.627 ms oleg209-client.virtnet: 1 54112 1.32 MB/sec execute 145 sec latency 21.852 ms oleg209-client.virtnet: 1 54446 1.32 MB/sec execute 146 sec latency 17.253 ms oleg209-client.virtnet: 1 54808 1.34 MB/sec execute 147 sec latency 18.564 ms oleg209-client.virtnet: 1 55087 1.33 MB/sec execute 148 sec latency 20.375 ms oleg209-client.virtnet: 1 55572 1.33 MB/sec execute 149 sec latency 11.590 ms oleg209-client.virtnet: 1 56511 1.37 MB/sec execute 150 sec latency 10.437 ms oleg209-client.virtnet: 1 57172 1.39 MB/sec execute 151 sec latency 11.868 ms oleg209-client.virtnet: 1 57704 1.39 MB/sec execute 152 sec latency 12.332 ms oleg209-client.virtnet:oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0001 06:17:43 (1730805463) targets are mounted 06:17:43 (1730805463) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12152 3591808 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37504 3568712 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49656 7160520 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 06:17:55 (1730805475) shut down Failover mds1 to oleg209-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 06:18:10 (1730805490) targets are mounted 06:18:10 (1730805490) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14112 3589916 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37640 3567640 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51752 7157556 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg209-server Stopping /mnt/lustre-mds2 (opts:) on oleg209-server 06:18:23 (1730805503) shut down 1 58206 1.41 MB/sec execute 153 sec latency 15.183 ms oleg209-client.virtnet: 1 58556 1.40 MB/sec execute 154 sec latency 225.377 ms oleg209-client.virtnet: 1 58556 1.39 MB/sec execute 155 sec latency 1225.567 ms oleg209-client.virtnet: 1 58556 1.38 MB/sec execute 156 sec latency 2225.768 ms oleg209-client.virtnet: 1 58556 1.37 MB/sec execute 157 sec latency 3225.938 ms oleg209-client.virtnet: 1 58796 1.37 MB/sec execute 158 sec latency 3734.609 ms oleg209-client.virtnet: 1 59193 1.37 MB/sec execute 159 sec latency 16.622 ms oleg209-client.virtnet: 1 59772 1.39 MB/sec execute 160 sec latency 16.409 ms oleg209-client.virtnet: 1 60633 1.42 MB/sec execute 161 sec latency 12.956 ms oleg209-client.virtnet: 1 61227 1.42 MB/sec execute 162 sec latency 8.701 ms oleg209-client.virtnet: 1 61817 1.44 MB/sec execute 163 sec latency 143.521 ms oleg209-client.virtnet: 1 62309 1.43 MB/sec execute 164 sec latency 207.401 ms oleg209-client.virtnet: 1 63095 1.46 MB/sec execute 165 sec latency 9.215 ms oleg209-client.virtnet: 1 63295 1.46 MB/sec execute 166 sec latency 806.332 ms oleg209-client.virtnet: 1 63295 1.45 MB/sec execute 167 sec latency 1806.583 ms oleg209-client.virtnet: 1 63295 1.44 MB/sec execute 168 sec latency 2807.014 ms oleg209-client.virtnet: 1 63295 1.43 MB/sec execute 169 sec latency 3807.208 ms oleg209-client.virtnet: 1 63295 1.42 MB/sec execute 170 sec latency 4807.444 ms oleg209-client.virtnet: 1 63295 1.42 MB/sec execute 171 sec latency 5807.643 ms oleg209-client.virtnet: 1 63295 1.41 MB/sec execute 172 sec latency 6807.942 ms oleg209-client.virtnet: 1 63295 1.40 MB/sec execute 173 sec latency 7808.140 ms oleg209-client.virtnet: 1 63295 1.39 MB/sec execute 174 sec latency 8808.412 ms oleg209-client.virtnet: 1 63295 1.38 MB/sec execute 175 sec latency 9808.639 ms oleg209-client.virtnet: 1 63295 1.38 MB/sec execute 176 sec latency 10808.827 ms oleg209-client.virtnet: 1 63295 1.37 MB/sec execute 177 sec latency 11808.963 ms oleg209-client.virtnet: 1 63295 1.36 MB/sec execute 178 sec latency 12809.118 ms oleg209-client.virtnet: 1 63295 1.35 MB/sec execute 179 sec latency 13809.421 ms oleg209-client.virtnet: 1 63295 1.34 MB/sec execute 180 sec latency 14809.671 ms oleg209-client.virtnet: 1 63295 1.34 MB/sec execute 181 sec latency 15809.886 ms oleg209-client.virtnet: 1 63295 1.33 MB/sec execute 182 sec latency 16810.149 ms oleg209-client.virtnet: 1 63295 1.32 MB/sec execute 183 sec latency 17810.382 ms oleg209-client.virtnet: 1 63295 1.32 MB/sec execute 184 sec latency 18810.551 ms oleg209-client.virtnet: 1 63596 1.32 MB/sec execute 185 sec latency 19482.658 ms oleg209-client.virtnet: 1 64020 1.33 MB/sec execute 186 sec latency 16.554 ms oleg209-client.virtnet: 1 64348 1.33 MB/sec execute 187 sec latency 16.320 ms oleg209-client.virtnet: 1 64687 1.32 MB/sec execute 188 sec latency 21.010 ms oleg209-client.virtnet: 1 65135 1.33 MB/sec execute 189 sec latency 18.342 ms oleg209-client.virtnet: 1 65665 1.34 MB/sec execute 190 sec latency 13.979 ms oleg209-client.virtnet: 1 66013 1.33 MB/sec execute 191 sec latency 13.922 ms oleg209-client.virtnet: 1 66349 1.34 MB/sec execute 192 sec latency 14.901 ms oleg209-client.virtnet: 1 66883 1.35 MB/sec execute 193 sec latency 17.777 ms oleg209-client.virtnet: 1 67503 1.37 MB/sec execute 194 sec latency 16.973 ms oleg209-client.virtnet: 1 67776 1.37 MB/sec execute 195 sec latency 16.280 ms oleg209-client.virtnet: 1 68075 1.37 MB/sec execute 196 sec latency 17.286 ms oleg209-client.virtnet: 1 68365 1.36 MB/sec eFailover mds2 to oleg209-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0001 06:18:38 (1730805518) targets are mounted 06:18:38 (1730805518) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12036 3594048 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37668 3566476 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49704 7160524 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 06:18:51 (1730805531) shut down Failover mds1 to oleg209-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 06:19:05 (1730805545) targets are mounted 06:19:05 (1730805545) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec xecute 197 sec latency 19.619 ms oleg209-client.virtnet: 1 68698 1.36 MB/sec execute 198 sec latency 17.861 ms oleg209-client.virtnet: 1 69085 1.37 MB/sec execute 199 sec latency 18.397 ms oleg209-client.virtnet: 1 69430 1.37 MB/sec execute 200 sec latency 21.604 ms oleg209-client.virtnet: 1 69839 1.37 MB/sec execute 201 sec latency 18.341 ms oleg209-client.virtnet: 1 70429 1.39 MB/sec execute 202 sec latency 18.263 ms oleg209-client.virtnet: 1 71084 1.41 MB/sec execute 203 sec latency 15.205 ms oleg209-client.virtnet: 1 71372 1.41 MB/sec execute 204 sec latency 15.226 ms oleg209-client.virtnet: 1 71775 1.40 MB/sec execute 205 sec latency 16.143 ms oleg209-client.virtnet: 1 72106 1.40 MB/sec execute 206 sec latency 17.923 ms oleg209-client.virtnet: 1 72527 1.41 MB/sec execute 207 sec latency 17.828 ms oleg209-client.virtnet: 1 72875 1.41 MB/sec execute 208 sec latency 20.918 ms oleg209-client.virtnet: 1 73289 1.41 MB/sec execute 209 sec latency 72.779 ms oleg209-client.virtnet: 1 73289 1.40 MB/sec execute 210 sec latency 1072.980 ms oleg209-client.virtnet: 1 73289 1.39 MB/sec execute 211 sec latency 2073.248 ms oleg209-client.virtnet: 1 73289 1.39 MB/sec execute 212 sec latency 3073.605 ms oleg209-client.virtnet: 1 73422 1.38 MB/sec execute 213 sec latency 3668.978 ms oleg209-client.virtnet: 1 74268 1.41 MB/sec execute 214 sec latency 14.913 ms oleg209-client.virtnet: 1 74744 1.42 MB/sec execute 215 sec latency 12.940 ms oleg209-client.virtnet: 1 75113 1.42 MB/sec execute 216 sec latency 12.469 ms oleg209-client.virtnet: 1 75458 1.41 MB/sec execute 217 sec latency 18.014 ms oleg209-client.virtnet: 1 75810 1.41 MB/sec execute 218 sec latency 17.647 ms oleg209-client.virtnet: 1 76142 1.42 MB/sec execute 219 sec latency 262.071 ms oleg209-client.virtnet: 1 76491 1.42 MB/sec execute 220 sec latency 17.818 ms oleg209-client.virtnet: 1 76952 1.42 MB/sec execute 221 sec latency 13.598 ms oleg209-client.virtnet: 1 76980 1.41 MB/sec execute 222 sec latency 917.252 ms oleg209-client.virtnet: 1 76980 1.40 MB/sec execute 223 sec latency 1917.502 ms oleg209-client.virtnet: 1 76980 1.40 MB/sec execute 224 sec latency 2917.854 ms oleg209-client.virtnet: 1 76980 1.39 MB/sec execute 225 sec latency 3918.195 ms oleg209-client.virtnet: 1 76980 1.39 MB/sec execute 226 sec latency 4918.533 ms oleg209-client.virtnet: 1 76980 1.38 MB/sec execute 227 sec latency 5918.899 ms oleg209-client.virtnet: 1 76980 1.37 MB/sec execute 228 sec latency 6919.178 ms oleg209-client.virtnet: 1 76980 1.37 MB/sec execute 229 sec latency 7919.508 ms oleg209-client.virtnet: 1 76980 1.36 MB/sec execute 230 sec latency 8919.956 ms oleg209-client.virtnet: 1 76980 1.36 MB/sec execute 231 sec latency 9920.272 ms oleg209-client.virtnet: 1 76980 1.35 MB/sec execute 232 sec latency 10920.535 ms oleg209-client.virtnet: 1 76980 1.34 MB/sec execute 233 sec latency 11920.757 ms oleg209-client.virtnet: 1 76980 1.34 MB/sec execute 234 sec latency 12921.027 ms oleg209-client.virtnet: 1 76980 1.33 MB/sec execute 235 sec latency 13921.301 ms oleg209-client.virtnet: 1 76980 1.33 MB/sec execute 236 sec latency 14921.453 ms oleg209-client.virtnet: 1 76980 1.32 MB/sec execute 237 sec latency 15921.779 ms oleg209-client.virtnet: 1 76980 1.32 MB/sec execute 238 sec latency 16922.027 ms oleg209-client.virtnet: 1 76980 1.31 MB/sec execute 239 sec latency 17922.257 ms oleg209-client.virtnet: 1 76980 1.30 MB/sec execute 240 sec latency 18922.550 ms oleg209-client.virtnet: 1 77302 1.32 MB/sec execute 241 sec latency 19412.047 ms oleg209-client.virtnet: 1 78120 1.34 MB/sec execute 242 sec latency 15.215 ms oleg209-client.virtnet: 1 78435 1.34 MB/sec execute 243 sec latency 17.954 ms oleg209-client.virtnet: 1 78692 1.33 MB/sec execute 244 sec latency 15.553 ms oleg209-client.virtnet: 1 78997 1.33 MB/sec execute 245 sec latency 23.796 ms oleg209-client.virtnet: 1 79364 1.33 MB/sec execute 246 sec latency 16.543 ms oleg209-client.virtnet: 1 79717 1.34 MB/sec execute 247 sec latency 15.912 ms oleg209-client.virtnet: 1 80089 1.33 MB/sec execute 248 sec latency 20.920 ms oleg209-client.virtnet: 1 80471 1.33 MB/sec execute 249 sec latency 12.147 ms oleg209-client.virtnet: 1 80981 1.35 MB/sec execute 250 sec latency 18.409 ms oleg209-client.virtnet: 1 81719 1.37 MB/sec execute 251 sec latency 19.035 ms oleg209-client.virtnet: 1 82061 1.37 MB/sec execute 252 sec latency 14.475 ms oleg209-client.virtnet: 1 82382 1.36 MB/sec execute 253 sec latency 19.696 ms oleg209-client.virtnet: 1 82718 1.36 MB/sec execute 254 sec latency 17.873 ms oleg209-client.virtnet: 1 83426 1.37 MB/sec execute 255 sec latency 8.979 ms oleg209-client.virtnet: 1 84127 1.37 MB/sec execute 256 sec latency 8.726 ms oleg209-client.virtnet: 1 85296 1.41 MB/sec execute 257 sec latency 9.370 ms oleg209-client.virtnet: 1 85877 1.41 MB/sec execute 258 sec latency 10.846 ms oleg209-client.virtnet: 1 86355 1.41 MB/sec execute 259 sec latency 15.359 ms oleg209-client.virtnet: 1 86762 1.42 MB/sec execute 260 sec latency 15.559 ms oleg209-client.virtnet: 1 87047 1.41 MB/sec execute 261 sec latency 19.782 ms oleg209-client.virtnet: 1 87437 1.41 MB/sec execute 262 sec latency 11.408 ms oleg209-client.virtnet: 1 87832 1.42 MB/sec execute 263 sec latency 17.427 ms oleg209-client.virtnet: 1 88534 1.43 MB/sec execute 264 sec latency 17.585 ms oleg209-client.virtnet: 1 88939 1.44 MB/sec execute 265 sec latency 15.343 ms oleg209-client.virtnet: 1 89175 1.44 MB/sec execute 266 sec latency 15.825 ms oleg209-client.virtnet: 1 89517 1.43 MB/sec execute 267 sec latency 16.640 ms oleg209-client.virtnet: 1 89906 1.43 MB/sec execute 268 sec latency 14.037 ms oleg209-client.virtnet: 1 90411 1.44 MB/sec execute 269 sec latency 11.719 ms oleg209-client.virtnet: 1 90775 1.44 MB/sec execute 270 sec latency 14.371 ms oleg209-client.virtnet: 1 91217 1.44 MB/sec execute 271 sec latency 13.627 ms oleg209-client.virtnet: 1 92075 1.46 MB/sec execute 272 sec latency 14.288 ms oleg209-client.virtnet: 1 92527 1.47 MB/sec execute 273 sec latency 12.880 ms oleg209-client.virtnet: 1 92889 1.47 MB/sec execute 274 sec latency 12.328 ms oleg209-client.virtnet: 1 93259 1.46 MB/sec execute 275 sec latency 14.234 ms oleg209-client.virtnet: 1 93750 1.47 MB/sec execute 276 sec latency 12.851 ms oleg209-client.virtnet: 1 94115 1.47 MB/sec execute 277 sec latency 14.396 ms oleg209-client.virtnet: 1 94610 1.47 MB/sec execute 278 sec latency 9.889 ms oleg209-client.virtnet: 1 95170 1.48 MB/sec execute 279 sec latency 14.688 ms oleg209-client.virtnet: 1 95885 1.50 MB/sec execute 280 sec latency 13.980 ms oleg209-client.virtnet: 1 96241 1.50 MB/sec execute 281 sec latency 11.828 ms oleg209-client.virtnet: 1 96617 1.49 MB/sec execute 282 sec latency 14.089 ms oleg209-client.virtnet: 1 97021 1.49 MB/sec execute 283 sec latency 13.595 ms oleg209-client.virtnet: 1 97441 1.50 MB/sec execute 284 sec latency 13.812 ms oleg209-client.virtnet: 1 97865 1.50 MB/sec execute 285 sec latency 17.815 ms oleg209-client.virtnet: 1 98291 1.50 MB/sec execute 286 sec latency 15.810 ms oleg209-client.virtnet: 1 99166 1.52 MB/sec execute 287 sec latency 16.705 ms oleg209-client.virtnet: 1 99702 1.53 MB/sec execute 288 sec latency 12.071 ms oleg209-client.virtnet: 1 100219 1.52 MB/sec execute 289 sec latency 13.466 ms oleg209-client.virtnet: 1 100915 1.53 MB/sec execute 290 sec latency 9.232 ms oleg209-client.virtnet: 1 101545 1.53 MB/sec execute 291 sec latency 9.367 ms oleg209-client.virtnet: 1 102554 1.55 MB/sec execute 292 sec latency 9.099 ms oleg209-client.virtnet: 1 103304 1.57 MB/sec execute 293 sec latency 8.493 ms oleg209-client.virtnet: 1 103895 1.56 MB/sec execute 294 sec latency 11.843 ms oleg209-client.virtnet: 1 104600 1.57 MB/sec execute 295 sec latency 8.460 ms oleg209-client.virtnet: 1 105309 1.57 MB/sec execute 296 sec latency 9.853 ms oleg209-client.virtnet: 1 106461 1.60 MB/sec execute 297 sec latency 8.652 ms oleg209-client.virtnet: 1 107015 1.60 MB/sec execute 298 sec latency 7.847 ms oleg209-client.virtnet: 1 107384 1.60 MB/sec execute 299 sec latency 18.237 ms oleg209-client.virtnet: 1 cleanup 300 sec oleg209-client.virtnet: 0 cleanup 300 sec oleg209-client.virtnet: oleg209-client.virtnet: Operation Count AvgLat MaxLat oleg209-client.virtnet: ---------------------------------------- oleg209-client.virtnet: NTCreateX 15527 10.624 19815.469 oleg209-client.virtnet: Close 11396 1.314 5.806 oleg209-client.virtnet: Rename 660 10.387 21.585 oleg209-client.virtnet: Unlink 3156 8.637 20009.522 oleg209-client.virtnet: Qpathinfo 14064 3.763 19227.895 oleg209-client.virtnet: Qfileinfo 2441 0.179 2.323 oleg209-client.virtnet: Qfsinfo 2586 1.693 3668.950 oleg209-client.virtnet: Sfileinfo 1284 6.075 14.591 oleg209-client.virtnet: Find 5438 0.513 19.147 oleg209-client.virtnet: WriteX 7660 1.081 8.404 oleg209-client.virtnet: ReadX 24304 0.024 5.706 oleg209-client.virtnet: LockX 50 1.281 2.082 oleg209-client.virtnet: UnlockX 50 1.388 2.146 oleg209-client.virtnet: Flush 1099 7.136 503.480 oleg209-client.virtnet: oleg209-client.virtnet: Throughput 1.60166 MB/sec 1 clients 1 procs max_latency=20009.550 ms oleg209-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg209-client.virtnet at Tue Nov 5 06:20:09 EST 2024 with return code 0 oleg209-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg209-client.virtnet oleg209-client.virtnet: /mnt/lustre/d70b.replay-single/oleg209-client.virtnet /mnt/lustre/d70b.replay-single/oleg209-client.virtnet oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg209-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg209-client.virtnet: removed directory: 'clients/client0' oleg209-client.virtnet: removed directory: 'clients' oleg209-client.virtnet: removed 'client.txt' oleg209-client.virtnet: /mnt/lustre/d70b.replay-single/oleg209-client.virtnet oleg209-client.virtnet: dbench successfully finished