== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 19:48:48 (1723247328) Starting client oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre Started clients oleg120-client.virtnet: 192.168.201.120@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg120-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=12479 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3593964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26144 3580876 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7174840 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg120-server Stopping /mnt/lustre-mds1 (opts:) on oleg120-server 19:48:57 (1723247337) shut down Failover mds1 to oleg120-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 19:49:12 (1723247352) targets are mounted 19:49:12 (1723247352) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2144 3593712 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26400 3568880 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28544 7162592 1% /mnt/lustre oleg120-client.virtnet: looking for dbench program oleg120-client.virtnet: /usr/bin/dbench oleg120-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg120-client.virtnet oleg120-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg120-client.virtnet' oleg120-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg120-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg120-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg120-client.virtnet at Fri Aug 9 19:48:49 EDT 2024 oleg120-client.virtnet: waiting for dbench pid 12504 oleg120-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg120-client.virtnet: oleg120-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg120-client.virtnet: failed to create barrier semaphore oleg120-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg120-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg120-client.virtnet: releasing clients oleg120-client.virtnet: 1 306 11.22 MB/sec warmup 1 sec latency 23.262 ms oleg120-client.virtnet: 1 669 10.58 MB/sec warmup 2 sec latency 18.527 ms oleg120-client.virtnet: 1 952 7.27 MB/sec warmup 3 sec latency 17.465 ms oleg120-client.virtnet: 1 1273 5.79 MB/sec warmup 4 sec latency 17.495 ms oleg120-client.virtnet: 1 1679 5.24 MB/sec warmup 5 sec latency 16.969 ms oleg120-client.virtnet: 1 2034 4.43 MB/sec warmup 6 sec latency 19.409 ms oleg120-client.virtnet: 1 2425 4.01 MB/sec warmup 7 sec latency 19.614 ms oleg120-client.virtnet: 1 2987 4.19 MB/sec warmup 8 sec latency 17.066 ms oleg120-client.virtnet: 1 3624 4.29 MB/sec warmup 9 sec latency 25.208 ms oleg120-client.virtnet: 1 3915 4.01 MB/sec warmup 10 sec latency 15.258 ms oleg120-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 288.385 ms oleg120-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 1288.566 ms oleg120-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 2288.740 ms oleg120-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 3288.896 ms oleg120-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 4289.089 ms oleg120-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 5289.257 ms oleg120-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 6289.427 ms oleg120-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 7289.589 ms oleg120-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 8289.751 ms oleg120-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 9289.921 ms oleg120-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 10290.105 ms oleg120-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 11290.273 ms oleg120-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 12290.437 ms oleg120-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 13290.636 ms oleg120-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 14290.906 ms oleg120-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 15291.096 ms oleg120-client.virtnet: 1 4391 1.50 MB/sec warmup 27 sec latency 15333.710 ms oleg120-client.virtnet: 1 4714 1.50 MB/sec warmup 28 sec latency 17.872 ms oleg120-client.virtnet: 1 5075 1.56 MB/sec warmup 29 sec latency 17.702 ms oleg120-client.virtnet: 1 5352 1.51 MB/sec warmup 30 sec latency 19.578 ms oleg120-client.virtnet: 1 5710 1.48 MB/sec warmup 31 sec latency 13.676 ms oleg120-client.virtnet: 1 6042 1.49 MB/sec warmup 32 sec latency 16.434 ms oleg120-client.virtnet: 1 6539 1.59 MB/sec warmup 33 sec latency 183.369 mstest_70b fail mds2 2 times Failing mds2 on oleg120-server Stopping /mnt/lustre-mds2 (opts:) on oleg120-server 19:49:25 (1723247365) shut down Failover mds2 to oleg120-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 19:49:41 (1723247381) targets are mounted 19:49:41 (1723247381) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11916 3592764 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37660 3568640 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49576 7161404 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg120-server Stopping /mnt/lustre-mds1 (opts:) on oleg120-server 19:49:54 (1723247394) shut down Failover mds1 to oleg120-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-client.virtnet: 1 7174 1.70 MB/sec warmup 34 sec latency 17.461 ms oleg120-client.virtnet: 1 7451 1.69 MB/sec warmup 35 sec latency 18.054 ms oleg120-client.virtnet: 1 7512 1.64 MB/sec warmup 36 sec latency 745.277 ms oleg120-client.virtnet: 1 7512 1.60 MB/sec warmup 37 sec latency 1745.436 ms oleg120-client.virtnet: 1 7512 1.56 MB/sec warmup 38 sec latency 2745.588 ms oleg120-client.virtnet: 1 7512 1.52 MB/sec warmup 39 sec latency 3745.731 ms oleg120-client.virtnet: 1 7512 1.48 MB/sec warmup 40 sec latency 4745.876 ms oleg120-client.virtnet: 1 7512 1.44 MB/sec warmup 41 sec latency 5746.025 ms oleg120-client.virtnet: 1 7512 1.41 MB/sec warmup 42 sec latency 6746.177 ms oleg120-client.virtnet: 1 7512 1.38 MB/sec warmup 43 sec latency 7746.322 ms oleg120-client.virtnet: 1 7512 1.34 MB/sec warmup 44 sec latency 8746.482 ms oleg120-client.virtnet: 1 7512 1.32 MB/sec warmup 45 sec latency 9746.689 ms oleg120-client.virtnet: 1 7512 1.29 MB/sec warmup 46 sec latency 10746.896 ms oleg120-client.virtnet: 1 7512 1.26 MB/sec warmup 47 sec latency 11747.063 ms oleg120-client.virtnet: 1 7512 1.23 MB/sec warmup 48 sec latency 12747.243 ms oleg120-client.virtnet: 1 7512 1.21 MB/sec warmup 49 sec latency 13747.403 ms oleg120-client.virtnet: 1 7512 1.18 MB/sec warmup 50 sec latency 14747.541 ms oleg120-client.virtnet: 1 7512 1.16 MB/sec warmup 51 sec latency 15747.693 ms oleg120-client.virtnet: 1 7512 1.14 MB/sec warmup 52 sec latency 16747.941 ms oleg120-client.virtnet: 1 7512 1.12 MB/sec warmup 53 sec latency 17748.120 ms oleg120-client.virtnet: 1 7512 1.10 MB/sec warmup 54 sec latency 18748.301 ms oleg120-client.virtnet: 1 7512 1.08 MB/sec warmup 55 sec latency 19748.443 ms oleg120-client.virtnet: 1 7750 1.06 MB/sec warmup 56 sec latency 19842.087 ms oleg120-client.virtnet: 1 8024 1.05 MB/sec warmup 57 sec latency 17.579 ms oleg120-client.virtnet: 1 8363 1.06 MB/sec warmup 58 sec latency 17.461 ms oleg120-client.virtnet: 1 8721 1.09 MB/sec warmup 59 sec latency 15.104 ms oleg120-client.virtnet: 1 9399 1.31 MB/sec execute 1 sec latency 19.988 ms oleg120-client.virtnet: 1 9724 1.31 MB/sec execute 2 sec latency 17.298 ms oleg120-client.virtnet: 1 10440 2.98 MB/sec execute 3 sec latency 17.878 ms oleg120-client.virtnet: 1 10828 3.19 MB/sec execute 4 sec latency 12.661 ms oleg120-client.virtnet: 1 11083 2.72 MB/sec execute 5 sec latency 16.120 ms oleg120-client.virtnet: 1 11378 2.33 MB/sec execute 6 sec latency 20.333 ms oleg120-client.virtnet: 1 11639 2.08 MB/sec execute 7 sec latency 17.688 ms oleg120-client.virtnet: 1 12043 2.33 MB/sec execute 8 sec latency 17.962 ms oleg120-client.virtnet: 1 12310 2.10 MB/sec execute 9 sec latency 15.913 ms oleg120-client.virtnet: 1 12612 1.91 MB/sec execute 10 sec latency 19.363 ms oleg120-client.virtnet: 1 12987 1.86 MB/sec execute 11 sec latency 11.625 ms oleg120-client.virtnet: 1 13371 2.05 MB/sec execute 12 sec latency 17.932 ms oleg120-client.virtnet: 1 14008 2.16 MB/sec execute 13 sec latency 18.330 ms oleg120-client.virtnet: 1 14407 2.32 MB/sec execute 14 sec latency 16.015 ms oleg120-client.virtnet: 1 14638 2.18 MB/sec execute 15 sec latency 16.114 ms oleg120-client.virtnet: 1 14941 2.07 MB/sec execute 16 sec latency 18.543 ms oleg120-client.virtnet: 1 15205 1.99 MB/sec execute 17 sec latency 18.821 ms oleg120-client.virtnet: 1 15620 2.10 MB/sec execute 18 sec latency 17.654 ms oleg120-client.virtnet: 1 15892 2.00 MB/sec execute oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 19:50:10 (1723247410) targets are mounted 19:50:10 (1723247410) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11916 3594216 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37572 3568936 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163152 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg120-server Stopping /mnt/lustre-mds2 (opts:) on oleg120-server 19:50:23 (1723247423) shut down Failover mds2 to oleg120-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 19:50:38 (1723247438) targets are mounted 19:50:38 (1723247438) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12040 3593880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37576 3566696 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160576 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg120-server Stopping /mnt/lustre-mds1 (opts:) on oleg120-server 19:50:51 (1723247451) shut down 19 sec latency 20.252 ms oleg120-client.virtnet: 1 16221 1.92 MB/sec execute 20 sec latency 12.211 ms oleg120-client.virtnet: 1 16592 1.89 MB/sec execute 21 sec latency 17.342 ms oleg120-client.virtnet: 1 17095 2.03 MB/sec execute 22 sec latency 18.218 ms oleg120-client.virtnet: 1 17715 2.18 MB/sec execute 23 sec latency 18.326 ms oleg120-client.virtnet: 1 18012 2.15 MB/sec execute 24 sec latency 15.275 ms oleg120-client.virtnet: 1 18248 2.07 MB/sec execute 25 sec latency 18.270 ms oleg120-client.virtnet: 1 18561 2.01 MB/sec execute 26 sec latency 22.427 ms oleg120-client.virtnet: 1 18870 2.00 MB/sec execute 27 sec latency 18.240 ms oleg120-client.virtnet: 1 19219 2.03 MB/sec execute 28 sec latency 18.823 ms oleg120-client.virtnet: 1 19501 1.97 MB/sec execute 29 sec latency 19.438 ms oleg120-client.virtnet: 1 19831 1.92 MB/sec execute 30 sec latency 11.987 ms oleg120-client.virtnet: 1 20181 1.90 MB/sec execute 31 sec latency 17.845 ms oleg120-client.virtnet: 1 20690 2.00 MB/sec execute 32 sec latency 17.457 ms oleg120-client.virtnet: 1 21141 2.01 MB/sec execute 33 sec latency 262.783 ms oleg120-client.virtnet: 1 21141 1.95 MB/sec execute 34 sec latency 1262.963 ms oleg120-client.virtnet: 1 21141 1.90 MB/sec execute 35 sec latency 2263.147 ms oleg120-client.virtnet: 1 21141 1.84 MB/sec execute 36 sec latency 3263.355 ms oleg120-client.virtnet: 1 21141 1.79 MB/sec execute 37 sec latency 4263.516 ms oleg120-client.virtnet: 1 21141 1.75 MB/sec execute 38 sec latency 5263.686 ms oleg120-client.virtnet: 1 21141 1.70 MB/sec execute 39 sec latency 6263.863 ms oleg120-client.virtnet: 1 21141 1.66 MB/sec execute 40 sec latency 7264.012 ms oleg120-client.virtnet: 1 21141 1.62 MB/sec execute 41 sec latency 8264.180 ms oleg120-client.virtnet: 1 21141 1.58 MB/sec execute 42 sec latency 9264.351 ms oleg120-client.virtnet: 1 21141 1.54 MB/sec execute 43 sec latency 10264.496 ms oleg120-client.virtnet: 1 21141 1.51 MB/sec execute 44 sec latency 11264.651 ms oleg120-client.virtnet: 1 21141 1.48 MB/sec execute 45 sec latency 12264.820 ms oleg120-client.virtnet: 1 21141 1.44 MB/sec execute 46 sec latency 13265.022 ms oleg120-client.virtnet: 1 21141 1.41 MB/sec execute 47 sec latency 14265.193 ms oleg120-client.virtnet: 1 21141 1.38 MB/sec execute 48 sec latency 15265.389 ms oleg120-client.virtnet: 1 21141 1.36 MB/sec execute 49 sec latency 16265.557 ms oleg120-client.virtnet: 1 21141 1.33 MB/sec execute 50 sec latency 17265.739 ms oleg120-client.virtnet: 1 21141 1.30 MB/sec execute 51 sec latency 18265.908 ms oleg120-client.virtnet: 1 21141 1.28 MB/sec execute 52 sec latency 19266.111 ms oleg120-client.virtnet: 1 21465 1.32 MB/sec execute 53 sec latency 19473.075 ms oleg120-client.virtnet: 1 21725 1.32 MB/sec execute 54 sec latency 18.267 ms oleg120-client.virtnet: 1 21987 1.29 MB/sec execute 55 sec latency 15.614 ms oleg120-client.virtnet: 1 22249 1.28 MB/sec execute 56 sec latency 18.153 ms oleg120-client.virtnet: 1 22560 1.28 MB/sec execute 57 sec latency 18.136 ms oleg120-client.virtnet: 1 22916 1.31 MB/sec execute 58 sec latency 17.302 ms oleg120-client.virtnet: 1 23191 1.30 MB/sec execute 59 sec latency 19.458 ms oleg120-client.virtnet: 1 23536 1.28 MB/sec execute 60 sec latency 11.373 ms oleg120-client.virtnet: 1 23908 1.30 MB/sec execute 61 sec latency 16.813 ms oleg120-client.virtnet: 1 24569 1.37 MB/sec execute 62 sec latency 17.946 ms oleg120-client.virtnet: 1 25013 1.41 MB/sec execute 63 sec latency 13.120 ms oleg120-clientFailover mds1 to oleg120-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 19:51:07 (1723247467) targets are mounted 19:51:07 (1723247467) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12088 3593548 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37548 3568884 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7162432 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg120-server Stopping /mnt/lustre-mds2 (opts:) on oleg120-server 19:51:20 (1723247480) shut down Failover mds2 to oleg120-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 19:51:35 (1723247495) targets are mounted 19:51:35 (1723247495) facet_failover done .virtnet: 1 25278 1.41 MB/sec execute 64 sec latency 15.655 ms oleg120-client.virtnet: 1 25575 1.39 MB/sec execute 65 sec latency 18.947 ms oleg120-client.virtnet: 1 25838 1.38 MB/sec execute 66 sec latency 17.942 ms oleg120-client.virtnet: 1 26144 1.38 MB/sec execute 67 sec latency 18.178 ms oleg120-client.virtnet: 1 26497 1.40 MB/sec execute 68 sec latency 15.720 ms oleg120-client.virtnet: 1 26762 1.39 MB/sec execute 69 sec latency 19.045 ms oleg120-client.virtnet: 1 27117 1.37 MB/sec execute 70 sec latency 12.062 ms oleg120-client.virtnet: 1 27463 1.38 MB/sec execute 71 sec latency 18.460 ms oleg120-client.virtnet: 1 28103 1.44 MB/sec execute 72 sec latency 18.690 ms oleg120-client.virtnet: 1 28555 1.48 MB/sec execute 73 sec latency 12.864 ms oleg120-client.virtnet: 1 28818 1.48 MB/sec execute 74 sec latency 15.779 ms oleg120-client.virtnet: 1 29123 1.46 MB/sec execute 75 sec latency 16.822 ms oleg120-client.virtnet: 1 29395 1.45 MB/sec execute 76 sec latency 21.320 ms oleg120-client.virtnet: 1 29742 1.46 MB/sec execute 77 sec latency 18.157 ms oleg120-client.virtnet: 1 30074 1.47 MB/sec execute 78 sec latency 15.723 ms oleg120-client.virtnet: 1 30373 1.45 MB/sec execute 79 sec latency 20.207 ms oleg120-client.virtnet: 1 30805 1.45 MB/sec execute 80 sec latency 11.597 ms oleg120-client.virtnet: 1 31317 1.50 MB/sec execute 81 sec latency 26.974 ms oleg120-client.virtnet: 1 31963 1.55 MB/sec execute 82 sec latency 17.656 ms oleg120-client.virtnet: 1 32250 1.54 MB/sec execute 83 sec latency 16.497 ms oleg120-client.virtnet: 1 32475 1.53 MB/sec execute 84 sec latency 16.097 ms oleg120-client.virtnet: 1 32787 1.52 MB/sec execute 85 sec latency 20.515 ms oleg120-client.virtnet: 1 33106 1.52 MB/sec execute 86 sec latency 18.210 ms oleg120-client.virtnet: 1 33467 1.53 MB/sec execute 87 sec latency 17.497 ms oleg120-client.virtnet: 1 33741 1.52 MB/sec execute 88 sec latency 19.866 ms oleg120-client.virtnet: 1 34050 1.51 MB/sec execute 89 sec latency 13.033 ms oleg120-client.virtnet: 1 34263 1.50 MB/sec execute 90 sec latency 456.206 ms oleg120-client.virtnet: 1 34263 1.49 MB/sec execute 91 sec latency 1456.401 ms oleg120-client.virtnet: 1 34263 1.47 MB/sec execute 92 sec latency 2456.630 ms oleg120-client.virtnet: 1 34263 1.45 MB/sec execute 93 sec latency 3456.793 ms oleg120-client.virtnet: 1 34263 1.44 MB/sec execute 94 sec latency 4456.973 ms oleg120-client.virtnet: 1 34263 1.42 MB/sec execute 95 sec latency 5457.136 ms oleg120-client.virtnet: 1 34263 1.41 MB/sec execute 96 sec latency 6457.289 ms oleg120-client.virtnet: 1 34263 1.39 MB/sec execute 97 sec latency 7457.445 ms oleg120-client.virtnet: 1 34263 1.38 MB/sec execute 98 sec latency 8457.601 ms oleg120-client.virtnet: 1 34263 1.37 MB/sec execute 99 sec latency 9457.757 ms oleg120-client.virtnet: 1 34263 1.35 MB/sec execute 100 sec latency 10457.929 ms oleg120-client.virtnet: 1 34263 1.34 MB/sec execute 101 sec latency 11458.112 ms oleg120-client.virtnet: 1 34263 1.33 MB/sec execute 102 sec latency 12458.298 ms oleg120-client.virtnet: 1 34263 1.31 MB/sec execute 103 sec latency 13458.505 ms oleg120-client.virtnet: 1 34263 1.30 MB/sec execute 104 sec latency 14458.691 ms oleg120-client.virtnet: 1 34263 1.29 MB/sec execute 105 sec latency 15458.870 ms oleg120-client.virtnet: 1 34263 1.28 MB/sec execute 106 sec latency 16459.067 ms oleg120-client.virtnet: 1 34263 1.26 MB/sec execute 107 sec latency 17459.272 ms oleg120-client.virtnet: 1 34263 1.25 MB/sec executeoleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12088 3592020 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37548 3569028 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7161048 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg120-server Stopping /mnt/lustre-mds1 (opts:) on oleg120-server 19:51:48 (1723247508) shut down Failover mds1 to oleg120-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 19:52:03 (1723247523) targets are mounted 19:52:03 (1723247523) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12084 3591640 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37412 3568684 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7160324 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg120-server Stopping /mnt/lustre-mds2 (opts:) on oleg120-server 19:52:16 (1723247536) shut down 108 sec latency 18459.465 ms oleg120-client.virtnet: 1 34263 1.24 MB/sec execute 109 sec latency 19459.644 ms oleg120-client.virtnet: 1 34568 1.24 MB/sec execute 110 sec latency 19545.810 ms oleg120-client.virtnet: 1 35239 1.29 MB/sec execute 111 sec latency 18.118 ms oleg120-client.virtnet: 1 35660 1.31 MB/sec execute 112 sec latency 13.385 ms oleg120-client.virtnet: 1 35914 1.31 MB/sec execute 113 sec latency 16.768 ms oleg120-client.virtnet: 1 36205 1.30 MB/sec execute 114 sec latency 15.615 ms oleg120-client.virtnet: 1 36470 1.29 MB/sec execute 115 sec latency 17.668 ms oleg120-client.virtnet: 1 36772 1.29 MB/sec execute 116 sec latency 18.298 ms oleg120-client.virtnet: 1 37119 1.31 MB/sec execute 117 sec latency 15.705 ms oleg120-client.virtnet: 1 37398 1.30 MB/sec execute 118 sec latency 18.664 ms oleg120-client.virtnet: 1 37760 1.29 MB/sec execute 119 sec latency 12.034 ms oleg120-client.virtnet: 1 38101 1.30 MB/sec execute 120 sec latency 18.233 ms oleg120-client.virtnet: 1 38741 1.33 MB/sec execute 121 sec latency 18.091 ms oleg120-client.virtnet: 1 39189 1.35 MB/sec execute 122 sec latency 13.402 ms oleg120-client.virtnet: 1 39446 1.35 MB/sec execute 123 sec latency 16.024 ms oleg120-client.virtnet: 1 39737 1.35 MB/sec execute 124 sec latency 15.691 ms oleg120-client.virtnet: 1 39992 1.34 MB/sec execute 125 sec latency 19.825 ms oleg120-client.virtnet: 1 40306 1.34 MB/sec execute 126 sec latency 17.852 ms oleg120-client.virtnet: 1 40694 1.35 MB/sec execute 127 sec latency 18.081 ms oleg120-client.virtnet: 1 41006 1.34 MB/sec execute 128 sec latency 19.797 ms oleg120-client.virtnet: 1 41393 1.34 MB/sec execute 129 sec latency 12.006 ms oleg120-client.virtnet: 1 41825 1.37 MB/sec execute 130 sec latency 18.003 ms oleg120-client.virtnet: 1 42484 1.39 MB/sec execute 131 sec latency 17.607 ms oleg120-client.virtnet: 1 42857 1.40 MB/sec execute 132 sec latency 15.516 ms oleg120-client.virtnet: 1 43097 1.40 MB/sec execute 133 sec latency 15.795 ms oleg120-client.virtnet: 1 43417 1.39 MB/sec execute 134 sec latency 18.076 ms oleg120-client.virtnet: 1 43734 1.39 MB/sec execute 135 sec latency 17.538 ms oleg120-client.virtnet: 1 44097 1.40 MB/sec execute 136 sec latency 17.954 ms oleg120-client.virtnet: 1 44384 1.39 MB/sec execute 137 sec latency 19.392 ms oleg120-client.virtnet: 1 44725 1.39 MB/sec execute 138 sec latency 17.317 ms oleg120-client.virtnet: 1 45054 1.39 MB/sec execute 139 sec latency 17.065 ms oleg120-client.virtnet: 1 45607 1.42 MB/sec execute 140 sec latency 17.759 ms oleg120-client.virtnet: 1 46228 1.44 MB/sec execute 141 sec latency 20.175 ms oleg120-client.virtnet: 1 46503 1.44 MB/sec execute 142 sec latency 16.565 ms oleg120-client.virtnet: 1 46799 1.43 MB/sec execute 143 sec latency 15.649 ms oleg120-client.virtnet: 1 46948 1.43 MB/sec execute 144 sec latency 413.972 ms oleg120-client.virtnet: 1 47233 1.43 MB/sec execute 145 sec latency 18.466 ms oleg120-client.virtnet: 1 47607 1.44 MB/sec execute 146 sec latency 18.157 ms oleg120-client.virtnet: 1 47622 1.43 MB/sec execute 147 sec latency 933.803 ms oleg120-client.virtnet: 1 47622 1.42 MB/sec execute 148 sec latency 1933.995 ms oleg120-client.virtnet: 1 47622 1.41 MB/sec execute 149 sec latency 2934.199 ms oleg120-client.virtnet: 1 47622 1.40 MB/sec execute 150 sec latency 3934.386 ms oleg120-client.virtnet: 1 47622 1.39 MB/sec execute 151 sec latency 4934.599 ms oleg120-client.virtnet: 1 47622 1.38 MB/sec execute 152 sec latency 5934.803 ms oleg120-client.virtnet: 1 47622 Failover mds2 to oleg120-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 19:52:32 (1723247552) targets are mounted 19:52:32 (1723247552) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12088 3593832 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37412 3569104 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162936 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg120-server Stopping /mnt/lustre-mds1 (opts:) on oleg120-server 19:52:45 (1723247565) shut down Failover mds1 to oleg120-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 19:53:01 (1723247581) targets are mounted 19:53:01 (1723247581) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1.37 MB/sec execute 153 sec latency 6934.997 ms oleg120-client.virtnet: 1 47622 1.36 MB/sec execute 154 sec latency 7935.146 ms oleg120-client.virtnet: 1 47622 1.35 MB/sec execute 155 sec latency 8935.340 ms oleg120-client.virtnet: 1 47622 1.35 MB/sec execute 156 sec latency 9935.560 ms oleg120-client.virtnet: 1 47622 1.34 MB/sec execute 157 sec latency 10935.789 ms oleg120-client.virtnet: 1 47622 1.33 MB/sec execute 158 sec latency 11936.010 ms oleg120-client.virtnet: 1 47622 1.32 MB/sec execute 159 sec latency 12936.215 ms oleg120-client.virtnet: 1 47622 1.31 MB/sec execute 160 sec latency 13936.446 ms oleg120-client.virtnet: 1 47622 1.30 MB/sec execute 161 sec latency 14936.654 ms oleg120-client.virtnet: 1 47622 1.30 MB/sec execute 162 sec latency 15936.857 ms oleg120-client.virtnet: 1 47622 1.29 MB/sec execute 163 sec latency 16937.105 ms oleg120-client.virtnet: 1 47622 1.28 MB/sec execute 164 sec latency 17937.387 ms oleg120-client.virtnet: 1 47622 1.27 MB/sec execute 165 sec latency 18937.643 ms oleg120-client.virtnet: 1 47646 1.26 MB/sec execute 166 sec latency 19868.985 ms oleg120-client.virtnet: 1 47922 1.26 MB/sec execute 167 sec latency 20.044 ms oleg120-client.virtnet: 1 48230 1.25 MB/sec execute 168 sec latency 15.214 ms oleg120-client.virtnet: 1 48564 1.25 MB/sec execute 169 sec latency 17.551 ms oleg120-client.virtnet: 1 49062 1.28 MB/sec execute 170 sec latency 18.517 ms oleg120-client.virtnet: 1 49671 1.30 MB/sec execute 171 sec latency 18.471 ms oleg120-client.virtnet: 1 49956 1.30 MB/sec execute 172 sec latency 16.777 ms oleg120-client.virtnet: 1 50179 1.29 MB/sec execute 173 sec latency 16.651 ms oleg120-client.virtnet: 1 50478 1.29 MB/sec execute 174 sec latency 20.365 ms oleg120-client.virtnet: 1 50759 1.29 MB/sec execute 175 sec latency 18.120 ms oleg120-client.virtnet: 1 51135 1.30 MB/sec execute 176 sec latency 17.754 ms oleg120-client.virtnet: 1 51392 1.30 MB/sec execute 177 sec latency 19.717 ms oleg120-client.virtnet: 1 51697 1.29 MB/sec execute 178 sec latency 12.423 ms oleg120-client.virtnet: 1 52048 1.29 MB/sec execute 179 sec latency 14.541 ms oleg120-client.virtnet: 1 52458 1.31 MB/sec execute 180 sec latency 18.240 ms oleg120-client.virtnet: 1 53054 1.32 MB/sec execute 181 sec latency 18.510 ms oleg120-client.virtnet: 1 53455 1.33 MB/sec execute 182 sec latency 14.121 ms oleg120-client.virtnet: 1 53688 1.33 MB/sec execute 183 sec latency 15.665 ms oleg120-client.virtnet: 1 53995 1.32 MB/sec execute 184 sec latency 16.812 ms oleg120-client.virtnet: 1 54296 1.32 MB/sec execute 185 sec latency 18.351 ms oleg120-client.virtnet: 1 54685 1.33 MB/sec execute 186 sec latency 18.113 ms oleg120-client.virtnet: 1 54961 1.33 MB/sec execute 187 sec latency 18.122 ms oleg120-client.virtnet: 1 55281 1.32 MB/sec execute 188 sec latency 11.481 ms oleg120-client.virtnet: 1 55654 1.32 MB/sec execute 189 sec latency 17.525 ms oleg120-client.virtnet: 1 56210 1.35 MB/sec execute 190 sec latency 22.774 ms oleg120-client.virtnet: 1 56808 1.36 MB/sec execute 191 sec latency 17.030 ms oleg120-client.virtnet: 1 57102 1.36 MB/sec execute 192 sec latency 16.532 ms oleg120-client.virtnet: 1 57343 1.36 MB/sec execute 193 sec latency 16.442 ms oleg120-client.virtnet: 1 57654 1.35 MB/sec execute 194 sec latency 22.538 ms oleg120-client.virtnet: 1 57950 1.36 MB/sec execute 195 sec latency 23.783 ms oleg120-client.virtnet: 1 58325 1.36 MB/sec execute 196 sec latency 17.880 ms oleg120-client.virtnet: 1 58583 1.36 MB/sec execute 197 sec latency 20.184 ms oleg12UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12084 3593720 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37412 3569020 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7162740 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg120-server Stopping /mnt/lustre-mds2 (opts:) on oleg120-server 19:53:13 (1723247593) shut down Failover mds2 to oleg120-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 19:53:29 (1723247609) targets are mounted 19:53:29 (1723247609) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11916 3594252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37732 3569104 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7163356 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg120-server Stopping /mnt/lustre-mds1 (opts:) on oleg120-server 19:53:42 (1723247622) shut down 0-client.virtnet: 1 58924 1.35 MB/sec execute 198 sec latency 12.009 ms oleg120-client.virtnet: 1 59242 1.35 MB/sec execute 199 sec latency 17.556 ms oleg120-client.virtnet: 1 59776 1.37 MB/sec execute 200 sec latency 17.656 ms oleg120-client.virtnet: 1 60127 1.38 MB/sec execute 201 sec latency 591.402 ms oleg120-client.virtnet: 1 60505 1.39 MB/sec execute 202 sec latency 13.156 ms oleg120-client.virtnet: 1 60767 1.39 MB/sec execute 203 sec latency 15.663 ms oleg120-client.virtnet: 1 60835 1.38 MB/sec execute 204 sec latency 704.869 ms oleg120-client.virtnet: 1 60835 1.37 MB/sec execute 205 sec latency 1705.047 ms oleg120-client.virtnet: 1 60835 1.37 MB/sec execute 206 sec latency 2705.229 ms oleg120-client.virtnet: 1 60835 1.36 MB/sec execute 207 sec latency 3705.397 ms oleg120-client.virtnet: 1 60835 1.35 MB/sec execute 208 sec latency 4705.622 ms oleg120-client.virtnet: 1 60835 1.35 MB/sec execute 209 sec latency 5705.809 ms oleg120-client.virtnet: 1 60835 1.34 MB/sec execute 210 sec latency 6706.075 ms oleg120-client.virtnet: 1 60835 1.33 MB/sec execute 211 sec latency 7706.273 ms oleg120-client.virtnet: 1 60835 1.33 MB/sec execute 212 sec latency 8706.472 ms oleg120-client.virtnet: 1 60835 1.32 MB/sec execute 213 sec latency 9706.700 ms oleg120-client.virtnet: 1 60835 1.31 MB/sec execute 214 sec latency 10706.954 ms oleg120-client.virtnet: 1 60835 1.31 MB/sec execute 215 sec latency 11707.162 ms oleg120-client.virtnet: 1 60835 1.30 MB/sec execute 216 sec latency 12707.337 ms oleg120-client.virtnet: 1 60835 1.30 MB/sec execute 217 sec latency 13707.545 ms oleg120-client.virtnet: 1 60835 1.29 MB/sec execute 218 sec latency 14707.718 ms oleg120-client.virtnet: 1 60835 1.28 MB/sec execute 219 sec latency 15707.915 ms oleg120-client.virtnet: 1 60835 1.28 MB/sec execute 220 sec latency 16708.092 ms oleg120-client.virtnet: 1 60835 1.27 MB/sec execute 221 sec latency 17708.282 ms oleg120-client.virtnet: 1 60835 1.27 MB/sec execute 222 sec latency 18708.446 ms oleg120-client.virtnet: 1 60838 1.26 MB/sec execute 223 sec latency 19669.654 ms oleg120-client.virtnet: 1 61121 1.26 MB/sec execute 224 sec latency 17.991 ms oleg120-client.virtnet: 1 61414 1.26 MB/sec execute 225 sec latency 19.139 ms oleg120-client.virtnet: 1 61783 1.27 MB/sec execute 226 sec latency 18.204 ms oleg120-client.virtnet: 1 62046 1.26 MB/sec execute 227 sec latency 20.293 ms oleg120-client.virtnet: 1 62352 1.26 MB/sec execute 228 sec latency 12.083 ms oleg120-client.virtnet: 1 62725 1.26 MB/sec execute 229 sec latency 16.306 ms oleg120-client.virtnet: 1 63227 1.27 MB/sec execute 230 sec latency 17.545 ms oleg120-client.virtnet: 1 63836 1.29 MB/sec execute 231 sec latency 18.091 ms oleg120-client.virtnet: 1 64142 1.29 MB/sec execute 232 sec latency 18.763 ms oleg120-client.virtnet: 1 64373 1.29 MB/sec execute 233 sec latency 16.383 ms oleg120-client.virtnet: 1 64680 1.29 MB/sec execute 234 sec latency 17.676 ms oleg120-client.virtnet: 1 64990 1.29 MB/sec execute 235 sec latency 18.903 ms oleg120-client.virtnet: 1 65344 1.29 MB/sec execute 236 sec latency 18.057 ms oleg120-client.virtnet: 1 65607 1.29 MB/sec execute 237 sec latency 20.161 ms oleg120-client.virtnet: 1 65905 1.29 MB/sec execute 238 sec latency 12.062 ms oleg120-client.virtnet: 1 66268 1.29 MB/sec execute 239 sec latency 17.460 ms oleg120-client.virtnet: 1 66769 1.30 MB/sec execute 240 sec latency 18.070 ms oleg120-client.virtnet: 1 67374 1.32 MB/sec execute 241 sec latency 18.018 ms oleg120-client.virtnet: 1 67674 1.32 Failover mds1 to oleg120-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 19:53:57 (1723247637) targets are mounted 19:53:57 (1723247637) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec MB/sec execute 242 sec latency 15.929 ms oleg120-client.virtnet: 1 67905 1.32 MB/sec execute 243 sec latency 16.150 ms oleg120-client.virtnet: 1 68221 1.31 MB/sec execute 244 sec latency 17.111 ms oleg120-client.virtnet: 1 68528 1.31 MB/sec execute 245 sec latency 16.328 ms oleg120-client.virtnet: 1 68900 1.32 MB/sec execute 246 sec latency 17.967 ms oleg120-client.virtnet: 1 69201 1.32 MB/sec execute 247 sec latency 19.583 ms oleg120-client.virtnet: 1 69526 1.31 MB/sec execute 248 sec latency 12.621 ms oleg120-client.virtnet: 1 69861 1.31 MB/sec execute 249 sec latency 17.756 ms oleg120-client.virtnet: 1 70395 1.33 MB/sec execute 250 sec latency 18.487 ms oleg120-client.virtnet: 1 70981 1.34 MB/sec execute 251 sec latency 18.310 ms oleg120-client.virtnet: 1 71267 1.34 MB/sec execute 252 sec latency 17.079 ms oleg120-client.virtnet: 1 71520 1.34 MB/sec execute 253 sec latency 15.558 ms oleg120-client.virtnet: 1 71840 1.34 MB/sec execute 254 sec latency 21.030 ms oleg120-client.virtnet: 1 72129 1.34 MB/sec execute 255 sec latency 18.022 ms oleg120-client.virtnet: 1 72479 1.34 MB/sec execute 256 sec latency 17.581 ms oleg120-client.virtnet: 1 72755 1.34 MB/sec execute 257 sec latency 19.946 ms oleg120-client.virtnet: 1 73061 1.33 MB/sec execute 258 sec latency 12.705 ms oleg120-client.virtnet: 1 73410 1.34 MB/sec execute 259 sec latency 17.482 ms oleg120-client.virtnet: 1 73922 1.35 MB/sec execute 260 sec latency 18.904 ms oleg120-client.virtnet: 1 74559 1.36 MB/sec execute 261 sec latency 18.227 ms oleg120-client.virtnet: 1 74837 1.36 MB/sec execute 262 sec latency 15.939 ms oleg120-client.virtnet: 1 75079 1.36 MB/sec execute 263 sec latency 16.009 ms oleg120-client.virtnet: 1 75402 1.36 MB/sec execute 264 sec latency 19.590 ms oleg120-client.virtnet: 1 75696 1.36 MB/sec execute 265 sec latency 17.592 ms oleg120-client.virtnet: 1 76049 1.36 MB/sec execute 266 sec latency 17.810 ms oleg120-client.virtnet: 1 76326 1.36 MB/sec execute 267 sec latency 19.451 ms oleg120-client.virtnet: 1 76638 1.36 MB/sec execute 268 sec latency 11.828 ms oleg120-client.virtnet: 1 76979 1.36 MB/sec execute 269 sec latency 17.792 ms oleg120-client.virtnet: 1 77518 1.37 MB/sec execute 270 sec latency 17.653 ms oleg120-client.virtnet: 1 78114 1.38 MB/sec execute 271 sec latency 18.027 ms oleg120-client.virtnet: 1 78393 1.38 MB/sec execute 272 sec latency 15.942 ms oleg120-client.virtnet: 1 78635 1.38 MB/sec execute 273 sec latency 15.846 ms oleg120-client.virtnet: 1 78954 1.38 MB/sec execute 274 sec latency 18.991 ms oleg120-client.virtnet: 1 79243 1.38 MB/sec execute 275 sec latency 17.610 ms oleg120-client.virtnet: 1 79597 1.38 MB/sec execute 276 sec latency 17.287 ms oleg120-client.virtnet: 1 79875 1.38 MB/sec execute 277 sec latency 19.902 ms oleg120-client.virtnet: 1 80206 1.38 MB/sec execute 278 sec latency 11.733 ms oleg120-client.virtnet: 1 80529 1.38 MB/sec execute 279 sec latency 17.779 ms oleg120-client.virtnet: 1 81062 1.39 MB/sec execute 280 sec latency 18.205 ms oleg120-client.virtnet: 1 81657 1.40 MB/sec execute 281 sec latency 17.750 ms oleg120-client.virtnet: 1 81934 1.40 MB/sec execute 282 sec latency 15.952 ms oleg120-client.virtnet: 1 82165 1.40 MB/sec execute 283 sec latency 16.282 ms oleg120-client.virtnet: 1 82471 1.40 MB/sec execute 284 sec latency 18.197 ms oleg120-client.virtnet: 1 82777 1.40 MB/sec execute 285 sec latency 18.252 ms oleg120-client.virtnet: 1 83131 1.40 MB/sec execute 286 sec latency 17.649 ms oleg120-client.virtnet: 1 83403 1.40 MB/sec execute 287 sec latency 20.124 ms oleg120-client.virtnet: 1 83704 1.39 MB/sec execute 288 sec latency 11.872 ms oleg120-client.virtnet: 1 84053 1.40 MB/sec execute 289 sec latency 18.079 ms oleg120-client.virtnet: 1 84560 1.41 MB/sec execute 290 sec latency 18.306 ms oleg120-client.virtnet: 1 85169 1.42 MB/sec execute 291 sec latency 18.195 ms oleg120-client.virtnet: 1 85458 1.42 MB/sec execute 292 sec latency 16.031 ms oleg120-client.virtnet: 1 85687 1.42 MB/sec execute 293 sec latency 16.248 ms oleg120-client.virtnet: 1 85994 1.41 MB/sec execute 294 sec latency 20.517 ms oleg120-client.virtnet: 1 86310 1.41 MB/sec execute 295 sec latency 17.725 ms oleg120-client.virtnet: 1 86669 1.42 MB/sec execute 296 sec latency 17.982 ms oleg120-client.virtnet: 1 86948 1.42 MB/sec execute 297 sec latency 18.938 ms oleg120-client.virtnet: 1 87262 1.41 MB/sec execute 298 sec latency 11.284 ms oleg120-client.virtnet: 1 87614 1.41 MB/sec execute 299 sec latency 17.242 ms oleg120-client.virtnet: 1 cleanup 300 sec oleg120-client.virtnet: 0 cleanup 300 sec oleg120-client.virtnet: oleg120-client.virtnet: Operation Count AvgLat MaxLat oleg120-client.virtnet: ---------------------------------------- oleg120-client.virtnet: NTCreateX 13652 11.870 19868.970 oleg120-client.virtnet: Close 10056 3.807 19473.054 oleg120-client.virtnet: Rename 580 13.974 23.766 oleg120-client.virtnet: Unlink 2749 2.966 7.501 oleg120-client.virtnet: Qpathinfo 12414 4.036 19545.790 oleg120-client.virtnet: Qfileinfo 2176 0.198 2.597 oleg120-client.virtnet: Qfsinfo 2266 0.352 18.625 oleg120-client.virtnet: Sfileinfo 1116 8.446 14.446 oleg120-client.virtnet: Find 4788 0.613 11.918 oleg120-client.virtnet: WriteX 6773 1.352 7.852 oleg120-client.virtnet: ReadX 21526 0.025 2.381 oleg120-client.virtnet: LockX 46 1.834 3.385 oleg120-client.virtnet: UnlockX 46 2.064 2.290 oleg120-client.virtnet: Flush 953 9.600 591.389 oleg120-client.virtnet: oleg120-client.virtnet: Throughput 1.41257 MB/sec 1 clients 1 procs max_latency=19868.985 ms oleg120-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg120-client.virtnet at Fri Aug 9 19:54:50 EDT 2024 with return code 0 oleg120-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg120-client.virtnet oleg120-client.virtnet: /mnt/lustre/d70b.replay-single/oleg120-client.virtnet /mnt/lustre/d70b.replay-single/oleg120-client.virtnet oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg120-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg120-client.virtnet: removed directory: 'clients/client0' oleg120-client.virtnet: removed directory: 'clients' oleg120-client.virtnet: removed 'client.txt' oleg120-client.virtnet: /mnt/lustre/d70b.replay-single/oleg120-client.virtnet oleg120-client.virtnet: dbench successfully finished