== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 00:50:24 (1721883024) Starting client oleg138-client.virtnet: -o user_xattr,flock oleg138-server@tcp:/lustre /mnt/lustre Started clients oleg138-client.virtnet: 192.168.201.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg138-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=11673 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26152 3556148 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1892 3593964 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28044 7150112 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 00:50:31 (1721883031) shut down Failover mds1 to oleg138-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 00:50:46 (1721883046) targets are mounted 00:50:46 (1721883046) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26408 3568844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1808 3594036 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28216 7162880 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg138-server Stopping /mnt/lustre-mds2 (opts:) on oleg138-server oleg138-client.virtnet: looking for dbench program oleg138-client.virtnet: /usr/bin/dbench oleg138-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg138-client.virtnet oleg138-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg138-client.virtnet' oleg138-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg138-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg138-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg138-client.virtnet at Thu Jul 25 00:50:24 EDT 2024 oleg138-client.virtnet: waiting for dbench pid 11697 oleg138-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg138-client.virtnet: oleg138-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg138-client.virtnet: failed to create barrier semaphore oleg138-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg138-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg138-client.virtnet: releasing clients oleg138-client.virtnet: 1 572 19.17 MB/sec warmup 1 sec latency 19.104 ms oleg138-client.virtnet: 1 1156 11.50 MB/sec warmup 2 sec latency 11.403 ms oleg138-client.virtnet: 1 1795 8.77 MB/sec warmup 3 sec latency 15.237 ms oleg138-client.virtnet: 1 2407 7.01 MB/sec warmup 4 sec latency 11.899 ms oleg138-client.virtnet: 1 3572 7.69 MB/sec warmup 5 sec latency 9.427 ms oleg138-client.virtnet: 1 3928 6.68 MB/sec warmup 6 sec latency 15.061 ms oleg138-client.virtnet: 1 4083 5.73 MB/sec warmup 7 sec latency 337.812 ms oleg138-client.virtnet: 1 4083 5.01 MB/sec warmup 8 sec latency 1338.032 ms oleg138-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 2338.262 ms oleg138-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 3338.421 ms oleg138-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 4338.646 ms oleg138-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 5338.840 ms oleg138-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 6338.988 ms oleg138-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 7339.167 ms oleg138-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 8339.410 ms oleg138-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 9339.720 ms oleg138-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 10339.968 ms oleg138-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 11340.159 ms oleg138-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 12340.283 ms oleg138-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 13340.454 ms oleg138-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 14340.684 ms oleg138-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 15340.831 ms oleg138-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 16340.999 ms oleg138-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 17341.192 ms oleg138-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 18341.416 ms oleg138-client.virtnet: 1 4452 1.57 MB/sec warmup 26 sec latency 18422.423 ms oleg138-client.virtnet: 1 4955 1.67 MB/sec warmup 27 sec latency 16.412 ms oleg138-client.virtnet: 1 5518 1.63 MB/sec warmup 28 sec latency 9.970 ms oleg138-client.virtnet: 1 6050 1.65 MB/sec warmup 29 sec latency 13.590 ms oleg138-client.virtnet: 1 6831 1.82 MB/sec warmup 30 sec latency 14.658 ms oleg138-client.virtnet: 1 7361 1.90 MB/sec warmup 31 sec latency 14.371 ms oleg138-client.virtnet: 1 7608 1.85 MB/sec warmup 32 sec latency 15.525 ms oleg138-client.virtnet: 1 7960 1.81 MB/sec warmup 33 sec latency 20.00:50:58 (1721883058) shut down Failover mds2 to oleg138-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0001 00:51:13 (1721883073) targets are mounted 00:51:13 (1721883073) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569064 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12076 3593852 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162916 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 00:51:26 (1721883086) shut down Failover mds1 to oleg138-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 00:51:42 (1721883102) targets are mounted 00:51:42 (1721883102) facet_failover done 438 ms oleg138-client.virtnet: 1 7986 1.76 MB/sec warmup 34 sec latency 921.027 ms oleg138-client.virtnet: 1 7986 1.71 MB/sec warmup 35 sec latency 1921.210 ms oleg138-client.virtnet: 1 7986 1.66 MB/sec warmup 36 sec latency 2921.418 ms oleg138-client.virtnet: 1 7986 1.62 MB/sec warmup 37 sec latency 3921.589 ms oleg138-client.virtnet: 1 7986 1.58 MB/sec warmup 38 sec latency 4921.781 ms oleg138-client.virtnet: 1 7986 1.54 MB/sec warmup 39 sec latency 5922.061 ms oleg138-client.virtnet: 1 7986 1.50 MB/sec warmup 40 sec latency 6922.264 ms oleg138-client.virtnet: 1 7986 1.46 MB/sec warmup 41 sec latency 7922.460 ms oleg138-client.virtnet: 1 7986 1.43 MB/sec warmup 42 sec latency 8922.680 ms oleg138-client.virtnet: 1 7986 1.39 MB/sec warmup 43 sec latency 9922.924 ms oleg138-client.virtnet: 1 7986 1.36 MB/sec warmup 44 sec latency 10923.160 ms oleg138-client.virtnet: 1 7986 1.33 MB/sec warmup 45 sec latency 11923.342 ms oleg138-client.virtnet: 1 7986 1.30 MB/sec warmup 46 sec latency 12923.550 ms oleg138-client.virtnet: 1 7986 1.27 MB/sec warmup 47 sec latency 13923.680 ms oleg138-client.virtnet: 1 7986 1.25 MB/sec warmup 48 sec latency 14923.932 ms oleg138-client.virtnet: 1 7986 1.22 MB/sec warmup 49 sec latency 15924.161 ms oleg138-client.virtnet: 1 7986 1.20 MB/sec warmup 50 sec latency 16924.384 ms oleg138-client.virtnet: 1 7986 1.17 MB/sec warmup 51 sec latency 17924.706 ms oleg138-client.virtnet: 1 7986 1.15 MB/sec warmup 52 sec latency 18925.047 ms oleg138-client.virtnet: 1 8073 1.13 MB/sec warmup 53 sec latency 19597.650 ms oleg138-client.virtnet: 1 8530 1.19 MB/sec warmup 54 sec latency 19.497 ms oleg138-client.virtnet: 1 8831 1.17 MB/sec warmup 55 sec latency 15.036 ms oleg138-client.virtnet: 1 9266 1.16 MB/sec warmup 56 sec latency 11.778 ms oleg138-client.virtnet: 1 9594 1.17 MB/sec warmup 57 sec latency 19.099 ms oleg138-client.virtnet: 1 10289 1.25 MB/sec warmup 58 sec latency 18.677 ms oleg138-client.virtnet: 1 10875 1.32 MB/sec warmup 59 sec latency 16.377 ms oleg138-client.virtnet: 1 11407 0.37 MB/sec execute 1 sec latency 21.926 ms oleg138-client.virtnet: 1 11734 0.99 MB/sec execute 2 sec latency 17.692 ms oleg138-client.virtnet: 1 12155 1.70 MB/sec execute 3 sec latency 18.759 ms oleg138-client.virtnet: 1 12455 1.35 MB/sec execute 4 sec latency 20.449 ms oleg138-client.virtnet: 1 12855 1.16 MB/sec execute 5 sec latency 10.941 ms oleg138-client.virtnet: 1 13266 1.36 MB/sec execute 6 sec latency 18.193 ms oleg138-client.virtnet: 1 13954 2.06 MB/sec execute 7 sec latency 18.838 ms oleg138-client.virtnet: 1 14451 2.35 MB/sec execute 8 sec latency 13.609 ms oleg138-client.virtnet: 1 14722 2.13 MB/sec execute 9 sec latency 16.802 ms oleg138-client.virtnet: 1 15018 1.97 MB/sec execute 10 sec latency 20.268 ms oleg138-client.virtnet: 1 15329 1.92 MB/sec execute 11 sec latency 17.988 ms oleg138-client.virtnet: 1 15683 2.02 MB/sec execute 12 sec latency 19.158 ms oleg138-client.virtnet: 1 15978 1.88 MB/sec execute 13 sec latency 19.855 ms oleg138-client.virtnet: 1 16374 1.78 MB/sec execute 14 sec latency 11.549 ms oleg138-client.virtnet: 1 16805 1.81 MB/sec execute 15 sec latency 18.354 ms oleg138-client.virtnet: 1 17533 2.10 MB/sec execute 16 sec latency 18.837 ms oleg138-client.virtnet: 1 17943 2.23 MB/sec execute 17 sec latency 13.262 ms oleg138-client.virtnet: 1 18183 2.12 MB/sec execute 18 sec latency 16.005 ms oleg138-client.virtnet: 1 18486 2.03 MB/sec executoleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37420 3569012 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12072 3593732 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7162744 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg138-server Stopping /mnt/lustre-mds2 (opts:) on oleg138-server 00:51:54 (1721883114) shut down Failover mds2 to oleg138-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0001 00:52:09 (1721883129) targets are mounted 00:52:09 (1721883129) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37420 3569288 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12072 3594808 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7164096 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 00:52:22 (1721883142) shut down e 19 sec latency 24.720 ms oleg138-client.virtnet: 1 18753 1.97 MB/sec execute 20 sec latency 17.524 ms oleg138-client.virtnet: 1 19177 2.06 MB/sec execute 21 sec latency 16.837 ms oleg138-client.virtnet: 1 19463 1.98 MB/sec execute 22 sec latency 19.639 ms oleg138-client.virtnet: 1 19791 1.91 MB/sec execute 23 sec latency 12.650 ms oleg138-client.virtnet: 1 20163 1.89 MB/sec execute 24 sec latency 13.384 ms oleg138-client.virtnet: 1 20709 2.03 MB/sec execute 25 sec latency 19.622 ms oleg138-client.virtnet: 1 21329 2.14 MB/sec execute 26 sec latency 17.485 ms oleg138-client.virtnet: 1 21746 2.13 MB/sec execute 27 sec latency 14.032 ms oleg138-client.virtnet: 1 22135 2.07 MB/sec execute 28 sec latency 18.604 ms oleg138-client.virtnet: 1 22378 2.05 MB/sec execute 29 sec latency 176.550 ms oleg138-client.virtnet: 1 22378 1.98 MB/sec execute 30 sec latency 1176.733 ms oleg138-client.virtnet: 1 22378 1.91 MB/sec execute 31 sec latency 2176.951 ms oleg138-client.virtnet: 1 22378 1.85 MB/sec execute 32 sec latency 3177.231 ms oleg138-client.virtnet: 1 22378 1.80 MB/sec execute 33 sec latency 4177.441 ms oleg138-client.virtnet: 1 22378 1.75 MB/sec execute 34 sec latency 5177.718 ms oleg138-client.virtnet: 1 22378 1.70 MB/sec execute 35 sec latency 6177.928 ms oleg138-client.virtnet: 1 22378 1.65 MB/sec execute 36 sec latency 7178.211 ms oleg138-client.virtnet: 1 22378 1.60 MB/sec execute 37 sec latency 8178.445 ms oleg138-client.virtnet: 1 22378 1.56 MB/sec execute 38 sec latency 9178.701 ms oleg138-client.virtnet: 1 22378 1.52 MB/sec execute 39 sec latency 10178.938 ms oleg138-client.virtnet: 1 22378 1.48 MB/sec execute 40 sec latency 11179.230 ms oleg138-client.virtnet: 1 22378 1.45 MB/sec execute 41 sec latency 12179.410 ms oleg138-client.virtnet: 1 22378 1.41 MB/sec execute 42 sec latency 13179.543 ms oleg138-client.virtnet: 1 22378 1.38 MB/sec execute 43 sec latency 14179.713 ms oleg138-client.virtnet: 1 22378 1.35 MB/sec execute 44 sec latency 15179.851 ms oleg138-client.virtnet: 1 22378 1.32 MB/sec execute 45 sec latency 16180.090 ms oleg138-client.virtnet: 1 22378 1.29 MB/sec execute 46 sec latency 17180.372 ms oleg138-client.virtnet: 1 22378 1.26 MB/sec execute 47 sec latency 18180.707 ms oleg138-client.virtnet: 1 22378 1.24 MB/sec execute 48 sec latency 19180.989 ms oleg138-client.virtnet: 1 22771 1.28 MB/sec execute 49 sec latency 19350.845 ms oleg138-client.virtnet: 1 23059 1.25 MB/sec execute 50 sec latency 19.688 ms oleg138-client.virtnet: 1 23497 1.24 MB/sec execute 51 sec latency 10.937 ms oleg138-client.virtnet: 1 23916 1.26 MB/sec execute 52 sec latency 18.097 ms oleg138-client.virtnet: 1 24584 1.35 MB/sec execute 53 sec latency 18.319 ms oleg138-client.virtnet: 1 25050 1.40 MB/sec execute 54 sec latency 14.228 ms oleg138-client.virtnet: 1 25305 1.39 MB/sec execute 55 sec latency 23.456 ms oleg138-client.virtnet: 1 25601 1.37 MB/sec execute 56 sec latency 19.980 ms oleg138-client.virtnet: 1 25864 1.36 MB/sec execute 57 sec latency 31.406 ms oleg138-client.virtnet: 1 26263 1.41 MB/sec execute 58 sec latency 17.846 ms oleg138-client.virtnet: 1 26551 1.39 MB/sec execute 59 sec latency 16.747 ms oleg138-client.virtnet: 1 26867 1.37 MB/sec execute 60 sec latency 17.722 ms oleg138-client.virtnet: 1 27293 1.37 MB/sec execute 61 sec latency 17.431 ms oleg138-client.virtnet: 1 27805 1.44 MB/sec execute 62 sec latency 18.030 ms oleg138-client.virtnet: 1 28555 1.50 MB/sec execute 63 sec latency 17.537 ms oleg138-cliFailover mds1 to oleg138-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 00:52:37 (1721883157) targets are mounted 00:52:37 (1721883157) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37728 3567936 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12096 3593932 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49824 7161868 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg138-server Stopping /mnt/lustre-mds2 (opts:) on oleg138-server 00:52:49 (1721883169) shut down Failover mds2 to oleg138-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0001 00:53:04 (1721883184) targets are mounted 00:53:04 (1721883184) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37708 3567444 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12008 3593640 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49716 7161084 1% /mnt/lustre ent.virtnet: 1 29125 1.50 MB/sec execute 64 sec latency 9.586 ms oleg138-client.virtnet: 1 29622 1.50 MB/sec execute 65 sec latency 13.956 ms oleg138-client.virtnet: 1 29987 1.53 MB/sec execute 66 sec latency 18.999 ms oleg138-client.virtnet: 1 30251 1.51 MB/sec execute 67 sec latency 19.725 ms oleg138-client.virtnet: 1 30714 1.51 MB/sec execute 68 sec latency 13.956 ms oleg138-client.virtnet: 1 31169 1.55 MB/sec execute 69 sec latency 17.701 ms oleg138-client.virtnet: 1 31915 1.61 MB/sec execute 70 sec latency 15.267 ms oleg138-client.virtnet: 1 32228 1.61 MB/sec execute 71 sec latency 15.668 ms oleg138-client.virtnet: 1 32468 1.59 MB/sec execute 72 sec latency 16.091 ms oleg138-client.virtnet: 1 32943 1.58 MB/sec execute 73 sec latency 12.326 ms oleg138-client.virtnet: 1 33289 1.59 MB/sec execute 74 sec latency 17.943 ms oleg138-client.virtnet: 1 33613 1.60 MB/sec execute 75 sec latency 15.886 ms oleg138-client.virtnet: 1 33935 1.58 MB/sec execute 76 sec latency 20.309 ms oleg138-client.virtnet: 1 34541 1.59 MB/sec execute 77 sec latency 12.750 ms oleg138-client.virtnet: 1 35649 1.70 MB/sec execute 78 sec latency 10.544 ms oleg138-client.virtnet: 1 36240 1.70 MB/sec execute 79 sec latency 10.630 ms oleg138-client.virtnet: 1 36766 1.70 MB/sec execute 80 sec latency 12.525 ms oleg138-client.virtnet: 1 37055 1.72 MB/sec execute 81 sec latency 514.434 ms oleg138-client.virtnet: 1 37667 1.70 MB/sec execute 82 sec latency 9.924 ms oleg138-client.virtnet: 1 38162 1.72 MB/sec execute 83 sec latency 14.154 ms oleg138-client.virtnet: 1 38401 1.74 MB/sec execute 84 sec latency 776.664 ms oleg138-client.virtnet: 1 38401 1.71 MB/sec execute 85 sec latency 1776.813 ms oleg138-client.virtnet: 1 38401 1.69 MB/sec execute 86 sec latency 2777.077 ms oleg138-client.virtnet: 1 38401 1.68 MB/sec execute 87 sec latency 3777.312 ms oleg138-client.virtnet: 1 38401 1.66 MB/sec execute 88 sec latency 4777.547 ms oleg138-client.virtnet: 1 38401 1.64 MB/sec execute 89 sec latency 5777.846 ms oleg138-client.virtnet: 1 38401 1.62 MB/sec execute 90 sec latency 6778.276 ms oleg138-client.virtnet: 1 38401 1.60 MB/sec execute 91 sec latency 7778.567 ms oleg138-client.virtnet: 1 38401 1.58 MB/sec execute 92 sec latency 8778.820 ms oleg138-client.virtnet: 1 38401 1.57 MB/sec execute 93 sec latency 9779.169 ms oleg138-client.virtnet: 1 38401 1.55 MB/sec execute 94 sec latency 10779.596 ms oleg138-client.virtnet: 1 38401 1.53 MB/sec execute 95 sec latency 11779.824 ms oleg138-client.virtnet: 1 38401 1.52 MB/sec execute 96 sec latency 12780.031 ms oleg138-client.virtnet: 1 38401 1.50 MB/sec execute 97 sec latency 13780.176 ms oleg138-client.virtnet: 1 38401 1.49 MB/sec execute 98 sec latency 14780.431 ms oleg138-client.virtnet: 1 38401 1.47 MB/sec execute 99 sec latency 15780.668 ms oleg138-client.virtnet: 1 38401 1.46 MB/sec execute 100 sec latency 16780.823 ms oleg138-client.virtnet: 1 38401 1.44 MB/sec execute 101 sec latency 17780.984 ms oleg138-client.virtnet: 1 38401 1.43 MB/sec execute 102 sec latency 18781.106 ms oleg138-client.virtnet: 1 39051 1.47 MB/sec execute 103 sec latency 19210.870 ms oleg138-client.virtnet: 1 39568 1.47 MB/sec execute 104 sec latency 10.129 ms oleg138-client.virtnet: 1 40187 1.48 MB/sec execute 105 sec latency 9.636 ms oleg138-client.virtnet: 1 40835 1.49 MB/sec execute 106 sec latency 10.124 ms oleg138-client.virtnet: 1 41506 1.50 MB/sec execute 107 sec latency 9.114 ms oleg138-client.virtnet: 1 42661 1.58 MB/sec etest_70b fail mds1 7 times Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 00:53:14 (1721883194) shut down Failover mds1 to oleg138-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 00:53:29 (1721883209) targets are mounted 00:53:29 (1721883209) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37472 3568756 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12192 3591704 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49664 7160460 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg138-server Stopping /mnt/lustre-mds2 (opts:) on oleg138-server 00:53:39 (1721883219) shut down Failover mds2 to oleg138-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0001 00:53:54 (1721883234) targets are mounted 00:53:54 (1721883234) facet_failover done xecute 108 sec latency 8.741 ms oleg138-client.virtnet: 1 43082 1.58 MB/sec execute 109 sec latency 10.459 ms oleg138-client.virtnet: 1 43648 1.58 MB/sec execute 110 sec latency 10.083 ms oleg138-client.virtnet: 1 44330 1.60 MB/sec execute 111 sec latency 9.300 ms oleg138-client.virtnet: 1 45038 1.60 MB/sec execute 112 sec latency 8.942 ms oleg138-client.virtnet: 1 46223 1.68 MB/sec execute 113 sec latency 8.400 ms oleg138-client.virtnet: 1 46796 1.68 MB/sec execute 114 sec latency 7.804 ms oleg138-client.virtnet: 1 47399 1.68 MB/sec execute 115 sec latency 8.955 ms oleg138-client.virtnet: 1 48131 1.70 MB/sec execute 116 sec latency 9.361 ms oleg138-client.virtnet: 1 49052 1.74 MB/sec execute 117 sec latency 8.386 ms oleg138-client.virtnet: 1 49992 1.78 MB/sec execute 118 sec latency 8.320 ms oleg138-client.virtnet: 1 50591 1.77 MB/sec execute 119 sec latency 10.587 ms oleg138-client.virtnet: 1 51237 1.80 MB/sec execute 120 sec latency 9.633 ms oleg138-client.virtnet: 1 51854 1.79 MB/sec execute 121 sec latency 9.392 ms oleg138-client.virtnet: 1 52702 1.83 MB/sec execute 122 sec latency 9.575 ms oleg138-client.virtnet: 1 53446 1.86 MB/sec execute 123 sec latency 12.463 ms oleg138-client.virtnet: 1 53688 1.85 MB/sec execute 124 sec latency 15.646 ms oleg138-client.virtnet: 1 54007 1.84 MB/sec execute 125 sec latency 19.327 ms oleg138-client.virtnet: 1 54319 1.84 MB/sec execute 126 sec latency 18.271 ms oleg138-client.virtnet: 1 54731 1.85 MB/sec execute 127 sec latency 17.527 ms oleg138-client.virtnet: 1 55119 1.84 MB/sec execute 128 sec latency 17.415 ms oleg138-client.virtnet: 1 55834 1.84 MB/sec execute 129 sec latency 9.097 ms oleg138-client.virtnet: 1 56984 1.91 MB/sec execute 130 sec latency 8.892 ms oleg138-client.virtnet: 1 57527 1.90 MB/sec execute 131 sec latency 8.802 ms oleg138-client.virtnet: 1 58213 1.92 MB/sec execute 132 sec latency 8.713 ms oleg138-client.virtnet: 1 58808 1.91 MB/sec execute 133 sec latency 9.750 ms oleg138-client.virtnet: 1 59673 1.94 MB/sec execute 134 sec latency 17.232 ms oleg138-client.virtnet: 1 59673 1.93 MB/sec execute 135 sec latency 1017.375 ms oleg138-client.virtnet: 1 59673 1.92 MB/sec execute 136 sec latency 2017.491 ms oleg138-client.virtnet: 1 59673 1.90 MB/sec execute 137 sec latency 3017.627 ms oleg138-client.virtnet: 1 59673 1.89 MB/sec execute 138 sec latency 4017.738 ms oleg138-client.virtnet: 1 59673 1.87 MB/sec execute 139 sec latency 5017.852 ms oleg138-client.virtnet: 1 59673 1.86 MB/sec execute 140 sec latency 6017.976 ms oleg138-client.virtnet: 1 59673 1.85 MB/sec execute 141 sec latency 7018.108 ms oleg138-client.virtnet: 1 59673 1.83 MB/sec execute 142 sec latency 8018.269 ms oleg138-client.virtnet: 1 59673 1.82 MB/sec execute 143 sec latency 9018.401 ms oleg138-client.virtnet: 1 59673 1.81 MB/sec execute 144 sec latency 10018.535 ms oleg138-client.virtnet: 1 59673 1.80 MB/sec execute 145 sec latency 11018.650 ms oleg138-client.virtnet: 1 59673 1.78 MB/sec execute 146 sec latency 12018.756 ms oleg138-client.virtnet: 1 59673 1.77 MB/sec execute 147 sec latency 13018.853 ms oleg138-client.virtnet: 1 59673 1.76 MB/sec execute 148 sec latency 14019.058 ms oleg138-client.virtnet: 1 59673 1.75 MB/sec execute 149 sec latency 15019.355 ms oleg138-client.virtnet: 1 59673 1.74 MB/sec execute 150 sec latency 16019.492 ms oleg138-client.virtnet: 1 59673 1.72 MB/sec execute 151 sec latency 17019.625 ms oleg138-client.virtnet: 1 59673 1.71 MB/sec execute 152 sec latency 18019.795 ms oleg138-client.virtnet: 1 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37520 3568028 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12196 3592520 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49716 7160548 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 00:54:07 (1721883247) shut down Failover mds1 to oleg138-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 00:54:22 (1721883262) targets are mounted 00:54:22 (1721883262) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37656 3568912 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11908 3594196 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7163108 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg138-server Stopping /mnt/lustre-mds2 (opts:) on oleg138-server 00:54:35 (1721883275) shut down 59760 1.71 MB/sec execute 153 sec latency 18813.650 ms oleg138-client.virtnet: 1 60385 1.73 MB/sec execute 154 sec latency 17.360 ms oleg138-client.virtnet: 1 60653 1.73 MB/sec execute 155 sec latency 17.123 ms oleg138-client.virtnet: 1 60877 1.72 MB/sec execute 156 sec latency 18.460 ms oleg138-client.virtnet: 1 61227 1.71 MB/sec execute 157 sec latency 21.986 ms oleg138-client.virtnet: 1 61535 1.71 MB/sec execute 158 sec latency 18.296 ms oleg138-client.virtnet: 1 61932 1.72 MB/sec execute 159 sec latency 13.852 ms oleg138-client.virtnet: 1 62316 1.71 MB/sec execute 160 sec latency 19.625 ms oleg138-client.virtnet: 1 62796 1.71 MB/sec execute 161 sec latency 16.826 ms oleg138-client.virtnet: 1 63338 1.73 MB/sec execute 162 sec latency 18.115 ms oleg138-client.virtnet: 1 63944 1.75 MB/sec execute 163 sec latency 17.327 ms oleg138-client.virtnet: 1 64313 1.75 MB/sec execute 164 sec latency 12.587 ms oleg138-client.virtnet: 1 64672 1.74 MB/sec execute 165 sec latency 18.074 ms oleg138-client.virtnet: 1 65057 1.74 MB/sec execute 166 sec latency 17.934 ms oleg138-client.virtnet: 1 65495 1.75 MB/sec execute 167 sec latency 12.849 ms oleg138-client.virtnet: 1 65892 1.74 MB/sec execute 168 sec latency 15.277 ms oleg138-client.virtnet: 1 66630 1.76 MB/sec execute 169 sec latency 10.012 ms oleg138-client.virtnet: 1 67501 1.79 MB/sec execute 170 sec latency 12.075 ms oleg138-client.virtnet: 1 67770 1.79 MB/sec execute 171 sec latency 15.694 ms oleg138-client.virtnet: 1 68097 1.78 MB/sec execute 172 sec latency 16.242 ms oleg138-client.virtnet: 1 68407 1.77 MB/sec execute 173 sec latency 16.532 ms oleg138-client.virtnet: 1 68821 1.79 MB/sec execute 174 sec latency 17.762 ms oleg138-client.virtnet: 1 69112 1.78 MB/sec execute 175 sec latency 16.295 ms oleg138-client.virtnet: 1 69444 1.77 MB/sec execute 176 sec latency 18.222 ms oleg138-client.virtnet: 1 69850 1.77 MB/sec execute 177 sec latency 17.256 ms oleg138-client.virtnet: 1 70430 1.79 MB/sec execute 178 sec latency 17.515 ms oleg138-client.virtnet: 1 71165 1.81 MB/sec execute 179 sec latency 13.107 ms oleg138-client.virtnet: 1 71400 1.81 MB/sec execute 180 sec latency 16.264 ms oleg138-client.virtnet: 1 71705 1.80 MB/sec execute 181 sec latency 20.433 ms oleg138-client.virtnet: 1 72098 1.80 MB/sec execute 182 sec latency 14.872 ms oleg138-client.virtnet: 1 72461 1.80 MB/sec execute 183 sec latency 17.407 ms oleg138-client.virtnet: 1 72740 1.80 MB/sec execute 184 sec latency 19.806 ms oleg138-client.virtnet: 1 73051 1.79 MB/sec execute 185 sec latency 13.184 ms oleg138-client.virtnet: 1 73399 1.79 MB/sec execute 186 sec latency 17.754 ms oleg138-client.virtnet: 1 73790 1.80 MB/sec execute 187 sec latency 239.568 ms oleg138-client.virtnet: 1 74552 1.82 MB/sec execute 188 sec latency 14.534 ms oleg138-client.virtnet: 1 74836 1.82 MB/sec execute 189 sec latency 16.824 ms oleg138-client.virtnet: 1 74964 1.81 MB/sec execute 190 sec latency 445.323 ms oleg138-client.virtnet: 1 74964 1.80 MB/sec execute 191 sec latency 1445.550 ms oleg138-client.virtnet: 1 74964 1.79 MB/sec execute 192 sec latency 2445.823 ms oleg138-client.virtnet: 1 74964 1.78 MB/sec execute 193 sec latency 3446.085 ms oleg138-client.virtnet: 1 74964 1.77 MB/sec execute 194 sec latency 4446.302 ms oleg138-client.virtnet: 1 74964 1.77 MB/sec execute 195 sec latency 5446.566 ms oleg138-client.virtnet: 1 74964 1.76 MB/sec execute 196 sec latency 6446.850 ms oleg138-client.virtnet: 1 74964 1.75 MB/sec execute 197 sec latency 7447.076 ms oleg138-clieFailover mds2 to oleg138-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0001 00:54:50 (1721883290) targets are mounted 00:54:50 (1721883290) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569144 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12068 3593980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7163124 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 00:55:03 (1721883303) shut down Failover mds1 to oleg138-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 00:55:17 (1721883317) targets are mounted 00:55:17 (1721883317) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37420 3569012 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12152 3593100 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49572 7162112 1% /mnt/lustre nt.virtnet: 1 74964 1.74 MB/sec execute 198 sec latency 8447.322 ms oleg138-client.virtnet: 1 74964 1.73 MB/sec execute 199 sec latency 9447.563 ms oleg138-client.virtnet: 1 74964 1.72 MB/sec execute 200 sec latency 10447.818 ms oleg138-client.virtnet: 1 74964 1.71 MB/sec execute 201 sec latency 11448.090 ms oleg138-client.virtnet: 1 74964 1.70 MB/sec execute 202 sec latency 12448.299 ms oleg138-client.virtnet: 1 74964 1.70 MB/sec execute 203 sec latency 13448.505 ms oleg138-client.virtnet: 1 74964 1.69 MB/sec execute 204 sec latency 14448.767 ms oleg138-client.virtnet: 1 74964 1.68 MB/sec execute 205 sec latency 15449.005 ms oleg138-client.virtnet: 1 74964 1.67 MB/sec execute 206 sec latency 16449.270 ms oleg138-client.virtnet: 1 74964 1.66 MB/sec execute 207 sec latency 17449.436 ms oleg138-client.virtnet: 1 74964 1.65 MB/sec execute 208 sec latency 18449.587 ms oleg138-client.virtnet: 1 74964 1.65 MB/sec execute 209 sec latency 19449.780 ms oleg138-client.virtnet: 1 75293 1.64 MB/sec execute 210 sec latency 19700.701 ms oleg138-client.virtnet: 1 75949 1.66 MB/sec execute 211 sec latency 10.748 ms oleg138-client.virtnet: 1 76336 1.65 MB/sec execute 212 sec latency 19.954 ms oleg138-client.virtnet: 1 76677 1.64 MB/sec execute 213 sec latency 12.111 ms oleg138-client.virtnet: 1 77443 1.66 MB/sec execute 214 sec latency 18.218 ms oleg138-client.virtnet: 1 78096 1.68 MB/sec execute 215 sec latency 18.418 ms oleg138-client.virtnet: 1 78386 1.68 MB/sec execute 216 sec latency 16.698 ms oleg138-client.virtnet: 1 78714 1.67 MB/sec execute 217 sec latency 15.689 ms oleg138-client.virtnet: 1 79058 1.67 MB/sec execute 218 sec latency 18.243 ms oleg138-client.virtnet: 1 79579 1.68 MB/sec execute 219 sec latency 15.406 ms oleg138-client.virtnet: 1 80115 1.68 MB/sec execute 220 sec latency 12.544 ms oleg138-client.virtnet: 1 80857 1.69 MB/sec execute 221 sec latency 15.237 ms oleg138-client.virtnet: 1 81812 1.72 MB/sec execute 222 sec latency 11.452 ms oleg138-client.virtnet: 1 82380 1.72 MB/sec execute 223 sec latency 10.840 ms oleg138-client.virtnet: 1 82814 1.72 MB/sec execute 224 sec latency 18.210 ms oleg138-client.virtnet: 1 83291 1.72 MB/sec execute 225 sec latency 12.557 ms oleg138-client.virtnet: 1 83805 1.72 MB/sec execute 226 sec latency 11.493 ms oleg138-client.virtnet: 1 84571 1.74 MB/sec execute 227 sec latency 13.793 ms oleg138-client.virtnet: 1 85461 1.76 MB/sec execute 228 sec latency 10.076 ms oleg138-client.virtnet: 1 85928 1.76 MB/sec execute 229 sec latency 11.877 ms oleg138-client.virtnet: 1 86385 1.76 MB/sec execute 230 sec latency 17.969 ms oleg138-client.virtnet: 1 86796 1.76 MB/sec execute 231 sec latency 16.901 ms oleg138-client.virtnet: 1 87162 1.75 MB/sec execute 232 sec latency 16.536 ms oleg138-client.virtnet: 1 87584 1.75 MB/sec execute 233 sec latency 17.737 ms oleg138-client.virtnet: 1 88159 1.77 MB/sec execute 234 sec latency 18.128 ms oleg138-client.virtnet: 1 88751 1.78 MB/sec execute 235 sec latency 18.258 ms oleg138-client.virtnet: 1 89069 1.78 MB/sec execute 236 sec latency 15.222 ms oleg138-client.virtnet: 1 89378 1.78 MB/sec execute 237 sec latency 16.904 ms oleg138-client.virtnet: 1 89670 1.77 MB/sec execute 238 sec latency 17.505 ms oleg138-client.virtnet: 1 90117 1.78 MB/sec execute 239 sec latency 16.997 ms oleg138-client.virtnet: 1 90427 1.77 MB/sec execute 240 sec latency 19.230 ms oleg138-client.virtnet: 1 90756 1.77 MB/sec execute 241 sec latency 11.821 ms oleg138-client.virtnet: 1 91124 1.77 MB/sec execute 242test_70b fail mds2 12 times Failing mds2 on oleg138-server Stopping /mnt/lustre-mds2 (opts:) on oleg138-server 00:55:30 (1721883330) shut down Failover mds2 to oleg138-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0001 00:55:45 (1721883345) targets are mounted 00:55:45 (1721883345) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec sec latency 56.688 ms oleg138-client.virtnet: 1 91634 1.78 MB/sec execute 243 sec latency 18.389 ms oleg138-client.virtnet: 1 92251 1.80 MB/sec execute 244 sec latency 18.389 ms oleg138-client.virtnet: 1 92386 1.79 MB/sec execute 245 sec latency 588.666 ms oleg138-client.virtnet: 1 92386 1.78 MB/sec execute 246 sec latency 1588.863 ms oleg138-client.virtnet: 1 92386 1.78 MB/sec execute 247 sec latency 2589.026 ms oleg138-client.virtnet: 1 92386 1.77 MB/sec execute 248 sec latency 3589.185 ms oleg138-client.virtnet: 1 92386 1.76 MB/sec execute 249 sec latency 4589.340 ms oleg138-client.virtnet: 1 92386 1.75 MB/sec execute 250 sec latency 5589.572 ms oleg138-client.virtnet: 1 92386 1.75 MB/sec execute 251 sec latency 6589.761 ms oleg138-client.virtnet: 1 92386 1.74 MB/sec execute 252 sec latency 7590.058 ms oleg138-client.virtnet: 1 92386 1.73 MB/sec execute 253 sec latency 8590.387 ms oleg138-client.virtnet: 1 92386 1.73 MB/sec execute 254 sec latency 9590.706 ms oleg138-client.virtnet: 1 92386 1.72 MB/sec execute 255 sec latency 10591.038 ms oleg138-client.virtnet: 1 92386 1.71 MB/sec execute 256 sec latency 11591.309 ms oleg138-client.virtnet: 1 92386 1.71 MB/sec execute 257 sec latency 12591.586 ms oleg138-client.virtnet: 1 92386 1.70 MB/sec execute 258 sec latency 13591.881 ms oleg138-client.virtnet: 1 92386 1.69 MB/sec execute 259 sec latency 14592.057 ms oleg138-client.virtnet: 1 92386 1.69 MB/sec execute 260 sec latency 15592.261 ms oleg138-client.virtnet: 1 92386 1.68 MB/sec execute 261 sec latency 16592.620 ms oleg138-client.virtnet: 1 92386 1.67 MB/sec execute 262 sec latency 17592.954 ms oleg138-client.virtnet: 1 92386 1.67 MB/sec execute 263 sec latency 18593.213 ms oleg138-client.virtnet: 1 92390 1.66 MB/sec execute 264 sec latency 19572.756 ms oleg138-client.virtnet: 1 92702 1.66 MB/sec execute 265 sec latency 34.785 ms oleg138-client.virtnet: 1 93097 1.66 MB/sec execute 266 sec latency 18.398 ms oleg138-client.virtnet: 1 93518 1.66 MB/sec execute 267 sec latency 16.861 ms oleg138-client.virtnet: 1 93988 1.66 MB/sec execute 268 sec latency 18.446 ms oleg138-client.virtnet: 1 94342 1.66 MB/sec execute 269 sec latency 13.085 ms oleg138-client.virtnet: 1 94700 1.66 MB/sec execute 270 sec latency 16.516 ms oleg138-client.virtnet: 1 95282 1.67 MB/sec execute 271 sec latency 18.331 ms oleg138-client.virtnet: 1 95953 1.68 MB/sec execute 272 sec latency 18.002 ms oleg138-client.virtnet: 1 96219 1.68 MB/sec execute 273 sec latency 15.847 ms oleg138-client.virtnet: 1 96527 1.68 MB/sec execute 274 sec latency 20.709 ms oleg138-client.virtnet: 1 96796 1.67 MB/sec execute 275 sec latency 18.558 ms oleg138-client.virtnet: 1 97233 1.68 MB/sec execute 276 sec latency 17.420 ms oleg138-client.virtnet: 1 97535 1.68 MB/sec execute 277 sec latency 14.705 ms oleg138-client.virtnet: 1 97989 1.67 MB/sec execute 278 sec latency 10.784 ms oleg138-client.virtnet: 1 99012 1.69 MB/sec execute 279 sec latency 9.629 ms oleg138-client.virtnet: 1 99749 1.71 MB/sec execute 280 sec latency 8.523 ms oleg138-client.virtnet: 1 100246 1.70 MB/sec execute 281 sec latency 15.917 ms oleg138-client.virtnet: 1 100854 1.71 MB/sec execute 282 sec latency 16.821 ms oleg138-client.virtnet: 1 101317 1.71 MB/sec execute 283 sec latency 13.030 ms oleg138-client.virtnet: 1 101812 1.71 MB/sec execute 284 sec latency 15.769 ms oleg138-client.virtnet: 1 102347 1.72 MB/sec execute 285 sec latency 17.977 ms oleg138-client.virtnet: 1 102977 1.73 MB/sec execute 286 sec latency 18.805 ms oleg138-client.virtnet: 1 103247 1.73 MB/sec execute 287 sec latency 16.059 ms oleg138-client.virtnet: 1 103503 1.73 MB/sec execute 288 sec latency 15.574 ms oleg138-client.virtnet: 1 103802 1.72 MB/sec execute 289 sec latency 23.595 ms oleg138-client.virtnet: 1 104116 1.72 MB/sec execute 290 sec latency 18.500 ms oleg138-client.virtnet: 1 104496 1.73 MB/sec execute 291 sec latency 17.655 ms oleg138-client.virtnet: 1 104765 1.72 MB/sec execute 292 sec latency 20.982 ms oleg138-client.virtnet: 1 105117 1.72 MB/sec execute 293 sec latency 11.790 ms oleg138-client.virtnet: 1 105496 1.72 MB/sec execute 294 sec latency 18.890 ms oleg138-client.virtnet: 1 106179 1.73 MB/sec execute 295 sec latency 17.947 ms oleg138-client.virtnet: 1 106659 1.74 MB/sec execute 296 sec latency 14.410 ms oleg138-client.virtnet: 1 106894 1.74 MB/sec execute 297 sec latency 15.756 ms oleg138-client.virtnet: 1 107188 1.73 MB/sec execute 298 sec latency 22.414 ms oleg138-client.virtnet: 1 107448 1.73 MB/sec execute 299 sec latency 18.379 ms oleg138-client.virtnet: 1 cleanup 300 sec oleg138-client.virtnet: 0 cleanup 300 sec oleg138-client.virtnet: oleg138-client.virtnet: Operation Count AvgLat MaxLat oleg138-client.virtnet: ---------------------------------------- oleg138-client.virtnet: NTCreateX 16728 8.972 19210.830 oleg138-client.virtnet: Close 12291 4.528 19700.673 oleg138-client.virtnet: Rename 712 10.600 20.927 oleg138-client.virtnet: Unlink 3396 2.292 13.171 oleg138-client.virtnet: Qpathinfo 15162 3.147 19572.724 oleg138-client.virtnet: Qfileinfo 2637 0.168 2.553 oleg138-client.virtnet: Qfsinfo 2790 0.278 17.093 oleg138-client.virtnet: Sfileinfo 1384 6.224 17.547 oleg138-client.virtnet: Find 5869 0.484 15.137 oleg138-client.virtnet: WriteX 8268 1.052 7.756 oleg138-client.virtnet: ReadX 26229 0.020 2.095 oleg138-client.virtnet: LockX 54 1.257 2.250 oleg138-client.virtnet: UnlockX 54 1.385 2.309 oleg138-client.virtnet: Flush 1185 7.173 514.412 oleg138-client.virtnet: oleg138-client.virtnet: Throughput 1.73002 MB/sec 1 clients 1 procs max_latency=19700.701 ms oleg138-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg138-client.virtnet at Thu Jul 25 00:56:25 EDT 2024 with return code 0 oleg138-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg138-client.virtnet oleg138-client.virtnet: /mnt/lustre/d70b.replay-single/oleg138-client.virtnet /mnt/lustre/d70b.replay-single/oleg138-client.virtnet oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg138-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg138-client.virtnet: removed directory: 'clients/client0' oleg138-client.virtnet: removed directory: 'clients' oleg138-client.virtnet: removed 'client.txt' oleg138-client.virtnet: /mnt/lustre/d70b.replay-single/oleg138-client.virtnet oleg138-client.virtnet: dbench successfully finished