== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 12:38:53 (1724776733) Starting client oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre Started clients oleg429-client.virtnet: 192.168.204.129@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg429-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=12371 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2512 1285176 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1752 3592936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26140 3556160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27892 7149096 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 12:39:01 (1724776741) shut down Failover mds1 to oleg429-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 12:39:16 (1724776756) targets are mounted 12:39:16 (1724776756) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1916 3592060 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26508 3567568 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28424 7159628 1% /mnt/lustre oleg429-client.virtnet: looking for dbench program oleg429-client.virtnet: /usr/bin/dbench oleg429-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg429-client.virtnet oleg429-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg429-client.virtnet' oleg429-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg429-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg429-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg429-client.virtnet at Tue Aug 27 12:38:54 EDT 2024 oleg429-client.virtnet: waiting for dbench pid 12396 oleg429-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg429-client.virtnet: oleg429-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg429-client.virtnet: failed to create barrier semaphore oleg429-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg429-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg429-client.virtnet: releasing clients oleg429-client.virtnet: 1 339 12.26 MB/sec warmup 1 sec latency 23.412 ms oleg429-client.virtnet: 1 769 10.68 MB/sec warmup 2 sec latency 16.049 ms oleg429-client.virtnet: 1 1244 7.69 MB/sec warmup 3 sec latency 13.172 ms oleg429-client.virtnet: 1 1502 6.51 MB/sec warmup 4 sec latency 453.945 ms oleg429-client.virtnet: 1 1920 5.29 MB/sec warmup 5 sec latency 18.088 ms oleg429-client.virtnet: 1 2571 4.82 MB/sec warmup 6 sec latency 11.880 ms oleg429-client.virtnet: 1 2761 4.56 MB/sec warmup 7 sec latency 842.673 ms oleg429-client.virtnet: 1 2761 3.99 MB/sec warmup 8 sec latency 1842.896 ms oleg429-client.virtnet: 1 2761 3.55 MB/sec warmup 9 sec latency 2843.131 ms oleg429-client.virtnet: 1 2761 3.19 MB/sec warmup 10 sec latency 3843.474 ms oleg429-client.virtnet: 1 2761 2.90 MB/sec warmup 11 sec latency 4843.696 ms oleg429-client.virtnet: 1 2761 2.66 MB/sec warmup 12 sec latency 5843.892 ms oleg429-client.virtnet: 1 2761 2.46 MB/sec warmup 13 sec latency 6844.163 ms oleg429-client.virtnet: 1 2761 2.28 MB/sec warmup 14 sec latency 7844.517 ms oleg429-client.virtnet: 1 2761 2.13 MB/sec warmup 15 sec latency 8844.864 ms oleg429-client.virtnet: 1 2761 2.00 MB/sec warmup 16 sec latency 9845.091 ms oleg429-client.virtnet: 1 2761 1.88 MB/sec warmup 17 sec latency 10845.271 ms oleg429-client.virtnet: 1 2761 1.77 MB/sec warmup 18 sec latency 11845.407 ms oleg429-client.virtnet: 1 2761 1.68 MB/sec warmup 19 sec latency 12845.519 ms oleg429-client.virtnet: 1 2761 1.60 MB/sec warmup 20 sec latency 13845.731 ms oleg429-client.virtnet: 1 2761 1.52 MB/sec warmup 21 sec latency 14845.931 ms oleg429-client.virtnet: 1 2761 1.45 MB/sec warmup 22 sec latency 15846.138 ms oleg429-client.virtnet: 1 2761 1.39 MB/sec warmup 23 sec latency 16846.293 ms oleg429-client.virtnet: 1 2761 1.33 MB/sec warmup 24 sec latency 17846.509 ms oleg429-client.virtnet: 1 2761 1.28 MB/sec warmup 25 sec latency 18846.734 ms oleg429-client.virtnet: 1 2982 1.29 MB/sec warmup 26 sec latency 19567.684 ms oleg429-client.virtnet: 1 3644 1.43 MB/sec warmup 27 sec latency 17.925 ms oleg429-client.virtnet: 1 3935 1.43 MB/sec warmup 28 sec latency 15.187 ms oleg429-client.virtnet: 1 4270 1.39 MB/sec warmup 29 sec latency 15.759 ms oleg429-client.virtnet: 1 4716 1.40 MB/sec warmup 30 sec latency 15.453 ms oleg429-client.virtnet: 1 5137 1.46 MB/sec warmup 31 sec latency 16.144 ms oleg429-client.virtnet: 1 5561 1.43 MB/sec warmup 32 sec latency 18.437 ms oleg429-client.virtnet: 1 5994 1.43 MB/sec warmup 33 sec latency test_70b fail mds2 2 times Failing mds2 on oleg429-server Stopping /mnt/lustre-mds2 (opts:) on oleg429-server 12:39:29 (1724776769) shut down Failover mds2 to oleg429-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 12:39:44 (1724776784) targets are mounted 12:39:44 (1724776784) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593084 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36628 3569000 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49572 7162084 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 12:39:57 (1724776797) shut down Failover mds1 to oleg429-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 12:40:12 (1724776812) targets are mounted 12:40:12 (1724776812) facet_failover done 11.740 ms oleg429-client.virtnet: 1 6945 1.61 MB/sec warmup 34 sec latency 12.768 ms oleg429-client.virtnet: 1 7371 1.68 MB/sec warmup 35 sec latency 16.038 ms oleg429-client.virtnet: 1 7687 1.64 MB/sec warmup 36 sec latency 14.944 ms oleg429-client.virtnet: 1 8092 1.63 MB/sec warmup 37 sec latency 18.063 ms oleg429-client.virtnet: 1 8500 1.69 MB/sec warmup 38 sec latency 28.047 ms oleg429-client.virtnet: 1 8771 1.65 MB/sec warmup 39 sec latency 25.841 ms oleg429-client.virtnet: 1 9125 1.62 MB/sec warmup 40 sec latency 21.579 ms oleg429-client.virtnet: 1 9571 1.63 MB/sec warmup 41 sec latency 23.839 ms oleg429-client.virtnet: 1 10497 1.76 MB/sec warmup 42 sec latency 12.199 ms oleg429-client.virtnet: 1 10949 1.81 MB/sec warmup 43 sec latency 12.885 ms oleg429-client.virtnet: 1 11205 1.78 MB/sec warmup 44 sec latency 15.573 ms oleg429-client.virtnet: 1 11529 1.76 MB/sec warmup 45 sec latency 18.779 ms oleg429-client.virtnet: 1 11851 1.75 MB/sec warmup 46 sec latency 17.457 ms oleg429-client.virtnet: 1 12426 1.78 MB/sec warmup 47 sec latency 11.751 ms oleg429-client.virtnet: 1 12868 1.75 MB/sec warmup 48 sec latency 11.226 ms oleg429-client.virtnet: 1 13581 1.85 MB/sec warmup 49 sec latency 16.545 ms oleg429-client.virtnet: 1 13624 1.82 MB/sec warmup 50 sec latency 934.969 ms oleg429-client.virtnet: 1 13624 1.78 MB/sec warmup 51 sec latency 1935.170 ms oleg429-client.virtnet: 1 13624 1.75 MB/sec warmup 52 sec latency 2935.392 ms oleg429-client.virtnet: 1 13624 1.71 MB/sec warmup 53 sec latency 3935.588 ms oleg429-client.virtnet: 1 14225 1.77 MB/sec warmup 54 sec latency 3998.832 ms oleg429-client.virtnet: 1 14514 1.77 MB/sec warmup 55 sec latency 15.481 ms oleg429-client.virtnet: 1 14803 1.74 MB/sec warmup 56 sec latency 15.661 ms oleg429-client.virtnet: 1 15185 1.73 MB/sec warmup 57 sec latency 17.874 ms oleg429-client.virtnet: 1 15649 1.77 MB/sec warmup 58 sec latency 17.485 ms oleg429-client.virtnet: 1 15930 1.74 MB/sec warmup 59 sec latency 18.473 ms oleg429-client.virtnet: 1 16649 1.44 MB/sec execute 1 sec latency 85.234 ms oleg429-client.virtnet: 1 17202 3.43 MB/sec execute 2 sec latency 17.475 ms oleg429-client.virtnet: 1 17209 2.29 MB/sec execute 3 sec latency 956.570 ms oleg429-client.virtnet: 1 17209 1.72 MB/sec execute 4 sec latency 1956.899 ms oleg429-client.virtnet: 1 17209 1.37 MB/sec execute 5 sec latency 2957.148 ms oleg429-client.virtnet: 1 17209 1.14 MB/sec execute 6 sec latency 3957.400 ms oleg429-client.virtnet: 1 17209 0.98 MB/sec execute 7 sec latency 4957.638 ms oleg429-client.virtnet: 1 17209 0.86 MB/sec execute 8 sec latency 5957.917 ms oleg429-client.virtnet: 1 17209 0.76 MB/sec execute 9 sec latency 6958.189 ms oleg429-client.virtnet: 1 17209 0.69 MB/sec execute 10 sec latency 7958.511 ms oleg429-client.virtnet: 1 17209 0.62 MB/sec execute 11 sec latency 8958.757 ms oleg429-client.virtnet: 1 17209 0.57 MB/sec execute 12 sec latency 9959.000 ms oleg429-client.virtnet: 1 17209 0.53 MB/sec execute 13 sec latency 10959.171 ms oleg429-client.virtnet: 1 17209 0.49 MB/sec execute 14 sec latency 11959.459 ms oleg429-client.virtnet: 1 17209 0.46 MB/sec execute 15 sec latency 12959.680 ms oleg429-client.virtnet: 1 17209 0.43 MB/sec execute 16 sec latency 13959.935 ms oleg429-client.virtnet: 1 17209 0.40 MB/sec execute 17 sec latency 14960.309 ms oleg429-client.virtnet: 1 17209 0.38 MB/sec execute 18 sec latency 15960.551 ms oleg429-client.virtnet: 1 17209 0.36 MB/sec executoleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13124 3593016 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38596 3567400 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51720 7160416 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg429-server Stopping /mnt/lustre-mds2 (opts:) on oleg429-server 12:40:25 (1724776825) shut down Failover mds2 to oleg429-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 12:40:40 (1724776840) targets are mounted 12:40:40 (1724776840) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13080 3592948 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36636 3567452 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49716 7160400 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 12:40:51 (1724776851) shut down e 19 sec latency 16960.854 ms oleg429-client.virtnet: 1 17209 0.34 MB/sec execute 20 sec latency 17961.094 ms oleg429-client.virtnet: 1 17209 0.33 MB/sec execute 21 sec latency 18961.308 ms oleg429-client.virtnet: 1 17682 0.53 MB/sec execute 22 sec latency 19518.467 ms oleg429-client.virtnet: 1 18084 0.58 MB/sec execute 23 sec latency 13.706 ms oleg429-client.virtnet: 1 18470 0.57 MB/sec execute 24 sec latency 15.780 ms oleg429-client.virtnet: 1 18740 0.57 MB/sec execute 25 sec latency 17.940 ms oleg429-client.virtnet: 1 19179 0.71 MB/sec execute 26 sec latency 17.358 ms oleg429-client.virtnet: 1 19541 0.69 MB/sec execute 27 sec latency 14.661 ms oleg429-client.virtnet: 1 20012 0.69 MB/sec execute 28 sec latency 11.877 ms oleg429-client.virtnet: 1 20655 0.85 MB/sec execute 29 sec latency 14.759 ms oleg429-client.virtnet: 1 21286 1.03 MB/sec execute 30 sec latency 17.133 ms oleg429-client.virtnet: 1 21609 1.04 MB/sec execute 31 sec latency 15.102 ms oleg429-client.virtnet: 1 21848 1.02 MB/sec execute 32 sec latency 16.134 ms oleg429-client.virtnet: 1 22186 1.01 MB/sec execute 33 sec latency 17.863 ms oleg429-client.virtnet: 1 22588 1.02 MB/sec execute 34 sec latency 17.747 ms oleg429-client.virtnet: 1 22949 1.08 MB/sec execute 35 sec latency 14.763 ms oleg429-client.virtnet: 1 23324 1.06 MB/sec execute 36 sec latency 19.830 ms oleg429-client.virtnet: 1 23720 1.07 MB/sec execute 37 sec latency 17.003 ms oleg429-client.virtnet: 1 24336 1.19 MB/sec execute 38 sec latency 17.087 ms oleg429-client.virtnet: 1 24961 1.29 MB/sec execute 39 sec latency 17.551 ms oleg429-client.virtnet: 1 25394 1.29 MB/sec execute 40 sec latency 13.400 ms oleg429-client.virtnet: 1 25795 1.28 MB/sec execute 41 sec latency 17.767 ms oleg429-client.virtnet: 1 26166 1.28 MB/sec execute 42 sec latency 17.756 ms oleg429-client.virtnet: 1 26545 1.32 MB/sec execute 43 sec latency 15.747 ms oleg429-client.virtnet: 1 27062 1.31 MB/sec execute 44 sec latency 17.603 ms oleg429-client.virtnet: 1 27267 1.30 MB/sec execute 45 sec latency 766.384 ms oleg429-client.virtnet: 1 27267 1.28 MB/sec execute 46 sec latency 1766.778 ms oleg429-client.virtnet: 1 27267 1.25 MB/sec execute 47 sec latency 2766.967 ms oleg429-client.virtnet: 1 27267 1.22 MB/sec execute 48 sec latency 3767.133 ms oleg429-client.virtnet: 1 27269 1.20 MB/sec execute 49 sec latency 4765.635 ms oleg429-client.virtnet: 1 28388 1.38 MB/sec execute 50 sec latency 9.748 ms oleg429-client.virtnet: 1 28787 1.39 MB/sec execute 51 sec latency 14.588 ms oleg429-client.virtnet: 1 29231 1.37 MB/sec execute 52 sec latency 16.053 ms oleg429-client.virtnet: 1 29933 1.43 MB/sec execute 53 sec latency 9.241 ms oleg429-client.virtnet: 1 30307 1.41 MB/sec execute 54 sec latency 37.387 ms oleg429-client.virtnet: 1 30646 1.39 MB/sec execute 55 sec latency 375.699 ms oleg429-client.virtnet: 1 31612 1.50 MB/sec execute 56 sec latency 11.859 ms oleg429-client.virtnet: 1 32111 1.55 MB/sec execute 57 sec latency 402.927 ms oleg429-client.virtnet: 1 32111 1.53 MB/sec execute 58 sec latency 1403.119 ms oleg429-client.virtnet: 1 32111 1.50 MB/sec execute 59 sec latency 2403.348 ms oleg429-client.virtnet: 1 32111 1.48 MB/sec execute 60 sec latency 3403.572 ms oleg429-client.virtnet: 1 32111 1.45 MB/sec execute 61 sec latency 4403.744 ms oleg429-client.virtnet: 1 32111 1.43 MB/sec execute 62 sec latency 5403.899 ms oleg429-client.virtnet: 1 32111 1.41 MB/sec execute 63 sec latency 6404.106 ms oleg429-client.virtnet: 1 Failover mds1 to oleg429-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 12:41:05 (1724776865) targets are mounted 12:41:05 (1724776865) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13016 3593120 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38668 3567960 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51684 7161080 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg429-server Stopping /mnt/lustre-mds2 (opts:) on oleg429-server 12:41:18 (1724776878) shut down Failover mds2 to oleg429-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 12:41:33 (1724776893) targets are mounted 12:41:33 (1724776893) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13080 3592948 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36556 3567532 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160480 1% /mnt/lustre 32111 1.38 MB/sec execute 64 sec latency 7404.296 ms oleg429-client.virtnet: 1 32111 1.36 MB/sec execute 65 sec latency 8404.499 ms oleg429-client.virtnet: 1 32111 1.34 MB/sec execute 66 sec latency 9404.732 ms oleg429-client.virtnet: 1 32111 1.32 MB/sec execute 67 sec latency 10404.903 ms oleg429-client.virtnet: 1 32111 1.30 MB/sec execute 68 sec latency 11405.025 ms oleg429-client.virtnet: 1 32111 1.28 MB/sec execute 69 sec latency 12405.147 ms oleg429-client.virtnet: 1 32111 1.27 MB/sec execute 70 sec latency 13405.253 ms oleg429-client.virtnet: 1 32111 1.25 MB/sec execute 71 sec latency 14405.424 ms oleg429-client.virtnet: 1 32111 1.23 MB/sec execute 72 sec latency 15405.697 ms oleg429-client.virtnet: 1 32111 1.21 MB/sec execute 73 sec latency 16405.964 ms oleg429-client.virtnet: 1 32111 1.20 MB/sec execute 74 sec latency 17406.161 ms oleg429-client.virtnet: 1 32111 1.18 MB/sec execute 75 sec latency 18406.287 ms oleg429-client.virtnet: 1 32558 1.18 MB/sec execute 76 sec latency 18511.341 ms oleg429-client.virtnet: 1 32954 1.18 MB/sec execute 77 sec latency 17.256 ms oleg429-client.virtnet: 1 33430 1.22 MB/sec execute 78 sec latency 16.480 ms oleg429-client.virtnet: 1 33990 1.21 MB/sec execute 79 sec latency 11.095 ms oleg429-client.virtnet: 1 34389 1.21 MB/sec execute 80 sec latency 14.786 ms oleg429-client.virtnet: 1 35151 1.27 MB/sec execute 81 sec latency 16.784 ms oleg429-client.virtnet: 1 35759 1.33 MB/sec execute 82 sec latency 12.300 ms oleg429-client.virtnet: 1 36079 1.31 MB/sec execute 83 sec latency 14.412 ms oleg429-client.virtnet: 1 36678 1.32 MB/sec execute 84 sec latency 9.966 ms oleg429-client.virtnet: 1 37167 1.34 MB/sec execute 85 sec latency 15.849 ms oleg429-client.virtnet: 1 37494 1.33 MB/sec execute 86 sec latency 19.350 ms oleg429-client.virtnet: 1 38103 1.35 MB/sec execute 87 sec latency 14.105 ms oleg429-client.virtnet: 1 38902 1.40 MB/sec execute 88 sec latency 17.392 ms oleg429-client.virtnet: 1 39302 1.44 MB/sec execute 89 sec latency 15.250 ms oleg429-client.virtnet: 1 39547 1.42 MB/sec execute 90 sec latency 14.362 ms oleg429-client.virtnet: 1 39923 1.42 MB/sec execute 91 sec latency 16.421 ms oleg429-client.virtnet: 1 40321 1.42 MB/sec execute 92 sec latency 17.776 ms oleg429-client.virtnet: 1 41041 1.44 MB/sec execute 93 sec latency 9.609 ms oleg429-client.virtnet: 1 41513 1.44 MB/sec execute 94 sec latency 16.598 ms oleg429-client.virtnet: 1 42071 1.48 MB/sec execute 95 sec latency 16.831 ms oleg429-client.virtnet: 1 42881 1.53 MB/sec execute 96 sec latency 14.313 ms oleg429-client.virtnet: 1 43342 1.52 MB/sec execute 97 sec latency 14.997 ms oleg429-client.virtnet: 1 43673 1.52 MB/sec execute 98 sec latency 16.984 ms oleg429-client.virtnet: 1 43733 1.51 MB/sec execute 99 sec latency 892.222 ms oleg429-client.virtnet: 1 43733 1.49 MB/sec execute 100 sec latency 1892.499 ms oleg429-client.virtnet: 1 43733 1.48 MB/sec execute 101 sec latency 2892.671 ms oleg429-client.virtnet: 1 44021 1.49 MB/sec execute 102 sec latency 3544.329 ms oleg429-client.virtnet: 1 44611 1.48 MB/sec execute 103 sec latency 10.137 ms oleg429-client.virtnet: 1 45518 1.54 MB/sec execute 104 sec latency 8.505 ms oleg429-client.virtnet: 1 46293 1.57 MB/sec execute 105 sec latency 13.018 ms oleg429-client.virtnet: 1 46563 1.57 MB/sec execute 106 sec latency 17.424 ms oleg429-client.virtnet: 1 46824 1.56 MB/sec execute 107 sec latency 316.066 ms oleg429-client.virtnet: 1 47383 1.56 MB/sec execute 108 sec latency test_70b fail mds1 7 times Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 12:41:44 (1724776904) shut down Failover mds1 to oleg429-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 12:41:57 (1724776917) targets are mounted 12:41:57 (1724776917) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13060 3592980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36592 3567468 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49652 7160448 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg429-server Stopping /mnt/lustre-mds2 (opts:) on oleg429-server 12:42:09 (1724776929) shut down Failover mds2 to oleg429-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 12:42:23 (1724776943) targets are mounted 12:42:23 (1724776943) facet_failover done 398.968 ms oleg429-client.virtnet: 1 48040 1.58 MB/sec execute 109 sec latency 10.880 ms oleg429-client.virtnet: 1 48139 1.56 MB/sec execute 110 sec latency 891.448 ms oleg429-client.virtnet: 1 48139 1.55 MB/sec execute 111 sec latency 1891.648 ms oleg429-client.virtnet: 1 48139 1.53 MB/sec execute 112 sec latency 2891.817 ms oleg429-client.virtnet: 1 48139 1.52 MB/sec execute 113 sec latency 3891.994 ms oleg429-client.virtnet: 1 48139 1.51 MB/sec execute 114 sec latency 4892.182 ms oleg429-client.virtnet: 1 48139 1.49 MB/sec execute 115 sec latency 5892.324 ms oleg429-client.virtnet: 1 48139 1.48 MB/sec execute 116 sec latency 6892.519 ms oleg429-client.virtnet: 1 48139 1.47 MB/sec execute 117 sec latency 7892.678 ms oleg429-client.virtnet: 1 48139 1.46 MB/sec execute 118 sec latency 8892.841 ms oleg429-client.virtnet: 1 48139 1.44 MB/sec execute 119 sec latency 9893.059 ms oleg429-client.virtnet: 1 48139 1.43 MB/sec execute 120 sec latency 10893.193 ms oleg429-client.virtnet: 1 48139 1.42 MB/sec execute 121 sec latency 11893.326 ms oleg429-client.virtnet: 1 48139 1.41 MB/sec execute 122 sec latency 12893.547 ms oleg429-client.virtnet: 1 48139 1.40 MB/sec execute 123 sec latency 13893.730 ms oleg429-client.virtnet: 1 48139 1.39 MB/sec execute 124 sec latency 14893.945 ms oleg429-client.virtnet: 1 48139 1.38 MB/sec execute 125 sec latency 15894.228 ms oleg429-client.virtnet: 1 48139 1.36 MB/sec execute 126 sec latency 16894.489 ms oleg429-client.virtnet: 1 48139 1.35 MB/sec execute 127 sec latency 17894.674 ms oleg429-client.virtnet: 1 48589 1.36 MB/sec execute 128 sec latency 18207.056 ms oleg429-client.virtnet: 1 49726 1.43 MB/sec execute 129 sec latency 8.651 ms oleg429-client.virtnet: 1 50263 1.43 MB/sec execute 130 sec latency 8.161 ms oleg429-client.virtnet: 1 50872 1.43 MB/sec execute 131 sec latency 9.918 ms oleg429-client.virtnet: 1 51517 1.45 MB/sec execute 132 sec latency 9.567 ms oleg429-client.virtnet: 1 52177 1.45 MB/sec execute 133 sec latency 9.503 ms oleg429-client.virtnet: 1 53307 1.52 MB/sec execute 134 sec latency 9.340 ms oleg429-client.virtnet: 1 53836 1.52 MB/sec execute 135 sec latency 10.596 ms oleg429-client.virtnet: 1 54440 1.52 MB/sec execute 136 sec latency 10.396 ms oleg429-client.virtnet: 1 55074 1.53 MB/sec execute 137 sec latency 10.160 ms oleg429-client.virtnet: 1 55802 1.54 MB/sec execute 138 sec latency 8.950 ms oleg429-client.virtnet: 1 56913 1.60 MB/sec execute 139 sec latency 9.244 ms oleg429-client.virtnet: 1 57484 1.60 MB/sec execute 140 sec latency 8.329 ms oleg429-client.virtnet: 1 58176 1.62 MB/sec execute 141 sec latency 9.234 ms oleg429-client.virtnet: 1 58806 1.61 MB/sec execute 142 sec latency 9.830 ms oleg429-client.virtnet: 1 59495 1.64 MB/sec execute 143 sec latency 13.786 ms oleg429-client.virtnet: 1 60348 1.68 MB/sec execute 144 sec latency 18.513 ms oleg429-client.virtnet: 1 60695 1.67 MB/sec execute 145 sec latency 12.526 ms oleg429-client.virtnet: 1 61087 1.67 MB/sec execute 146 sec latency 15.841 ms oleg429-client.virtnet: 1 61720 1.69 MB/sec execute 147 sec latency 15.040 ms oleg429-client.virtnet: 1 62239 1.68 MB/sec execute 148 sec latency 10.763 ms oleg429-client.virtnet: 1 62821 1.68 MB/sec execute 149 sec latency 10.013 ms oleg429-client.virtnet: 1 63206 1.69 MB/sec execute 150 sec latency 742.631 ms oleg429-client.virtnet: 1 63206 1.68 MB/sec execute 151 sec latency 1742.818 ms oleg429-client.virtnet: 1 63206 1.67 MB/sec execute 152 sec latency 2743.021 ms oleg429-client.virtnet: 1 63506 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13068 3593076 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36568 3569780 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7162856 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 12:42:35 (1724776955) shut down Failover mds1 to oleg429-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 12:42:50 (1724776970) targets are mounted 12:42:50 (1724776970) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13180 3593784 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36384 3567964 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7161748 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg429-server Stopping /mnt/lustre-mds2 (opts:) on oleg429-server 12:43:03 (1724776983) shut down 1.67 MB/sec execute 153 sec latency 3333.360 ms oleg429-client.virtnet: 1 64053 1.70 MB/sec execute 154 sec latency 16.848 ms oleg429-client.virtnet: 1 64384 1.69 MB/sec execute 155 sec latency 15.355 ms oleg429-client.virtnet: 1 64980 1.69 MB/sec execute 156 sec latency 11.171 ms oleg429-client.virtnet: 1 65584 1.70 MB/sec execute 157 sec latency 17.280 ms oleg429-client.virtnet: 1 66177 1.70 MB/sec execute 158 sec latency 12.048 ms oleg429-client.virtnet: 1 67080 1.73 MB/sec execute 159 sec latency 86.908 ms oleg429-client.virtnet: 1 67563 1.75 MB/sec execute 160 sec latency 12.686 ms oleg429-client.virtnet: 1 67745 1.74 MB/sec execute 161 sec latency 448.973 ms oleg429-client.virtnet: 1 67745 1.73 MB/sec execute 162 sec latency 1449.052 ms oleg429-client.virtnet: 1 67745 1.72 MB/sec execute 163 sec latency 2449.235 ms oleg429-client.virtnet: 1 67745 1.71 MB/sec execute 164 sec latency 3449.435 ms oleg429-client.virtnet: 1 67745 1.70 MB/sec execute 165 sec latency 4449.678 ms oleg429-client.virtnet: 1 67745 1.69 MB/sec execute 166 sec latency 5449.877 ms oleg429-client.virtnet: 1 67745 1.68 MB/sec execute 167 sec latency 6450.056 ms oleg429-client.virtnet: 1 67745 1.67 MB/sec execute 168 sec latency 7450.244 ms oleg429-client.virtnet: 1 67745 1.66 MB/sec execute 169 sec latency 8450.490 ms oleg429-client.virtnet: 1 67745 1.65 MB/sec execute 170 sec latency 9450.866 ms oleg429-client.virtnet: 1 67745 1.64 MB/sec execute 171 sec latency 10451.188 ms oleg429-client.virtnet: 1 67745 1.63 MB/sec execute 172 sec latency 11451.390 ms oleg429-client.virtnet: 1 67745 1.62 MB/sec execute 173 sec latency 12451.588 ms oleg429-client.virtnet: 1 67745 1.61 MB/sec execute 174 sec latency 13451.768 ms oleg429-client.virtnet: 1 67745 1.60 MB/sec execute 175 sec latency 14452.032 ms oleg429-client.virtnet: 1 67745 1.60 MB/sec execute 176 sec latency 15452.261 ms oleg429-client.virtnet: 1 67745 1.59 MB/sec execute 177 sec latency 16452.371 ms oleg429-client.virtnet: 1 67745 1.58 MB/sec execute 178 sec latency 17452.655 ms oleg429-client.virtnet: 1 67745 1.57 MB/sec execute 179 sec latency 18452.989 ms oleg429-client.virtnet: 1 67860 1.56 MB/sec execute 180 sec latency 19059.688 ms oleg429-client.virtnet: 1 68208 1.56 MB/sec execute 181 sec latency 12.795 ms oleg429-client.virtnet: 1 68560 1.56 MB/sec execute 182 sec latency 17.602 ms oleg429-client.virtnet: 1 69014 1.56 MB/sec execute 183 sec latency 16.675 ms oleg429-client.virtnet: 1 69452 1.56 MB/sec execute 184 sec latency 11.578 ms oleg429-client.virtnet: 1 69837 1.56 MB/sec execute 185 sec latency 17.735 ms oleg429-client.virtnet: 1 70589 1.58 MB/sec execute 186 sec latency 17.594 ms oleg429-client.virtnet: 1 71164 1.60 MB/sec execute 187 sec latency 12.531 ms oleg429-client.virtnet: 1 71421 1.60 MB/sec execute 188 sec latency 15.350 ms oleg429-client.virtnet: 1 71849 1.59 MB/sec execute 189 sec latency 19.123 ms oleg429-client.virtnet: 1 72243 1.59 MB/sec execute 190 sec latency 17.865 ms oleg429-client.virtnet: 1 72956 1.60 MB/sec execute 191 sec latency 9.477 ms oleg429-client.virtnet: 1 73562 1.61 MB/sec execute 192 sec latency 18.835 ms oleg429-client.virtnet: 1 74192 1.62 MB/sec execute 193 sec latency 19.308 ms oleg429-client.virtnet: 1 74699 1.64 MB/sec execute 194 sec latency 16.951 ms oleg429-client.virtnet: 1 75210 1.64 MB/sec execute 195 sec latency 11.768 ms oleg429-client.virtnet: 1 75541 1.63 MB/sec execute 196 sec latency 18.810 ms oleg429-client.virtnet: 1 76054 1.65 MB/sec execute 197 sec latency 16Failover mds2 to oleg429-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 12:43:17 (1724776997) targets are mounted 12:43:17 (1724776997) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13196 3590828 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36508 3569984 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49704 7160812 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg429-server Stopping /mnt/lustre-mds1 (opts:) on oleg429-server 12:43:29 (1724777009) shut down Failover mds1 to oleg429-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 12:43:44 (1724777024) targets are mounted 12:43:44 (1724777024) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13116 3590000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36508 3569924 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7159924 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg429-server Stopping /mnt/lustre-mds2 (opts:) on oleg429-server 12:43:56 (1724777036) shut down .350 ms oleg429-client.virtnet: 1 76557 1.64 MB/sec execute 198 sec latency 11.110 ms oleg429-client.virtnet: 1 76961 1.64 MB/sec execute 199 sec latency 18.410 ms oleg429-client.virtnet: 1 77731 1.66 MB/sec execute 200 sec latency 17.366 ms oleg429-client.virtnet: 1 78415 1.68 MB/sec execute 201 sec latency 11.214 ms oleg429-client.virtnet: 1 78892 1.68 MB/sec execute 202 sec latency 11.993 ms oleg429-client.virtnet: 1 79529 1.69 MB/sec execute 203 sec latency 11.658 ms oleg429-client.virtnet: 1 79615 1.68 MB/sec execute 204 sec latency 856.596 ms oleg429-client.virtnet: 1 79615 1.68 MB/sec execute 205 sec latency 1856.777 ms oleg429-client.virtnet: 1 79615 1.67 MB/sec execute 206 sec latency 2856.997 ms oleg429-client.virtnet: 1 79815 1.66 MB/sec execute 207 sec latency 3466.660 ms oleg429-client.virtnet: 1 80476 1.66 MB/sec execute 208 sec latency 8.257 ms oleg429-client.virtnet: 1 81412 1.69 MB/sec execute 209 sec latency 14.764 ms oleg429-client.virtnet: 1 81918 1.70 MB/sec execute 210 sec latency 12.376 ms oleg429-client.virtnet: 1 82416 1.70 MB/sec execute 211 sec latency 15.497 ms oleg429-client.virtnet: 1 82776 1.70 MB/sec execute 212 sec latency 406.409 ms oleg429-client.virtnet: 1 83422 1.70 MB/sec execute 213 sec latency 9.297 ms oleg429-client.virtnet: 1 84031 1.70 MB/sec execute 214 sec latency 104.269 ms oleg429-client.virtnet: 1 84031 1.70 MB/sec execute 215 sec latency 1104.401 ms oleg429-client.virtnet: 1 84031 1.69 MB/sec execute 216 sec latency 2104.599 ms oleg429-client.virtnet: 1 84031 1.68 MB/sec execute 217 sec latency 3104.846 ms oleg429-client.virtnet: 1 84031 1.67 MB/sec execute 218 sec latency 4105.062 ms oleg429-client.virtnet: 1 84031 1.67 MB/sec execute 219 sec latency 5105.301 ms oleg429-client.virtnet: 1 84031 1.66 MB/sec execute 220 sec latency 6105.458 ms oleg429-client.virtnet: 1 84031 1.65 MB/sec execute 221 sec latency 7105.703 ms oleg429-client.virtnet: 1 84031 1.64 MB/sec execute 222 sec latency 8105.998 ms oleg429-client.virtnet: 1 84031 1.64 MB/sec execute 223 sec latency 9106.301 ms oleg429-client.virtnet: 1 84031 1.63 MB/sec execute 224 sec latency 10106.499 ms oleg429-client.virtnet: 1 84031 1.62 MB/sec execute 225 sec latency 11106.627 ms oleg429-client.virtnet: 1 84031 1.61 MB/sec execute 226 sec latency 12106.856 ms oleg429-client.virtnet: 1 84031 1.61 MB/sec execute 227 sec latency 13106.996 ms oleg429-client.virtnet: 1 84031 1.60 MB/sec execute 228 sec latency 14107.140 ms oleg429-client.virtnet: 1 84031 1.59 MB/sec execute 229 sec latency 15107.434 ms oleg429-client.virtnet: 1 84031 1.59 MB/sec execute 230 sec latency 16107.662 ms oleg429-client.virtnet: 1 84031 1.58 MB/sec execute 231 sec latency 17107.854 ms oleg429-client.virtnet: 1 84031 1.57 MB/sec execute 232 sec latency 18108.071 ms oleg429-client.virtnet: 1 84031 1.57 MB/sec execute 233 sec latency 19108.228 ms oleg429-client.virtnet: 1 84627 1.58 MB/sec execute 234 sec latency 19234.445 ms oleg429-client.virtnet: 1 85422 1.60 MB/sec execute 235 sec latency 10.484 ms oleg429-client.virtnet: 1 85947 1.60 MB/sec execute 236 sec latency 13.203 ms oleg429-client.virtnet: 1 86346 1.60 MB/sec execute 237 sec latency 15.645 ms oleg429-client.virtnet: 1 86742 1.60 MB/sec execute 238 sec latency 17.378 ms oleg429-client.virtnet: 1 87287 1.60 MB/sec execute 239 sec latency 10.219 ms oleg429-client.virtnet: 1 87762 1.60 MB/sec execute 240 sec latency 18.137 ms oleg429-client.virtnet: 1 88553 1.62 MB/sec execute 241 sec latency 17.004 ms oleg429-client.virtnet: Failover mds2 to oleg429-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 12:44:10 (1724777050) targets are mounted 12:44:10 (1724777050) facet_failover done oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 89010 1.63 MB/sec execute 242 sec latency 15.251 ms oleg429-client.virtnet: 1 89255 1.63 MB/sec execute 243 sec latency 15.285 ms oleg429-client.virtnet: 1 89772 1.63 MB/sec execute 244 sec latency 20.481 ms oleg429-client.virtnet: 1 90245 1.64 MB/sec execute 245 sec latency 16.174 ms oleg429-client.virtnet: 1 90529 1.63 MB/sec execute 246 sec latency 19.097 ms oleg429-client.virtnet: 1 90934 1.63 MB/sec execute 247 sec latency 11.841 ms oleg429-client.virtnet: 1 91381 1.63 MB/sec execute 248 sec latency 17.793 ms oleg429-client.virtnet: 1 92113 1.65 MB/sec execute 249 sec latency 17.829 ms oleg429-client.virtnet: 1 92657 1.66 MB/sec execute 250 sec latency 11.271 ms oleg429-client.virtnet: 1 93247 1.66 MB/sec execute 251 sec latency 9.926 ms oleg429-client.virtnet: 1 93922 1.67 MB/sec execute 252 sec latency 9.851 ms oleg429-client.virtnet: 1 94619 1.67 MB/sec execute 253 sec latency 9.412 ms oleg429-client.virtnet: 1 95657 1.69 MB/sec execute 254 sec latency 9.615 ms oleg429-client.virtnet: 1 96276 1.70 MB/sec execute 255 sec latency 10.353 ms oleg429-client.virtnet: 1 96631 1.70 MB/sec execute 256 sec latency 444.722 ms oleg429-client.virtnet: 1 96631 1.69 MB/sec execute 257 sec latency 1444.939 ms oleg429-client.virtnet: 1 96631 1.68 MB/sec execute 258 sec latency 2445.158 ms oleg429-client.virtnet: 1 96631 1.68 MB/sec execute 259 sec latency 3445.344 ms oleg429-client.virtnet: 1 96867 1.68 MB/sec execute 260 sec latency 3792.997 ms oleg429-client.virtnet: 1 97396 1.68 MB/sec execute 261 sec latency 12.708 ms oleg429-client.virtnet: 1 97677 1.68 MB/sec execute 262 sec latency 19.841 ms oleg429-client.virtnet: 1 98082 1.67 MB/sec execute 263 sec latency 12.651 ms oleg429-client.virtnet: 1 98425 1.67 MB/sec execute 264 sec latency 18.556 ms oleg429-client.virtnet: 1 99200 1.69 MB/sec execute 265 sec latency 17.494 ms oleg429-client.virtnet: 1 99706 1.70 MB/sec execute 266 sec latency 12.338 ms oleg429-client.virtnet: 1 100285 1.70 MB/sec execute 267 sec latency 11.120 ms oleg429-client.virtnet: 1 100755 1.71 MB/sec execute 268 sec latency 18.600 ms oleg429-client.virtnet: 1 101150 1.70 MB/sec execute 269 sec latency 13.617 ms oleg429-client.virtnet: 1 101866 1.71 MB/sec execute 270 sec latency 9.353 ms oleg429-client.virtnet: 1 103002 1.74 MB/sec execute 271 sec latency 10.484 ms oleg429-client.virtnet: 1 103515 1.74 MB/sec execute 272 sec latency 10.844 ms oleg429-client.virtnet: 1 104072 1.74 MB/sec execute 273 sec latency 14.280 ms oleg429-client.virtnet: 1 104687 1.74 MB/sec execute 274 sec latency 13.879 ms oleg429-client.virtnet: 1 105385 1.75 MB/sec execute 275 sec latency 8.553 ms oleg429-client.virtnet: 1 106528 1.78 MB/sec execute 276 sec latency 9.563 ms oleg429-client.virtnet: 1 107043 1.77 MB/sec execute 277 sec latency 9.270 ms oleg429-client.virtnet: 1 107672 1.78 MB/sec execute 278 sec latency 11.628 ms oleg429-client.virtnet: 1 108143 1.78 MB/sec execute 279 sec latency 19.564 ms oleg429-client.virtnet: 1 108464 1.78 MB/sec execute 280 sec latency 15.376 ms oleg429-client.virtnet: 1 108853 1.77 MB/sec execute 281 sec latency 16.776 ms oleg429-client.virtnet: 1 109627 1.79 MB/sec execute 282 sec latency 17.527 ms oleg429-client.virtnet: 1 110285 1.80 MB/sec execute 283 sec latency 18.819 ms oleg429-client.virtnet: 1 110862 1.80 MB/sec execute 284 sec latency 9.945 ms oleg429-client.virtnet: 1 111553 1.81 MB/sec execute 285 sec latency 8.657 ms oleg429-client.virtnet: 1 112014 1.80 MB/sec execute 286 sec latency 11.798 ms oleg429-client.virtnet: 1 112419 1.80 MB/sec execute 287 sec latency 16.650 ms oleg429-client.virtnet: 1 113172 1.82 MB/sec execute 288 sec latency 17.503 ms oleg429-client.virtnet: 1 113659 1.83 MB/sec execute 289 sec latency 16.042 ms oleg429-client.virtnet: 1 114018 1.83 MB/sec execute 290 sec latency 16.577 ms oleg429-client.virtnet: 1 114489 1.82 MB/sec execute 291 sec latency 13.582 ms oleg429-client.virtnet: 1 115115 1.83 MB/sec execute 292 sec latency 12.439 ms oleg429-client.virtnet: 1 115841 1.83 MB/sec execute 293 sec latency 8.908 ms oleg429-client.virtnet: 1 116311 1.84 MB/sec execute 294 sec latency 18.163 ms oleg429-client.virtnet: 1 117216 1.86 MB/sec execute 295 sec latency 18.054 ms oleg429-client.virtnet: 1 117541 1.85 MB/sec execute 296 sec latency 15.214 ms oleg429-client.virtnet: 1 117870 1.85 MB/sec execute 297 sec latency 18.049 ms oleg429-client.virtnet: 1 118523 1.86 MB/sec execute 298 sec latency 12.837 ms oleg429-client.virtnet: 1 118867 1.85 MB/sec execute 299 sec latency 15.999 ms oleg429-client.virtnet: 1 cleanup 300 sec oleg429-client.virtnet: 0 cleanup 300 sec oleg429-client.virtnet: oleg429-client.virtnet: Operation Count AvgLat MaxLat oleg429-client.virtnet: ---------------------------------------- oleg429-client.virtnet: NTCreateX 17804 9.277 19058.570 oleg429-client.virtnet: Close 13085 1.153 5.569 oleg429-client.virtnet: Rename 754 9.172 19.825 oleg429-client.virtnet: Unlink 3597 2.020 8.010 oleg429-client.virtnet: Qpathinfo 16139 4.470 19518.452 oleg429-client.virtnet: Qfileinfo 2813 0.149 2.503 oleg429-client.virtnet: Qfsinfo 2958 0.231 4.011 oleg429-client.virtnet: Sfileinfo 1450 5.197 15.124 oleg429-client.virtnet: Find 6237 1.188 4765.624 oleg429-client.virtnet: WriteX 8791 0.924 10.525 oleg429-client.virtnet: ReadX 27927 0.019 1.633 oleg429-client.virtnet: LockX 58 1.178 2.647 oleg429-client.virtnet: UnlockX 58 1.227 2.577 oleg429-client.virtnet: Flush 1247 6.353 406.395 oleg429-client.virtnet: oleg429-client.virtnet: Throughput 1.85338 MB/sec 1 clients 1 procs max_latency=19518.467 ms oleg429-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg429-client.virtnet at Tue Aug 27 12:44:54 EDT 2024 with return code 0 oleg429-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg429-client.virtnet oleg429-client.virtnet: /mnt/lustre/d70b.replay-single/oleg429-client.virtnet /mnt/lustre/d70b.replay-single/oleg429-client.virtnet oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg429-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg429-client.virtnet: removed directory: 'clients/client0' oleg429-client.virtnet: removed directory: 'clients' oleg429-client.virtnet: removed 'client.txt' oleg429-client.virtnet: /mnt/lustre/d70b.replay-single/oleg429-client.virtnet oleg429-client.virtnet: dbench successfully finished