== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 00:22:10 (1721881330) Starting client oleg155-client.virtnet: -o user_xattr,flock oleg155-server@tcp:/lustre /mnt/lustre Started clients oleg155-client.virtnet: 192.168.201.155@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg155-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=3958 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3716096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3687424 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 35840 7403520 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:22:16 (1721881336) shut down Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:22:29 (1721881349) targets are mounted 00:22:29 (1721881349) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3679232 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 46080 7414784 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:22:39 (1721881359) shut down oleg155-client.virtnet: looking for dbench program oleg155-client.virtnet: /usr/bin/dbench oleg155-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg155-client.virtnet oleg155-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg155-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg155-client.virtnet' oleg155-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg155-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg155-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg155-client.virtnet at Thu Jul 25 00:22:11 EDT 2024 oleg155-client.virtnet: waiting for dbench pid 3982 oleg155-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg155-client.virtnet: oleg155-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg155-client.virtnet: failed to create barrier semaphore oleg155-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg155-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg155-client.virtnet: releasing clients oleg155-client.virtnet: 1 462 16.17 MB/sec warmup 1 sec latency 17.212 ms oleg155-client.virtnet: 1 789 10.73 MB/sec warmup 2 sec latency 87.242 ms oleg155-client.virtnet: 1 1439 8.67 MB/sec warmup 3 sec latency 24.764 ms oleg155-client.virtnet: 1 2069 6.66 MB/sec warmup 4 sec latency 23.922 ms oleg155-client.virtnet: 1 2753 6.39 MB/sec warmup 5 sec latency 135.526 ms oleg155-client.virtnet: 1 2753 5.33 MB/sec warmup 6 sec latency 1135.674 ms oleg155-client.virtnet: 1 2753 4.56 MB/sec warmup 7 sec latency 2135.890 ms oleg155-client.virtnet: 1 2753 3.99 MB/sec warmup 8 sec latency 3136.113 ms oleg155-client.virtnet: 1 2753 3.55 MB/sec warmup 9 sec latency 4136.311 ms oleg155-client.virtnet: 1 2753 3.20 MB/sec warmup 10 sec latency 5136.457 ms oleg155-client.virtnet: 1 2753 2.90 MB/sec warmup 11 sec latency 6136.643 ms oleg155-client.virtnet: 1 2753 2.66 MB/sec warmup 12 sec latency 7136.841 ms oleg155-client.virtnet: 1 2753 2.46 MB/sec warmup 13 sec latency 8136.991 ms oleg155-client.virtnet: 1 2753 2.28 MB/sec warmup 14 sec latency 9137.178 ms oleg155-client.virtnet: 1 2753 2.13 MB/sec warmup 15 sec latency 10137.335 ms oleg155-client.virtnet: 1 2753 2.00 MB/sec warmup 16 sec latency 11137.455 ms oleg155-client.virtnet: 1 2753 1.88 MB/sec warmup 17 sec latency 12137.563 ms oleg155-client.virtnet: 1 2753 1.78 MB/sec warmup 18 sec latency 13137.726 ms oleg155-client.virtnet: 1 2753 1.68 MB/sec warmup 19 sec latency 14137.873 ms oleg155-client.virtnet: 1 2753 1.60 MB/sec warmup 20 sec latency 15138.025 ms oleg155-client.virtnet: 1 2754 1.52 MB/sec warmup 21 sec latency 16135.531 ms oleg155-client.virtnet: 1 3688 1.76 MB/sec warmup 22 sec latency 27.028 ms oleg155-client.virtnet: 1 4179 1.74 MB/sec warmup 23 sec latency 12.493 ms oleg155-client.virtnet: 1 4618 1.75 MB/sec warmup 24 sec latency 45.765 ms oleg155-client.virtnet: 1 5208 1.81 MB/sec warmup 25 sec latency 36.392 ms oleg155-client.virtnet: 1 5905 1.81 MB/sec warmup 26 sec latency 21.800 ms oleg155-client.virtnet: 1 6762 1.98 MB/sec warmup 27 sec latency 31.807 ms oleg155-client.virtnet: 1 6762 1.91 MB/sec warmup 28 sec latency 1031.969 ms oleg155-client.virtnet: 1 6762 1.84 MB/sec warmup 29 sec latency 2032.160 ms oleg155-client.virtnet: 1 6762 1.78 MB/sec warmup 30 sec latency 3032.305 ms oleg155-client.virtnet: 1 6762 1.72 MB/sec warmup 31 sec latency 4032.493 ms oleg155-client.virtnet: 1 6762 1.67 MB/sec warmup 32 sec latency 5032.688 msFailover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:22:51 (1721881371) targets are mounted 00:22:51 (1721881371) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3718144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3707904 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7426048 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:23:10 (1721881390) shut down Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:23:23 (1721881403) targets are mounted 00:23:23 (1721881403) facet_failover done oleg155-client.virtnet: 1 6762 1.62 MB/sec warmup 33 sec latency 6032.968 ms oleg155-client.virtnet: 1 6762 1.57 MB/sec warmup 34 sec latency 7033.153 ms oleg155-client.virtnet: 1 6762 1.53 MB/sec warmup 35 sec latency 8033.320 ms oleg155-client.virtnet: 1 6762 1.48 MB/sec warmup 36 sec latency 9033.656 ms oleg155-client.virtnet: 1 6762 1.44 MB/sec warmup 37 sec latency 10034.019 ms oleg155-client.virtnet: 1 6762 1.41 MB/sec warmup 38 sec latency 11034.222 ms oleg155-client.virtnet: 1 6762 1.37 MB/sec warmup 39 sec latency 12034.415 ms oleg155-client.virtnet: 1 6762 1.33 MB/sec warmup 40 sec latency 13034.565 ms oleg155-client.virtnet: 1 6762 1.30 MB/sec warmup 41 sec latency 14034.777 ms oleg155-client.virtnet: 1 6762 1.27 MB/sec warmup 42 sec latency 15035.001 ms oleg155-client.virtnet: 1 6762 1.24 MB/sec warmup 43 sec latency 16035.242 ms oleg155-client.virtnet: 1 6762 1.21 MB/sec warmup 44 sec latency 17035.471 ms oleg155-client.virtnet: 1 6762 1.19 MB/sec warmup 45 sec latency 18035.627 ms oleg155-client.virtnet: 1 6762 1.16 MB/sec warmup 46 sec latency 19035.786 ms oleg155-client.virtnet: 1 7199 1.23 MB/sec warmup 47 sec latency 19289.672 ms oleg155-client.virtnet: 1 7471 1.23 MB/sec warmup 48 sec latency 15.865 ms oleg155-client.virtnet: 1 7775 1.21 MB/sec warmup 49 sec latency 16.686 ms oleg155-client.virtnet: 1 8194 1.22 MB/sec warmup 50 sec latency 44.451 ms oleg155-client.virtnet: 1 8819 1.26 MB/sec warmup 51 sec latency 31.186 ms oleg155-client.virtnet: 1 9440 1.27 MB/sec warmup 52 sec latency 20.400 ms oleg155-client.virtnet: 1 9873 1.32 MB/sec warmup 53 sec latency 388.176 ms oleg155-client.virtnet: 1 9873 1.30 MB/sec warmup 54 sec latency 1388.335 ms oleg155-client.virtnet: 1 9873 1.28 MB/sec warmup 55 sec latency 2388.492 ms oleg155-client.virtnet: 1 9873 1.25 MB/sec warmup 56 sec latency 3388.667 ms oleg155-client.virtnet: 1 9873 1.23 MB/sec warmup 57 sec latency 4388.833 ms oleg155-client.virtnet: 1 9873 1.21 MB/sec warmup 58 sec latency 5388.980 ms oleg155-client.virtnet: 1 9873 1.19 MB/sec warmup 59 sec latency 6389.108 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 1 sec latency 8389.340 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 2 sec latency 9389.501 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 3 sec latency 10389.667 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 4 sec latency 11389.834 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 5 sec latency 12389.982 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 6 sec latency 13390.131 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 7 sec latency 14390.320 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 8 sec latency 15390.458 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 9 sec latency 16390.604 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 10 sec latency 17390.714 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 11 sec latency 18390.882 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 12 sec latency 19391.048 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 13 sec latency 20391.234 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 14 sec latency 21391.443 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 15 sec latency 22391.609 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 16 sec latency 23391.770 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 17 sec latency 24391.910 ms oleg155-client.oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3718144 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7454720 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:23:39 (1721881419) shut down Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:23:51 (1721881431) targets are mounted 00:23:51 (1721881431) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3699712 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7436288 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:24:04 (1721881444) shut down virtnet: 1 9873 0.00 MB/sec execute 18 sec latency 25392.104 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 19 sec latency 26392.307 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 20 sec latency 27392.491 ms oleg155-client.virtnet: 1 9873 0.00 MB/sec execute 21 sec latency 28392.599 ms oleg155-client.virtnet: 1 10769 0.31 MB/sec execute 22 sec latency 28444.227 ms oleg155-client.virtnet: 1 11273 0.35 MB/sec execute 23 sec latency 9.826 ms oleg155-client.virtnet: 1 11653 0.38 MB/sec execute 24 sec latency 37.857 ms oleg155-client.virtnet: 1 12323 0.53 MB/sec execute 25 sec latency 30.782 ms oleg155-client.virtnet: 1 12999 0.58 MB/sec execute 26 sec latency 19.685 ms oleg155-client.virtnet: 1 13833 0.79 MB/sec execute 27 sec latency 43.142 ms oleg155-client.virtnet: 1 13833 0.76 MB/sec execute 28 sec latency 1043.321 ms oleg155-client.virtnet: 1 13833 0.73 MB/sec execute 29 sec latency 2043.528 ms oleg155-client.virtnet: 1 13833 0.71 MB/sec execute 30 sec latency 3043.762 ms oleg155-client.virtnet: 1 13833 0.69 MB/sec execute 31 sec latency 4043.988 ms oleg155-client.virtnet: 1 13833 0.66 MB/sec execute 32 sec latency 5044.219 ms oleg155-client.virtnet: 1 13833 0.64 MB/sec execute 33 sec latency 6044.404 ms oleg155-client.virtnet: 1 13833 0.63 MB/sec execute 34 sec latency 7044.560 ms oleg155-client.virtnet: 1 13833 0.61 MB/sec execute 35 sec latency 8044.740 ms oleg155-client.virtnet: 1 13833 0.59 MB/sec execute 36 sec latency 9044.954 ms oleg155-client.virtnet: 1 13833 0.58 MB/sec execute 37 sec latency 10045.061 ms oleg155-client.virtnet: 1 13833 0.56 MB/sec execute 38 sec latency 11045.251 ms oleg155-client.virtnet: 1 13833 0.55 MB/sec execute 39 sec latency 12045.391 ms oleg155-client.virtnet: 1 13833 0.53 MB/sec execute 40 sec latency 13045.493 ms oleg155-client.virtnet: 1 13833 0.52 MB/sec execute 41 sec latency 14045.662 ms oleg155-client.virtnet: 1 13833 0.51 MB/sec execute 42 sec latency 15045.838 ms oleg155-client.virtnet: 1 13833 0.49 MB/sec execute 43 sec latency 16045.999 ms oleg155-client.virtnet: 1 13833 0.48 MB/sec execute 44 sec latency 17046.142 ms oleg155-client.virtnet: 1 13833 0.47 MB/sec execute 45 sec latency 18046.319 ms oleg155-client.virtnet: 1 13833 0.46 MB/sec execute 46 sec latency 19046.434 ms oleg155-client.virtnet: 1 14468 0.57 MB/sec execute 47 sec latency 19162.060 ms oleg155-client.virtnet: 1 14940 0.57 MB/sec execute 48 sec latency 31.248 ms oleg155-client.virtnet: 1 15469 0.60 MB/sec execute 49 sec latency 23.621 ms oleg155-client.virtnet: 1 16089 0.65 MB/sec execute 50 sec latency 27.144 ms oleg155-client.virtnet: 1 16797 0.69 MB/sec execute 51 sec latency 22.957 ms oleg155-client.virtnet: 1 17850 0.87 MB/sec execute 52 sec latency 20.753 ms oleg155-client.virtnet: 1 17850 0.85 MB/sec execute 53 sec latency 1018.588 ms oleg155-client.virtnet: 1 17850 0.83 MB/sec execute 54 sec latency 2018.733 ms oleg155-client.virtnet: 1 17850 0.82 MB/sec execute 55 sec latency 3018.903 ms oleg155-client.virtnet: 1 17850 0.80 MB/sec execute 56 sec latency 4019.060 ms oleg155-client.virtnet: 1 17850 0.79 MB/sec execute 57 sec latency 5019.202 ms oleg155-client.virtnet: 1 17850 0.78 MB/sec execute 58 sec latency 6019.347 ms oleg155-client.virtnet: 1 17850 0.76 MB/sec execute 59 sec latency 7019.490 ms oleg155-client.virtnet: 1 17850 0.75 MB/sec execute 60 sec latency 8019.659 ms oleg155-client.virtnet: 1 17850 0.74 MB/sec execute 61 sec latency 9019.820 ms oleg155-client.virtnet: Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:24:16 (1721881456) targets are mounted 00:24:16 (1721881456) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3742720 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3682304 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7425024 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:24:29 (1721881469) shut down Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:24:41 (1721881481) targets are mounted 00:24:41 (1721881481) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3748864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 44032 3725312 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7474176 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:24:54 (1721881494) shut down 1 17850 0.73 MB/sec execute 62 sec latency 10019.944 ms oleg155-client.virtnet: 1 17850 0.71 MB/sec execute 63 sec latency 11020.095 ms oleg155-client.virtnet: 1 17850 0.70 MB/sec execute 64 sec latency 12020.259 ms oleg155-client.virtnet: 1 17850 0.69 MB/sec execute 65 sec latency 13020.421 ms oleg155-client.virtnet: 1 17850 0.68 MB/sec execute 66 sec latency 14020.524 ms oleg155-client.virtnet: 1 17850 0.67 MB/sec execute 67 sec latency 15020.633 ms oleg155-client.virtnet: 1 17850 0.66 MB/sec execute 68 sec latency 16020.790 ms oleg155-client.virtnet: 1 17850 0.65 MB/sec execute 69 sec latency 17020.915 ms oleg155-client.virtnet: 1 17850 0.64 MB/sec execute 70 sec latency 18021.050 ms oleg155-client.virtnet: 1 17850 0.63 MB/sec execute 71 sec latency 19021.170 ms oleg155-client.virtnet: 1 18232 0.64 MB/sec execute 72 sec latency 19228.693 ms oleg155-client.virtnet: 1 18667 0.65 MB/sec execute 73 sec latency 36.299 ms oleg155-client.virtnet: 1 19322 0.70 MB/sec execute 74 sec latency 31.825 ms oleg155-client.virtnet: 1 19932 0.70 MB/sec execute 75 sec latency 22.208 ms oleg155-client.virtnet: 1 20759 0.78 MB/sec execute 76 sec latency 23.558 ms oleg155-client.virtnet: 1 21609 0.85 MB/sec execute 77 sec latency 18.753 ms oleg155-client.virtnet: 1 21630 0.84 MB/sec execute 78 sec latency 952.194 ms oleg155-client.virtnet: 1 21630 0.83 MB/sec execute 79 sec latency 1952.353 ms oleg155-client.virtnet: 1 21630 0.82 MB/sec execute 80 sec latency 2952.520 ms oleg155-client.virtnet: 1 21630 0.81 MB/sec execute 81 sec latency 3952.643 ms oleg155-client.virtnet: 1 21630 0.80 MB/sec execute 82 sec latency 4952.873 ms oleg155-client.virtnet: 1 21630 0.79 MB/sec execute 83 sec latency 5953.050 ms oleg155-client.virtnet: 1 21630 0.78 MB/sec execute 84 sec latency 6953.261 ms oleg155-client.virtnet: 1 21630 0.77 MB/sec execute 85 sec latency 7953.413 ms oleg155-client.virtnet: 1 21630 0.76 MB/sec execute 86 sec latency 8953.523 ms oleg155-client.virtnet: 1 21630 0.75 MB/sec execute 87 sec latency 9953.712 ms oleg155-client.virtnet: 1 21630 0.74 MB/sec execute 88 sec latency 10953.857 ms oleg155-client.virtnet: 1 21630 0.73 MB/sec execute 89 sec latency 11953.995 ms oleg155-client.virtnet: 1 21630 0.72 MB/sec execute 90 sec latency 12954.140 ms oleg155-client.virtnet: 1 21630 0.72 MB/sec execute 91 sec latency 13954.277 ms oleg155-client.virtnet: 1 21630 0.71 MB/sec execute 92 sec latency 14954.445 ms oleg155-client.virtnet: 1 21630 0.70 MB/sec execute 93 sec latency 15954.638 ms oleg155-client.virtnet: 1 21630 0.69 MB/sec execute 94 sec latency 16954.795 ms oleg155-client.virtnet: 1 21630 0.69 MB/sec execute 95 sec latency 17954.983 ms oleg155-client.virtnet: 1 21630 0.68 MB/sec execute 96 sec latency 18955.121 ms oleg155-client.virtnet: 1 22001 0.68 MB/sec execute 97 sec latency 19229.930 ms oleg155-client.virtnet: 1 22464 0.69 MB/sec execute 98 sec latency 35.155 ms oleg155-client.virtnet: 1 23111 0.72 MB/sec execute 99 sec latency 27.272 ms oleg155-client.virtnet: 1 23767 0.73 MB/sec execute 100 sec latency 24.902 ms oleg155-client.virtnet: 1 24862 0.82 MB/sec execute 101 sec latency 19.411 ms oleg155-client.virtnet: 1 25398 0.83 MB/sec execute 102 sec latency 8.329 ms oleg155-client.virtnet: 1 25403 0.82 MB/sec execute 103 sec latency 992.581 ms oleg155-client.virtnet: 1 25403 0.81 MB/sec execute 104 sec latency 1992.780 ms oleg155-client.virtnet: 1 25403 0.81 MB/sec execute 105 sec latency 2992.931 ms oleg155-client.virtnet: 1 Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:25:06 (1721881506) targets are mounted 00:25:06 (1721881506) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3733504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3712000 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7445504 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:25:19 (1721881519) shut down Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:25:31 (1721881531) targets are mounted 00:25:31 (1721881531) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3739648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 41984 3707904 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7447552 1% /mnt/lustre 25403 0.80 MB/sec execute 106 sec latency 3993.120 ms oleg155-client.virtnet: 1 25403 0.79 MB/sec execute 107 sec latency 4993.271 ms oleg155-client.virtnet: 1 25403 0.78 MB/sec execute 108 sec latency 5993.427 ms oleg155-client.virtnet: 1 25403 0.78 MB/sec execute 109 sec latency 6993.544 ms oleg155-client.virtnet: 1 25403 0.77 MB/sec execute 110 sec latency 7993.711 ms oleg155-client.virtnet: 1 25403 0.76 MB/sec execute 111 sec latency 8993.870 ms oleg155-client.virtnet: 1 25403 0.76 MB/sec execute 112 sec latency 9994.072 ms oleg155-client.virtnet: 1 25403 0.75 MB/sec execute 113 sec latency 10994.239 ms oleg155-client.virtnet: 1 25403 0.74 MB/sec execute 114 sec latency 11994.349 ms oleg155-client.virtnet: 1 25403 0.74 MB/sec execute 115 sec latency 12994.482 ms oleg155-client.virtnet: 1 25403 0.73 MB/sec execute 116 sec latency 13994.621 ms oleg155-client.virtnet: 1 25403 0.72 MB/sec execute 117 sec latency 14994.755 ms oleg155-client.virtnet: 1 25403 0.72 MB/sec execute 118 sec latency 15994.946 ms oleg155-client.virtnet: 1 25403 0.71 MB/sec execute 119 sec latency 16995.099 ms oleg155-client.virtnet: 1 25403 0.71 MB/sec execute 120 sec latency 17995.264 ms oleg155-client.virtnet: 1 25403 0.70 MB/sec execute 121 sec latency 18995.457 ms oleg155-client.virtnet: 1 25680 0.70 MB/sec execute 122 sec latency 19334.179 ms oleg155-client.virtnet: 1 26299 0.73 MB/sec execute 123 sec latency 30.385 ms oleg155-client.virtnet: 1 26863 0.73 MB/sec execute 124 sec latency 28.245 ms oleg155-client.virtnet: 1 27633 0.77 MB/sec execute 125 sec latency 21.652 ms oleg155-client.virtnet: 1 28562 0.81 MB/sec execute 126 sec latency 28.142 ms oleg155-client.virtnet: 1 29098 0.82 MB/sec execute 127 sec latency 43.040 ms oleg155-client.virtnet: 1 29159 0.81 MB/sec execute 128 sec latency 794.918 ms oleg155-client.virtnet: 1 29159 0.81 MB/sec execute 129 sec latency 1795.108 ms oleg155-client.virtnet: 1 29159 0.80 MB/sec execute 130 sec latency 2795.271 ms oleg155-client.virtnet: 1 29159 0.79 MB/sec execute 131 sec latency 3795.554 ms oleg155-client.virtnet: 1 29159 0.79 MB/sec execute 132 sec latency 4795.674 ms oleg155-client.virtnet: 1 29159 0.78 MB/sec execute 133 sec latency 5795.796 ms oleg155-client.virtnet: 1 29159 0.78 MB/sec execute 134 sec latency 6795.930 ms oleg155-client.virtnet: 1 29159 0.77 MB/sec execute 135 sec latency 7796.066 ms oleg155-client.virtnet: 1 29159 0.77 MB/sec execute 136 sec latency 8796.196 ms oleg155-client.virtnet: 1 29159 0.76 MB/sec execute 137 sec latency 9796.335 ms oleg155-client.virtnet: 1 29159 0.75 MB/sec execute 138 sec latency 10796.445 ms oleg155-client.virtnet: 1 29159 0.75 MB/sec execute 139 sec latency 11796.558 ms oleg155-client.virtnet: 1 29159 0.74 MB/sec execute 140 sec latency 12796.697 ms oleg155-client.virtnet: 1 29159 0.74 MB/sec execute 141 sec latency 13796.804 ms oleg155-client.virtnet: 1 29159 0.73 MB/sec execute 142 sec latency 14796.900 ms oleg155-client.virtnet: 1 29159 0.73 MB/sec execute 143 sec latency 15797.021 ms oleg155-client.virtnet: 1 29159 0.72 MB/sec execute 144 sec latency 16797.135 ms oleg155-client.virtnet: 1 29159 0.72 MB/sec execute 145 sec latency 17797.275 ms oleg155-client.virtnet: 1 29159 0.71 MB/sec execute 146 sec latency 18797.458 ms oleg155-client.virtnet: 1 29412 0.71 MB/sec execute 147 sec latency 19169.373 ms oleg155-client.virtnet: 1 30074 0.74 MB/sec execute 148 sec latency 35.374 ms oleg155-client.virtnet: 1 30764 0.74 MB/sec execute 149 sec latency 19.098 ms oleg155-client.virtnet: 1 test_70b fail mds1 9 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:25:44 (1721881544) shut down Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:25:56 (1721881556) targets are mounted 00:25:56 (1721881556) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3705856 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7449600 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:26:09 (1721881569) shut down Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:26:21 (1721881581) targets are mounted 00:26:21 (1721881581) facet_failover done 31647 0.78 MB/sec execute 150 sec latency 27.161 ms oleg155-client.virtnet: 1 32358 0.81 MB/sec execute 151 sec latency 11.759 ms oleg155-client.virtnet: 1 32848 0.81 MB/sec execute 152 sec latency 30.125 ms oleg155-client.virtnet: 1 32954 0.81 MB/sec execute 153 sec latency 750.416 ms oleg155-client.virtnet: 1 32954 0.80 MB/sec execute 154 sec latency 1750.570 ms oleg155-client.virtnet: 1 32954 0.80 MB/sec execute 155 sec latency 2750.726 ms oleg155-client.virtnet: 1 32954 0.79 MB/sec execute 156 sec latency 3750.886 ms oleg155-client.virtnet: 1 32954 0.79 MB/sec execute 157 sec latency 4751.034 ms oleg155-client.virtnet: 1 32954 0.78 MB/sec execute 158 sec latency 5751.212 ms oleg155-client.virtnet: 1 32954 0.78 MB/sec execute 159 sec latency 6751.359 ms oleg155-client.virtnet: 1 32954 0.77 MB/sec execute 160 sec latency 7751.550 ms oleg155-client.virtnet: 1 32954 0.77 MB/sec execute 161 sec latency 8751.700 ms oleg155-client.virtnet: 1 32954 0.76 MB/sec execute 162 sec latency 9751.856 ms oleg155-client.virtnet: 1 32954 0.76 MB/sec execute 163 sec latency 10751.985 ms oleg155-client.virtnet: 1 32954 0.76 MB/sec execute 164 sec latency 11752.096 ms oleg155-client.virtnet: 1 32954 0.75 MB/sec execute 165 sec latency 12752.249 ms oleg155-client.virtnet: 1 32954 0.75 MB/sec execute 166 sec latency 13752.407 ms oleg155-client.virtnet: 1 32954 0.74 MB/sec execute 167 sec latency 14752.596 ms oleg155-client.virtnet: 1 32954 0.74 MB/sec execute 168 sec latency 15752.772 ms oleg155-client.virtnet: 1 32954 0.73 MB/sec execute 169 sec latency 16752.937 ms oleg155-client.virtnet: 1 32954 0.73 MB/sec execute 170 sec latency 17753.089 ms oleg155-client.virtnet: 1 32954 0.72 MB/sec execute 171 sec latency 18753.247 ms oleg155-client.virtnet: 1 33401 0.74 MB/sec execute 172 sec latency 19151.650 ms oleg155-client.virtnet: 1 33974 0.74 MB/sec execute 173 sec latency 29.896 ms oleg155-client.virtnet: 1 34733 0.77 MB/sec execute 174 sec latency 26.384 ms oleg155-client.virtnet: 1 35667 0.80 MB/sec execute 175 sec latency 22.551 ms oleg155-client.virtnet: 1 36196 0.81 MB/sec execute 176 sec latency 37.289 ms oleg155-client.virtnet: 1 36682 0.81 MB/sec execute 177 sec latency 22.450 ms oleg155-client.virtnet: 1 36924 0.83 MB/sec execute 178 sec latency 721.002 ms oleg155-client.virtnet: 1 36924 0.82 MB/sec execute 179 sec latency 1721.100 ms oleg155-client.virtnet: 1 36924 0.82 MB/sec execute 180 sec latency 2721.256 ms oleg155-client.virtnet: 1 36924 0.81 MB/sec execute 181 sec latency 3721.415 ms oleg155-client.virtnet: 1 36924 0.81 MB/sec execute 182 sec latency 4721.573 ms oleg155-client.virtnet: 1 36924 0.80 MB/sec execute 183 sec latency 5721.765 ms oleg155-client.virtnet: 1 36924 0.80 MB/sec execute 184 sec latency 6721.880 ms oleg155-client.virtnet: 1 36924 0.79 MB/sec execute 185 sec latency 7722.038 ms oleg155-client.virtnet: 1 36924 0.79 MB/sec execute 186 sec latency 8722.186 ms oleg155-client.virtnet: 1 36924 0.79 MB/sec execute 187 sec latency 9722.340 ms oleg155-client.virtnet: 1 36924 0.78 MB/sec execute 188 sec latency 10722.462 ms oleg155-client.virtnet: 1 36924 0.78 MB/sec execute 189 sec latency 11722.551 ms oleg155-client.virtnet: 1 36924 0.77 MB/sec execute 190 sec latency 12722.670 ms oleg155-client.virtnet: 1 36924 0.77 MB/sec execute 191 sec latency 13722.847 ms oleg155-client.virtnet: 1 36924 0.77 MB/sec execute 192 sec latency 14723.002 ms oleg155-client.virtnet: 1 36924 0.76 MB/sec execute 193 sec latency 15723.105 ms oleg155-client.virtnet: 1 36oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3734528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3697664 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7432192 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:26:34 (1721881594) shut down Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:26:47 (1721881607) targets are mounted 00:26:47 (1721881607) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3737600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3715072 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7452672 1% /mnt/lustre test_70b fail mds1 12 times Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 00:26:59 (1721881619) shut down 924 0.76 MB/sec execute 194 sec latency 16723.226 ms oleg155-client.virtnet: 1 36924 0.75 MB/sec execute 195 sec latency 17723.477 ms oleg155-client.virtnet: 1 36924 0.75 MB/sec execute 196 sec latency 18723.791 ms oleg155-client.virtnet: 1 37210 0.75 MB/sec execute 197 sec latency 19152.671 ms oleg155-client.virtnet: 1 37663 0.75 MB/sec execute 198 sec latency 23.280 ms oleg155-client.virtnet: 1 38102 0.75 MB/sec execute 199 sec latency 40.897 ms oleg155-client.virtnet: 1 39114 0.80 MB/sec execute 200 sec latency 24.074 ms oleg155-client.virtnet: 1 39509 0.80 MB/sec execute 201 sec latency 12.791 ms oleg155-client.virtnet: 1 39862 0.80 MB/sec execute 202 sec latency 57.164 ms oleg155-client.virtnet: 1 40093 0.80 MB/sec execute 203 sec latency 494.839 ms oleg155-client.virtnet: 1 40093 0.80 MB/sec execute 204 sec latency 1495.067 ms oleg155-client.virtnet: 1 40093 0.80 MB/sec execute 205 sec latency 2495.321 ms oleg155-client.virtnet: 1 40093 0.79 MB/sec execute 206 sec latency 3495.593 ms oleg155-client.virtnet: 1 40093 0.79 MB/sec execute 207 sec latency 4495.832 ms oleg155-client.virtnet: 1 40093 0.78 MB/sec execute 208 sec latency 5496.079 ms oleg155-client.virtnet: 1 40093 0.78 MB/sec execute 209 sec latency 6496.281 ms oleg155-client.virtnet: 1 40093 0.78 MB/sec execute 210 sec latency 7496.467 ms oleg155-client.virtnet: 1 40093 0.77 MB/sec execute 211 sec latency 8496.667 ms oleg155-client.virtnet: 1 40093 0.77 MB/sec execute 212 sec latency 9496.762 ms oleg155-client.virtnet: 1 40093 0.77 MB/sec execute 213 sec latency 10496.953 ms oleg155-client.virtnet: 1 40093 0.76 MB/sec execute 214 sec latency 11497.099 ms oleg155-client.virtnet: 1 40093 0.76 MB/sec execute 215 sec latency 12497.260 ms oleg155-client.virtnet: 1 40093 0.75 MB/sec execute 216 sec latency 13497.457 ms oleg155-client.virtnet: 1 40093 0.75 MB/sec execute 217 sec latency 14497.597 ms oleg155-client.virtnet: 1 40093 0.75 MB/sec execute 218 sec latency 15497.754 ms oleg155-client.virtnet: 1 40093 0.74 MB/sec execute 219 sec latency 16497.939 ms oleg155-client.virtnet: 1 40093 0.74 MB/sec execute 220 sec latency 17498.099 ms oleg155-client.virtnet: 1 40093 0.74 MB/sec execute 221 sec latency 18498.268 ms oleg155-client.virtnet: 1 40537 0.75 MB/sec execute 222 sec latency 18913.866 ms oleg155-client.virtnet: 1 40872 0.75 MB/sec execute 223 sec latency 37.213 ms oleg155-client.virtnet: 1 41229 0.75 MB/sec execute 224 sec latency 44.185 ms oleg155-client.virtnet: 1 41818 0.76 MB/sec execute 225 sec latency 29.845 ms oleg155-client.virtnet: 1 42717 0.79 MB/sec execute 226 sec latency 25.313 ms oleg155-client.virtnet: 1 43274 0.79 MB/sec execute 227 sec latency 16.872 ms oleg155-client.virtnet: 1 43408 0.79 MB/sec execute 228 sec latency 574.885 ms oleg155-client.virtnet: 1 43408 0.79 MB/sec execute 229 sec latency 1575.133 ms oleg155-client.virtnet: 1 43408 0.79 MB/sec execute 230 sec latency 2575.394 ms oleg155-client.virtnet: 1 43408 0.78 MB/sec execute 231 sec latency 3575.571 ms oleg155-client.virtnet: 1 43408 0.78 MB/sec execute 232 sec latency 4575.740 ms oleg155-client.virtnet: 1 43408 0.78 MB/sec execute 233 sec latency 5575.902 ms oleg155-client.virtnet: 1 43408 0.77 MB/sec execute 234 sec latency 6576.070 ms oleg155-client.virtnet: 1 43408 0.77 MB/sec execute 235 sec latency 7576.285 ms oleg155-client.virtnet: 1 43408 0.77 MB/sec execute 236 sec latency 8576.564 ms oleg155-client.virtnet: 1 43408 0.76 MB/sec execute 237 sec latency 9576.884 ms oleg155-client.virtnet: 1 43408 0.76Failover mds1 to oleg155-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 00:27:12 (1721881632) targets are mounted 00:27:12 (1721881632) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec MB/sec execute 238 sec latency 10577.075 ms oleg155-client.virtnet: 1 43408 0.76 MB/sec execute 239 sec latency 11577.220 ms oleg155-client.virtnet: 1 43408 0.75 MB/sec execute 240 sec latency 12577.365 ms oleg155-client.virtnet: 1 43408 0.75 MB/sec execute 241 sec latency 13577.521 ms oleg155-client.virtnet: 1 43408 0.75 MB/sec execute 242 sec latency 14577.770 ms oleg155-client.virtnet: 1 43408 0.74 MB/sec execute 243 sec latency 15577.977 ms oleg155-client.virtnet: 1 43408 0.74 MB/sec execute 244 sec latency 16578.187 ms oleg155-client.virtnet: 1 43408 0.74 MB/sec execute 245 sec latency 17578.446 ms oleg155-client.virtnet: 1 43408 0.73 MB/sec execute 246 sec latency 18578.686 ms oleg155-client.virtnet: 1 43635 0.74 MB/sec execute 247 sec latency 19078.395 ms oleg155-client.virtnet: 1 44284 0.75 MB/sec execute 248 sec latency 33.679 ms oleg155-client.virtnet: 1 44952 0.75 MB/sec execute 249 sec latency 31.931 ms oleg155-client.virtnet: 1 45873 0.78 MB/sec execute 250 sec latency 22.471 ms oleg155-client.virtnet: 1 46424 0.79 MB/sec execute 251 sec latency 15.186 ms oleg155-client.virtnet: 1 46845 0.79 MB/sec execute 252 sec latency 35.905 ms oleg155-client.virtnet: 1 47332 0.80 MB/sec execute 253 sec latency 21.293 ms oleg155-client.virtnet: 1 47957 0.81 MB/sec execute 254 sec latency 37.902 ms oleg155-client.virtnet: 1 48614 0.81 MB/sec execute 255 sec latency 26.217 ms oleg155-client.virtnet: 1 49678 0.85 MB/sec execute 256 sec latency 28.320 ms oleg155-client.virtnet: 1 50214 0.85 MB/sec execute 257 sec latency 8.279 ms oleg155-client.virtnet: 1 50620 0.85 MB/sec execute 258 sec latency 40.663 ms oleg155-client.virtnet: 1 50939 0.85 MB/sec execute 259 sec latency 46.846 ms oleg155-client.virtnet: 1 51554 0.86 MB/sec execute 260 sec latency 29.250 ms oleg155-client.virtnet: 1 52194 0.87 MB/sec execute 261 sec latency 25.780 ms oleg155-client.virtnet: 1 53074 0.89 MB/sec execute 262 sec latency 33.168 ms oleg155-client.virtnet: 1 53557 0.90 MB/sec execute 263 sec latency 12.761 ms oleg155-client.virtnet: 1 53969 0.90 MB/sec execute 264 sec latency 48.532 ms oleg155-client.virtnet: 1 54296 0.90 MB/sec execute 265 sec latency 40.773 ms oleg155-client.virtnet: 1 54898 0.91 MB/sec execute 266 sec latency 35.528 ms oleg155-client.virtnet: 1 55220 0.91 MB/sec execute 267 sec latency 40.426 ms oleg155-client.virtnet: 1 55678 0.91 MB/sec execute 268 sec latency 47.077 ms oleg155-client.virtnet: 1 56217 0.93 MB/sec execute 269 sec latency 40.718 ms oleg155-client.virtnet: 1 56984 0.95 MB/sec execute 270 sec latency 40.145 ms oleg155-client.virtnet: 1 57374 0.95 MB/sec execute 271 sec latency 12.366 ms oleg155-client.virtnet: 1 57691 0.95 MB/sec execute 272 sec latency 40.334 ms oleg155-client.virtnet: 1 58273 0.96 MB/sec execute 273 sec latency 31.928 ms oleg155-client.virtnet: 1 58657 0.96 MB/sec execute 274 sec latency 21.191 ms oleg155-client.virtnet: 1 59220 0.96 MB/sec execute 275 sec latency 43.404 ms oleg155-client.virtnet: 1 59794 0.98 MB/sec execute 276 sec latency 42.587 ms oleg155-client.virtnet: 1 60472 0.99 MB/sec execute 277 sec latency 25.517 ms oleg155-client.virtnet: 1 60763 0.99 MB/sec execute 278 sec latency 16.321 ms oleg155-client.virtnet: 1 61068 0.99 MB/sec execute 279 sec latency 75.703 ms oleg155-client.virtnet: 1 61378 0.99 MB/sec execute 280 sec latency 42.329 ms oleg155-client.virtnet: 1 61791 1.00 MB/sec execute 281 sec latency 18.211 ms oleg155-client.virtnet: 1 62085 1.00 MB/sec execute 282 sec latency 46.238 ms oleg155-client.virtnet: 1 62501 0.99 MB/sec execute 283 sec latency 27.148 ms oleg155-client.virtnet: 1 63227 1.01 MB/sec execute 284 sec latency 27.953 ms oleg155-client.virtnet: 1 63898 1.03 MB/sec execute 285 sec latency 27.097 ms oleg155-client.virtnet: 1 64305 1.03 MB/sec execute 286 sec latency 12.491 ms oleg155-client.virtnet: 1 64611 1.03 MB/sec execute 287 sec latency 69.691 ms oleg155-client.virtnet: 1 64915 1.03 MB/sec execute 288 sec latency 45.975 ms oleg155-client.virtnet: 1 65302 1.04 MB/sec execute 289 sec latency 17.872 ms oleg155-client.virtnet: 1 65680 1.04 MB/sec execute 290 sec latency 44.636 ms oleg155-client.virtnet: 1 66098 1.03 MB/sec execute 291 sec latency 39.403 ms oleg155-client.virtnet: 1 66770 1.05 MB/sec execute 292 sec latency 43.319 ms oleg155-client.virtnet: 1 67621 1.07 MB/sec execute 293 sec latency 26.719 ms oleg155-client.virtnet: 1 68106 1.07 MB/sec execute 294 sec latency 30.832 ms oleg155-client.virtnet: 1 68513 1.07 MB/sec execute 295 sec latency 31.383 ms oleg155-client.virtnet: 1 69141 1.08 MB/sec execute 296 sec latency 25.975 ms oleg155-client.virtnet: 1 69797 1.08 MB/sec execute 297 sec latency 22.717 ms oleg155-client.virtnet: 1 70693 1.10 MB/sec execute 298 sec latency 29.499 ms oleg155-client.virtnet: 1 71312 1.11 MB/sec execute 299 sec latency 18.189 ms oleg155-client.virtnet: 1 cleanup 300 sec oleg155-client.virtnet: 0 cleanup 300 sec oleg155-client.virtnet: oleg155-client.virtnet: Operation Count AvgLat MaxLat oleg155-client.virtnet: ---------------------------------------- oleg155-client.virtnet: NTCreateX 10714 11.201 28444.218 oleg155-client.virtnet: Close 7830 8.358 19334.166 oleg155-client.virtnet: Rename 453 7.910 19.473 oleg155-client.virtnet: Unlink 2195 1.937 10.327 oleg155-client.virtnet: Qpathinfo 9680 3.385 18913.853 oleg155-client.virtnet: Qfileinfo 1682 0.129 2.315 oleg155-client.virtnet: Qfsinfo 1797 0.096 3.493 oleg155-client.virtnet: Sfileinfo 857 49.080 19169.361 oleg155-client.virtnet: Find 3754 0.411 13.464 oleg155-client.virtnet: WriteX 5277 0.710 5.162 oleg155-client.virtnet: ReadX 16831 0.017 1.124 oleg155-client.virtnet: LockX 36 0.867 1.763 oleg155-client.virtnet: UnlockX 36 0.895 1.918 oleg155-client.virtnet: Flush 754 43.663 19151.612 oleg155-client.virtnet: oleg155-client.virtnet: Throughput 1.11422 MB/sec 1 clients 1 procs max_latency=28444.227 ms oleg155-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg155-client.virtnet at Thu Jul 25 00:28:11 EDT 2024 with return code 0 oleg155-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg155-client.virtnet oleg155-client.virtnet: /mnt/lustre/d70b.replay-single/oleg155-client.virtnet /mnt/lustre/d70b.replay-single/oleg155-client.virtnet oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg155-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg155-client.virtnet: removed directory: 'clients/client0' oleg155-client.virtnet: removed directory: 'clients' oleg155-client.virtnet: removed 'client.txt' oleg155-client.virtnet: /mnt/lustre/d70b.replay-single/oleg155-client.virtnet oleg155-client.virtnet: dbench successfully finished