== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 19:02:56 (1718319776) Starting client oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre Started clients oleg140-client.virtnet: 192.168.201.140@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg140-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=11681 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2128 1285560 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1976 3592960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26144 3556156 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28120 7149116 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg140-server Stopping /mnt/lustre-mds1 (opts:) on oleg140-server 19:03:03 (1718319783) shut down Failover mds1 to oleg140-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 19:03:17 (1718319797) targets are mounted 19:03:17 (1718319797) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2396 1285292 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2136 1285552 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1752 3592056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26524 3569728 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28276 7161784 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg140-server Stopping /mnt/lustre-mds2 (opts:) on oleg140-server 19:03:28 (1718319808) shut down oleg140-client.virtnet: looking for dbench program oleg140-client.virtnet: /usr/bin/dbench oleg140-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg140-client.virtnet oleg140-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg140-client.virtnet' oleg140-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg140-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg140-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg140-client.virtnet at Thu Jun 13 19:02:57 EDT 2024 oleg140-client.virtnet: waiting for dbench pid 11706 oleg140-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg140-client.virtnet: oleg140-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg140-client.virtnet: failed to create barrier semaphore oleg140-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg140-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg140-client.virtnet: releasing clients oleg140-client.virtnet: 1 616 20.98 MB/sec warmup 1 sec latency 12.195 ms oleg140-client.virtnet: 1 1251 11.54 MB/sec warmup 2 sec latency 9.633 ms oleg140-client.virtnet: 1 2010 8.84 MB/sec warmup 3 sec latency 8.408 ms oleg140-client.virtnet: 1 2979 8.38 MB/sec warmup 4 sec latency 8.778 ms oleg140-client.virtnet: 1 3846 7.95 MB/sec warmup 5 sec latency 12.994 ms oleg140-client.virtnet: 1 4411 6.78 MB/sec warmup 6 sec latency 10.693 ms oleg140-client.virtnet: 1 5180 6.47 MB/sec warmup 7 sec latency 7.654 ms oleg140-client.virtnet: 1 5966 5.89 MB/sec warmup 8 sec latency 12.006 ms oleg140-client.virtnet: 1 7174 6.41 MB/sec warmup 9 sec latency 7.721 ms oleg140-client.virtnet: 1 7777 5.94 MB/sec warmup 10 sec latency 8.995 ms oleg140-client.virtnet: 1 8527 5.84 MB/sec warmup 11 sec latency 9.052 ms oleg140-client.virtnet: 1 9268 5.41 MB/sec warmup 12 sec latency 8.582 ms oleg140-client.virtnet: 1 10001 5.47 MB/sec warmup 13 sec latency 14.101 ms oleg140-client.virtnet: 1 10768 5.50 MB/sec warmup 14 sec latency 14.410 ms oleg140-client.virtnet: 1 11180 5.22 MB/sec warmup 15 sec latency 14.583 ms oleg140-client.virtnet: 1 11729 5.02 MB/sec warmup 16 sec latency 14.386 ms oleg140-client.virtnet: 1 12380 4.92 MB/sec warmup 17 sec latency 9.470 ms oleg140-client.virtnet: 1 13150 4.78 MB/sec warmup 18 sec latency 8.089 ms oleg140-client.virtnet: 1 14328 5.06 MB/sec warmup 19 sec latency 9.316 ms oleg140-client.virtnet: 1 14886 4.88 MB/sec warmup 20 sec latency 11.002 ms oleg140-client.virtnet: 1 15331 4.74 MB/sec warmup 21 sec latency 13.951 ms oleg140-client.virtnet: 1 16000 4.67 MB/sec warmup 22 sec latency 14.443 ms oleg140-client.virtnet: 1 16805 4.59 MB/sec warmup 23 sec latency 8.180 ms oleg140-client.virtnet: 1 17975 4.84 MB/sec warmup 24 sec latency 8.649 ms oleg140-client.virtnet: 1 18556 4.68 MB/sec warmup 25 sec latency 11.825 ms oleg140-client.virtnet: 1 19308 4.68 MB/sec warmup 26 sec latency 9.149 ms oleg140-client.virtnet: 1 20112 4.57 MB/sec warmup 27 sec latency 9.012 ms oleg140-client.virtnet: 1 21157 4.68 MB/sec warmup 28 sec latency 8.657 ms oleg140-client.virtnet: 1 21779 4.68 MB/sec warmup 29 sec latency 12.384 ms oleg140-client.virtnet: 1 22418 4.59 MB/sec warmup 30 sec latency 9.011 ms oleg140-client.virtnet: 1 22843 4.54 MB/sec warmup 31 sec latency 473.978 ms oleg140-client.virtnet: 1 22843 4.40 MB/sec warmup 32 sec latency 1474.137 ms oleg140-client.virtnet: 1 22843 4.27 MB/sec warmup 33 sec latency 2474.350 ms oleg140-client.virtnet: 1 22843 4.14 MB/sFailover mds2 to oleg140-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 19:03:42 (1718319822) targets are mounted 19:03:42 (1718319822) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1285660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13108 3590944 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36512 3569976 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49620 7160920 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg140-server Stopping /mnt/lustre-mds1 (opts:) on oleg140-server 19:03:53 (1718319833) shut down Failover mds1 to oleg140-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 19:04:08 (1718319848) targets are mounted 19:04:08 (1718319848) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13496 3592636 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36388 3570044 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49884 7162680 1% /mnt/lustre ec warmup 34 sec latency 3474.568 ms oleg140-client.virtnet: 1 22843 4.02 MB/sec warmup 35 sec latency 4474.810 ms oleg140-client.virtnet: 1 22843 3.91 MB/sec warmup 36 sec latency 5475.114 ms oleg140-client.virtnet: 1 22843 3.81 MB/sec warmup 37 sec latency 6475.347 ms oleg140-client.virtnet: 1 22843 3.71 MB/sec warmup 38 sec latency 7475.546 ms oleg140-client.virtnet: 1 22843 3.61 MB/sec warmup 39 sec latency 8475.689 ms oleg140-client.virtnet: 1 22843 3.52 MB/sec warmup 40 sec latency 9475.898 ms oleg140-client.virtnet: 1 22843 3.43 MB/sec warmup 41 sec latency 10476.043 ms oleg140-client.virtnet: 1 22843 3.35 MB/sec warmup 42 sec latency 11476.211 ms oleg140-client.virtnet: 1 22843 3.28 MB/sec warmup 43 sec latency 12476.343 ms oleg140-client.virtnet: 1 22843 3.20 MB/sec warmup 44 sec latency 13476.469 ms oleg140-client.virtnet: 1 22843 3.13 MB/sec warmup 45 sec latency 14476.633 ms oleg140-client.virtnet: 1 22843 3.06 MB/sec warmup 46 sec latency 15476.867 ms oleg140-client.virtnet: 1 22843 3.00 MB/sec warmup 47 sec latency 16477.062 ms oleg140-client.virtnet: 1 22843 2.93 MB/sec warmup 48 sec latency 17477.258 ms oleg140-client.virtnet: 1 22908 2.88 MB/sec warmup 49 sec latency 18347.922 ms oleg140-client.virtnet: 1 23659 2.85 MB/sec warmup 50 sec latency 8.437 ms oleg140-client.virtnet: 1 24346 2.91 MB/sec warmup 51 sec latency 15.218 ms oleg140-client.virtnet: 1 25112 2.97 MB/sec warmup 52 sec latency 13.085 ms oleg140-client.virtnet: 1 25665 2.93 MB/sec warmup 53 sec latency 9.582 ms oleg140-client.virtnet: 1 26356 2.96 MB/sec warmup 54 sec latency 9.700 ms oleg140-client.virtnet: 1 27048 2.92 MB/sec warmup 55 sec latency 9.927 ms oleg140-client.virtnet: 1 28103 3.00 MB/sec warmup 56 sec latency 8.839 ms oleg140-client.virtnet: 1 28798 3.05 MB/sec warmup 57 sec latency 9.689 ms oleg140-client.virtnet: 1 29163 3.01 MB/sec warmup 58 sec latency 16.994 ms oleg140-client.virtnet: 1 29551 2.98 MB/sec warmup 59 sec latency 16.085 ms oleg140-client.virtnet: 1 30624 0.61 MB/sec execute 1 sec latency 15.669 ms oleg140-client.virtnet: 1 31776 4.65 MB/sec execute 2 sec latency 8.173 ms oleg140-client.virtnet: 1 32412 4.64 MB/sec execute 3 sec latency 9.958 ms oleg140-client.virtnet: 1 32791 3.61 MB/sec execute 4 sec latency 18.550 ms oleg140-client.virtnet: 1 33144 3.20 MB/sec execute 5 sec latency 15.388 ms oleg140-client.virtnet: 1 33579 3.19 MB/sec execute 6 sec latency 24.005 ms oleg140-client.virtnet: 1 33954 2.78 MB/sec execute 7 sec latency 17.067 ms oleg140-client.virtnet: 1 34733 3.12 MB/sec execute 8 sec latency 11.307 ms oleg140-client.virtnet: 1 35751 3.64 MB/sec execute 9 sec latency 9.793 ms oleg140-client.virtnet: 1 36291 3.34 MB/sec execute 10 sec latency 9.656 ms oleg140-client.virtnet: 1 36902 3.46 MB/sec execute 11 sec latency 14.119 ms oleg140-client.virtnet: 1 37346 3.20 MB/sec execute 12 sec latency 16.727 ms oleg140-client.virtnet: 1 37962 3.09 MB/sec execute 13 sec latency 14.838 ms oleg140-client.virtnet: 1 38987 3.59 MB/sec execute 14 sec latency 14.187 ms oleg140-client.virtnet: 1 39529 3.48 MB/sec execute 15 sec latency 8.926 ms oleg140-client.virtnet: 1 40212 3.39 MB/sec execute 16 sec latency 9.958 ms oleg140-client.virtnet: 1 40920 3.39 MB/sec execute 17 sec latency 9.847 ms oleg140-client.virtnet: 1 41836 3.51 MB/sec execute 18 sec latency 8.275 ms oleg140-client.virtnet: 1 42882 3.73 MB/sec execute 19 sec latency 8.167 ms oleg140-client.virtnet: 1 43478 3.60 MB/setest_70b fail mds2 4 times Failing mds2 on oleg140-server Stopping /mnt/lustre-mds2 (opts:) on oleg140-server 19:04:19 (1718319859) shut down Failover mds2 to oleg140-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 19:04:33 (1718319873) targets are mounted 19:04:33 (1718319873) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1285660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13104 3590948 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36512 3569980 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160928 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg140-server Stopping /mnt/lustre-mds1 (opts:) on oleg140-server 19:04:45 (1718319885) shut down Failover mds1 to oleg140-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 19:04:59 (1718319899) targets are mounted 19:04:59 (1718319899) facet_failover done c execute 20 sec latency 13.893 ms oleg140-client.virtnet: 1 44188 3.64 MB/sec execute 21 sec latency 8.978 ms oleg140-client.virtnet: 1 44199 3.48 MB/sec execute 22 sec latency 983.114 ms oleg140-client.virtnet: 1 44199 3.32 MB/sec execute 23 sec latency 1983.341 ms oleg140-client.virtnet: 1 44199 3.19 MB/sec execute 24 sec latency 2983.583 ms oleg140-client.virtnet: 1 44199 3.06 MB/sec execute 25 sec latency 3983.733 ms oleg140-client.virtnet: 1 44199 2.94 MB/sec execute 26 sec latency 4983.986 ms oleg140-client.virtnet: 1 44199 2.83 MB/sec execute 27 sec latency 5984.142 ms oleg140-client.virtnet: 1 44199 2.73 MB/sec execute 28 sec latency 6984.331 ms oleg140-client.virtnet: 1 44199 2.64 MB/sec execute 29 sec latency 7984.537 ms oleg140-client.virtnet: 1 44199 2.55 MB/sec execute 30 sec latency 8984.747 ms oleg140-client.virtnet: 1 44199 2.47 MB/sec execute 31 sec latency 9984.978 ms oleg140-client.virtnet: 1 44199 2.39 MB/sec execute 32 sec latency 10985.157 ms oleg140-client.virtnet: 1 44199 2.32 MB/sec execute 33 sec latency 11985.283 ms oleg140-client.virtnet: 1 44199 2.25 MB/sec execute 34 sec latency 12985.465 ms oleg140-client.virtnet: 1 44199 2.18 MB/sec execute 35 sec latency 13985.663 ms oleg140-client.virtnet: 1 44199 2.12 MB/sec execute 36 sec latency 14985.909 ms oleg140-client.virtnet: 1 44199 2.07 MB/sec execute 37 sec latency 15986.159 ms oleg140-client.virtnet: 1 44199 2.01 MB/sec execute 38 sec latency 16986.469 ms oleg140-client.virtnet: 1 44199 1.96 MB/sec execute 39 sec latency 17986.779 ms oleg140-client.virtnet: 1 44314 1.91 MB/sec execute 40 sec latency 18701.816 ms oleg140-client.virtnet: 1 44946 1.91 MB/sec execute 41 sec latency 9.018 ms oleg140-client.virtnet: 1 45582 2.00 MB/sec execute 42 sec latency 16.649 ms oleg140-client.virtnet: 1 46265 2.07 MB/sec execute 43 sec latency 13.678 ms oleg140-client.virtnet: 1 46637 2.05 MB/sec execute 44 sec latency 13.813 ms oleg140-client.virtnet: 1 46981 2.02 MB/sec execute 45 sec latency 17.042 ms oleg140-client.virtnet: 1 47532 2.07 MB/sec execute 46 sec latency 14.742 ms oleg140-client.virtnet: 1 47963 2.04 MB/sec execute 47 sec latency 15.555 ms oleg140-client.virtnet: 1 48572 2.03 MB/sec execute 48 sec latency 14.112 ms oleg140-client.virtnet: 1 49508 2.14 MB/sec execute 49 sec latency 13.126 ms oleg140-client.virtnet: 1 49959 2.18 MB/sec execute 50 sec latency 15.541 ms oleg140-client.virtnet: 1 50241 2.15 MB/sec execute 51 sec latency 15.514 ms oleg140-client.virtnet: 1 50617 2.12 MB/sec execute 52 sec latency 16.624 ms oleg140-client.virtnet: 1 51196 2.16 MB/sec execute 53 sec latency 12.355 ms oleg140-client.virtnet: 1 51728 2.13 MB/sec execute 54 sec latency 12.885 ms oleg140-client.virtnet: 1 52179 2.13 MB/sec execute 55 sec latency 14.872 ms oleg140-client.virtnet: 1 52984 2.21 MB/sec execute 56 sec latency 15.539 ms oleg140-client.virtnet: 1 53455 2.25 MB/sec execute 57 sec latency 14.669 ms oleg140-client.virtnet: 1 53702 2.22 MB/sec execute 58 sec latency 15.336 ms oleg140-client.virtnet: 1 54324 2.21 MB/sec execute 59 sec latency 14.727 ms oleg140-client.virtnet: 1 54942 2.23 MB/sec execute 60 sec latency 11.362 ms oleg140-client.virtnet: 1 55627 2.22 MB/sec execute 61 sec latency 9.491 ms oleg140-client.virtnet: 1 56595 2.31 MB/sec execute 62 sec latency 14.160 ms oleg140-client.virtnet: 1 57199 2.34 MB/sec execute 63 sec latency 10.249 ms oleg140-client.virtnet: 1 57778 2.32 MB/sec execute 64 sec latency 11.290 ms oleg140-clioleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13192 3590832 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36524 3569964 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49716 7160796 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg140-server Stopping /mnt/lustre-mds2 (opts:) on oleg140-server 19:05:11 (1718319911) shut down Failover mds2 to oleg140-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 19:05:26 (1718319926) targets are mounted 19:05:26 (1718319926) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1285668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13116 3590444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36576 3569388 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49692 7159832 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg140-server Stopping /mnt/lustre-mds1 (opts:) on oleg140-server 19:05:39 (1718319939) shut down ent.virtnet: 1 58389 2.35 MB/sec execute 65 sec latency 16.087 ms oleg140-client.virtnet: 1 58983 2.33 MB/sec execute 66 sec latency 10.166 ms oleg140-client.virtnet: 1 59953 2.40 MB/sec execute 67 sec latency 10.093 ms oleg140-client.virtnet: 1 60603 2.45 MB/sec execute 68 sec latency 11.916 ms oleg140-client.virtnet: 1 61083 2.42 MB/sec execute 69 sec latency 21.093 ms oleg140-client.virtnet: 1 61550 2.41 MB/sec execute 70 sec latency 15.959 ms oleg140-client.virtnet: 1 61817 2.42 MB/sec execute 71 sec latency 568.220 ms oleg140-client.virtnet: 1 62530 2.40 MB/sec execute 72 sec latency 575.725 ms oleg140-client.virtnet: 1 63654 2.48 MB/sec execute 73 sec latency 7.516 ms oleg140-client.virtnet: 1 63700 2.45 MB/sec execute 74 sec latency 923.800 ms oleg140-client.virtnet: 1 63700 2.42 MB/sec execute 75 sec latency 1924.005 ms oleg140-client.virtnet: 1 63700 2.39 MB/sec execute 76 sec latency 2924.215 ms oleg140-client.virtnet: 1 63700 2.36 MB/sec execute 77 sec latency 3924.500 ms oleg140-client.virtnet: 1 63700 2.33 MB/sec execute 78 sec latency 4924.822 ms oleg140-client.virtnet: 1 63700 2.30 MB/sec execute 79 sec latency 5925.135 ms oleg140-client.virtnet: 1 63700 2.27 MB/sec execute 80 sec latency 6925.330 ms oleg140-client.virtnet: 1 63700 2.24 MB/sec execute 81 sec latency 7925.544 ms oleg140-client.virtnet: 1 63700 2.21 MB/sec execute 82 sec latency 8925.762 ms oleg140-client.virtnet: 1 63700 2.18 MB/sec execute 83 sec latency 9925.915 ms oleg140-client.virtnet: 1 63700 2.16 MB/sec execute 84 sec latency 10926.139 ms oleg140-client.virtnet: 1 63700 2.13 MB/sec execute 85 sec latency 11926.372 ms oleg140-client.virtnet: 1 63700 2.11 MB/sec execute 86 sec latency 12926.687 ms oleg140-client.virtnet: 1 63700 2.08 MB/sec execute 87 sec latency 13926.960 ms oleg140-client.virtnet: 1 63700 2.06 MB/sec execute 88 sec latency 14927.542 ms oleg140-client.virtnet: 1 63700 2.04 MB/sec execute 89 sec latency 15927.738 ms oleg140-client.virtnet: 1 63700 2.01 MB/sec execute 90 sec latency 16927.849 ms oleg140-client.virtnet: 1 63700 1.99 MB/sec execute 91 sec latency 17928.052 ms oleg140-client.virtnet: 1 63700 1.97 MB/sec execute 92 sec latency 18928.263 ms oleg140-client.virtnet: 1 63896 1.98 MB/sec execute 93 sec latency 19582.957 ms oleg140-client.virtnet: 1 64211 1.98 MB/sec execute 94 sec latency 14.877 ms oleg140-client.virtnet: 1 64470 1.96 MB/sec execute 95 sec latency 16.223 ms oleg140-client.virtnet: 1 64794 1.95 MB/sec execute 96 sec latency 20.897 ms oleg140-client.virtnet: 1 65254 1.97 MB/sec execute 97 sec latency 15.438 ms oleg140-client.virtnet: 1 65565 1.95 MB/sec execute 98 sec latency 16.027 ms oleg140-client.virtnet: 1 65991 1.94 MB/sec execute 99 sec latency 15.115 ms oleg140-client.virtnet: 1 66427 1.94 MB/sec execute 100 sec latency 15.578 ms oleg140-client.virtnet: 1 67300 1.99 MB/sec execute 101 sec latency 13.206 ms oleg140-client.virtnet: 1 67793 2.01 MB/sec execute 102 sec latency 14.276 ms oleg140-client.virtnet: 1 68118 1.99 MB/sec execute 103 sec latency 17.011 ms oleg140-client.virtnet: 1 68805 2.02 MB/sec execute 104 sec latency 10.580 ms oleg140-client.virtnet: 1 69147 2.00 MB/sec execute 105 sec latency 16.916 ms oleg140-client.virtnet: 1 69716 2.00 MB/sec execute 106 sec latency 10.212 ms oleg140-client.virtnet: 1 70185 2.02 MB/sec execute 107 sec latency 16.240 ms oleg140-client.virtnet: 1 70974 2.06 MB/sec execute 108 sec latency 13.763 ms oleg140-client.virtnet: 1 71548 2.06 MB/seFailover mds1 to oleg140-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 19:05:53 (1718319953) targets are mounted 19:05:53 (1718319953) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593188 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36624 3569744 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49568 7162932 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg140-server Stopping /mnt/lustre-mds2 (opts:) on oleg140-server 19:06:06 (1718319966) shut down Failover mds2 to oleg140-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 19:06:20 (1718319980) targets are mounted 19:06:20 (1718319980) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2024 1285664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13104 3592672 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36396 3569792 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162464 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg140-server Stopping /mnt/lustre-mds1 (opts:) on oleg140-server c execute 109 sec latency 9.696 ms oleg140-client.virtnet: 1 71962 2.05 MB/sec execute 110 sec latency 14.965 ms oleg140-client.virtnet: 1 72479 2.07 MB/sec execute 111 sec latency 15.016 ms oleg140-client.virtnet: 1 72843 2.05 MB/sec execute 112 sec latency 16.405 ms oleg140-client.virtnet: 1 73392 2.05 MB/sec execute 113 sec latency 16.266 ms oleg140-client.virtnet: 1 73989 2.08 MB/sec execute 114 sec latency 13.912 ms oleg140-client.virtnet: 1 74797 2.11 MB/sec execute 115 sec latency 12.141 ms oleg140-client.virtnet: 1 75324 2.10 MB/sec execute 116 sec latency 10.971 ms oleg140-client.virtnet: 1 76003 2.12 MB/sec execute 117 sec latency 13.215 ms oleg140-client.virtnet: 1 76497 2.11 MB/sec execute 118 sec latency 10.502 ms oleg140-client.virtnet: 1 77021 2.11 MB/sec execute 119 sec latency 14.156 ms oleg140-client.virtnet: 1 78070 2.17 MB/sec execute 120 sec latency 11.902 ms oleg140-client.virtnet: 1 78544 2.17 MB/sec execute 121 sec latency 14.369 ms oleg140-client.virtnet: 1 79063 2.16 MB/sec execute 122 sec latency 14.538 ms oleg140-client.virtnet: 1 79643 2.18 MB/sec execute 123 sec latency 14.946 ms oleg140-client.virtnet: 1 80030 2.16 MB/sec execute 124 sec latency 15.819 ms oleg140-client.virtnet: 1 80464 2.16 MB/sec execute 125 sec latency 54.534 ms oleg140-client.virtnet: 1 80688 2.15 MB/sec execute 126 sec latency 647.199 ms oleg140-client.virtnet: 1 81866 2.22 MB/sec execute 127 sec latency 8.290 ms oleg140-client.virtnet: 1 82191 2.20 MB/sec execute 128 sec latency 207.035 ms oleg140-client.virtnet: 1 82191 2.18 MB/sec execute 129 sec latency 1207.215 ms oleg140-client.virtnet: 1 82191 2.17 MB/sec execute 130 sec latency 2207.401 ms oleg140-client.virtnet: 1 82191 2.15 MB/sec execute 131 sec latency 3207.639 ms oleg140-client.virtnet: 1 82191 2.13 MB/sec execute 132 sec latency 4207.900 ms oleg140-client.virtnet: 1 82191 2.12 MB/sec execute 133 sec latency 5208.086 ms oleg140-client.virtnet: 1 82191 2.10 MB/sec execute 134 sec latency 6208.308 ms oleg140-client.virtnet: 1 82191 2.09 MB/sec execute 135 sec latency 7208.521 ms oleg140-client.virtnet: 1 82191 2.07 MB/sec execute 136 sec latency 8208.745 ms oleg140-client.virtnet: 1 82191 2.06 MB/sec execute 137 sec latency 9209.001 ms oleg140-client.virtnet: 1 82191 2.04 MB/sec execute 138 sec latency 10209.159 ms oleg140-client.virtnet: 1 82191 2.03 MB/sec execute 139 sec latency 11209.303 ms oleg140-client.virtnet: 1 82191 2.01 MB/sec execute 140 sec latency 12209.433 ms oleg140-client.virtnet: 1 82191 2.00 MB/sec execute 141 sec latency 13209.551 ms oleg140-client.virtnet: 1 82191 1.98 MB/sec execute 142 sec latency 14209.705 ms oleg140-client.virtnet: 1 82191 1.97 MB/sec execute 143 sec latency 15209.947 ms oleg140-client.virtnet: 1 82191 1.96 MB/sec execute 144 sec latency 16210.092 ms oleg140-client.virtnet: 1 82191 1.94 MB/sec execute 145 sec latency 17210.258 ms oleg140-client.virtnet: 1 82191 1.93 MB/sec execute 146 sec latency 18210.494 ms oleg140-client.virtnet: 1 82312 1.92 MB/sec execute 147 sec latency 18954.842 ms oleg140-client.virtnet: 1 82799 1.92 MB/sec execute 148 sec latency 16.115 ms oleg140-client.virtnet: 1 83250 1.93 MB/sec execute 149 sec latency 14.752 ms oleg140-client.virtnet: 1 84037 1.92 MB/sec execute 150 sec latency 8.254 ms oleg140-client.virtnet: 1 85154 1.98 MB/sec execute 151 sec latency 10.230 ms oleg140-client.virtnet: 1 85549 1.98 MB/sec execute 152 sec latency 13.192 ms oleg140-client.virtnet: 1 86125 1.97 MB/sec execute 153 sec latency 12.952 ms oleg14019:06:31 (1718319991) shut down Failover mds1 to oleg140-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 19:06:45 (1718320005) targets are mounted 19:06:45 (1718320005) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13108 3592788 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36392 3570040 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162828 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg140-server Stopping /mnt/lustre-mds2 (opts:) on oleg140-server 19:06:57 (1718320017) shut down Failover mds2 to oleg140-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 19:07:11 (1718320031) targets are mounted 19:07:11 (1718320031) facet_failover done -client.virtnet: 1 86871 1.99 MB/sec execute 154 sec latency 9.140 ms oleg140-client.virtnet: 1 87512 1.98 MB/sec execute 155 sec latency 10.871 ms oleg140-client.virtnet: 1 88387 2.01 MB/sec execute 156 sec latency 14.799 ms oleg140-client.virtnet: 1 88998 2.04 MB/sec execute 157 sec latency 14.880 ms oleg140-client.virtnet: 1 89586 2.03 MB/sec execute 158 sec latency 10.490 ms oleg140-client.virtnet: 1 89977 2.03 MB/sec execute 159 sec latency 15.609 ms oleg140-client.virtnet: 1 90602 2.03 MB/sec execute 160 sec latency 11.358 ms oleg140-client.virtnet: 1 91497 2.06 MB/sec execute 161 sec latency 8.084 ms oleg140-client.virtnet: 1 92522 2.09 MB/sec execute 162 sec latency 9.110 ms oleg140-client.virtnet: 1 93144 2.08 MB/sec execute 163 sec latency 9.575 ms oleg140-client.virtnet: 1 93878 2.10 MB/sec execute 164 sec latency 8.528 ms oleg140-client.virtnet: 1 94673 2.10 MB/sec execute 165 sec latency 8.721 ms oleg140-client.virtnet: 1 95841 2.15 MB/sec execute 166 sec latency 9.679 ms oleg140-client.virtnet: 1 96386 2.14 MB/sec execute 167 sec latency 8.860 ms oleg140-client.virtnet: 1 96969 2.14 MB/sec execute 168 sec latency 10.182 ms oleg140-client.virtnet: 1 97477 2.15 MB/sec execute 169 sec latency 13.776 ms oleg140-client.virtnet: 1 98218 2.15 MB/sec execute 170 sec latency 9.065 ms oleg140-client.virtnet: 1 99193 2.18 MB/sec execute 171 sec latency 11.770 ms oleg140-client.virtnet: 1 99779 2.19 MB/sec execute 172 sec latency 10.118 ms oleg140-client.virtnet: 1 100361 2.19 MB/sec execute 173 sec latency 10.912 ms oleg140-client.virtnet: 1 100983 2.20 MB/sec execute 174 sec latency 11.273 ms oleg140-client.virtnet: 1 101526 2.19 MB/sec execute 175 sec latency 11.485 ms oleg140-client.virtnet: 1 102127 2.21 MB/sec execute 176 sec latency 14.141 ms oleg140-client.virtnet: 1 102983 2.23 MB/sec execute 177 sec latency 13.543 ms oleg140-client.virtnet: 1 103350 2.23 MB/sec execute 178 sec latency 16.308 ms oleg140-client.virtnet: 1 103928 2.22 MB/sec execute 179 sec latency 12.292 ms oleg140-client.virtnet: 1 104124 2.21 MB/sec execute 180 sec latency 738.396 ms oleg140-client.virtnet: 1 104124 2.20 MB/sec execute 181 sec latency 1738.628 ms oleg140-client.virtnet: 1 104124 2.19 MB/sec execute 182 sec latency 2738.849 ms oleg140-client.virtnet: 1 104124 2.18 MB/sec execute 183 sec latency 3739.056 ms oleg140-client.virtnet: 1 104124 2.17 MB/sec execute 184 sec latency 4739.282 ms oleg140-client.virtnet: 1 104124 2.15 MB/sec execute 185 sec latency 5739.534 ms oleg140-client.virtnet: 1 104124 2.14 MB/sec execute 186 sec latency 6739.848 ms oleg140-client.virtnet: 1 104124 2.13 MB/sec execute 187 sec latency 7740.208 ms oleg140-client.virtnet: 1 104124 2.12 MB/sec execute 188 sec latency 8740.550 ms oleg140-client.virtnet: 1 104124 2.11 MB/sec execute 189 sec latency 9740.792 ms oleg140-client.virtnet: 1 104124 2.10 MB/sec execute 190 sec latency 10740.931 ms oleg140-client.virtnet: 1 104124 2.09 MB/sec execute 191 sec latency 11741.112 ms oleg140-client.virtnet: 1 104124 2.08 MB/sec execute 192 sec latency 12741.282 ms oleg140-client.virtnet: 1 104124 2.06 MB/sec execute 193 sec latency 13741.390 ms oleg140-client.virtnet: 1 104124 2.05 MB/sec execute 194 sec latency 14741.549 ms oleg140-client.virtnet: 1 104124 2.04 MB/sec execute 195 sec latency 15741.745 ms oleg140-client.virtnet: 1 104124 2.03 MB/sec execute 196 sec latency 16742.029 ms oleg140-client.virtnet: 1 104124 2.02 MB/sec execute 197 sec latency 17742.261 ms oleg140-client.virtnet: 1 104124 2.01 MB/sec execute 1oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1285668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13116 3592796 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36396 3569912 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49512 7162708 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg140-server Stopping /mnt/lustre-mds1 (opts:) on oleg140-server 19:07:23 (1718320043) shut down Failover mds1 to oleg140-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 19:07:37 (1718320057) targets are mounted 19:07:37 (1718320057) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2068 1285620 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12976 3593052 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36952 3569428 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49928 7162480 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg140-server Stopping /mnt/lustre-mds2 (opts:) on oleg140-server 19:07:48 (1718320068) shut down Failover mds2 to oleg140-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 98 sec latency 18742.430 ms oleg140-client.virtnet: 1 104578 2.02 MB/sec execute 199 sec latency 19061.530 ms oleg140-client.virtnet: 1 105234 2.02 MB/sec execute 200 sec latency 9.559 ms oleg140-client.virtnet: 1 105838 2.03 MB/sec execute 201 sec latency 13.892 ms oleg140-client.virtnet: 1 106684 2.06 MB/sec execute 202 sec latency 15.471 ms oleg140-client.virtnet: 1 107231 2.05 MB/sec execute 203 sec latency 11.206 ms oleg140-client.virtnet: 1 107709 2.05 MB/sec execute 204 sec latency 15.257 ms oleg140-client.virtnet: 1 108342 2.05 MB/sec execute 205 sec latency 11.078 ms oleg140-client.virtnet: 1 109245 2.07 MB/sec execute 206 sec latency 9.677 ms oleg140-client.virtnet: 1 110288 2.10 MB/sec execute 207 sec latency 7.904 ms oleg140-client.virtnet: 1 110929 2.09 MB/sec execute 208 sec latency 9.253 ms oleg140-client.virtnet: 1 111715 2.10 MB/sec execute 209 sec latency 7.900 ms oleg140-client.virtnet: 1 112560 2.11 MB/sec execute 210 sec latency 7.165 ms oleg140-client.virtnet: 1 113760 2.15 MB/sec execute 211 sec latency 7.418 ms oleg140-client.virtnet: 1 114411 2.14 MB/sec execute 212 sec latency 10.414 ms oleg140-client.virtnet: 1 115192 2.15 MB/sec execute 213 sec latency 7.273 ms oleg140-client.virtnet: 1 115987 2.15 MB/sec execute 214 sec latency 8.162 ms oleg140-client.virtnet: 1 117215 2.19 MB/sec execute 215 sec latency 7.938 ms oleg140-client.virtnet: 1 117844 2.19 MB/sec execute 216 sec latency 8.504 ms oleg140-client.virtnet: 1 118610 2.20 MB/sec execute 217 sec latency 8.457 ms oleg140-client.virtnet: 1 119425 2.20 MB/sec execute 218 sec latency 8.229 ms oleg140-client.virtnet: 1 120601 2.24 MB/sec execute 219 sec latency 8.176 ms oleg140-client.virtnet: 1 121156 2.24 MB/sec execute 220 sec latency 8.783 ms oleg140-client.virtnet: 1 121884 2.24 MB/sec execute 221 sec latency 9.417 ms oleg140-client.virtnet: 1 122692 2.24 MB/sec execute 222 sec latency 10.265 ms oleg140-client.virtnet: 1 123627 2.26 MB/sec execute 223 sec latency 7.813 ms oleg140-client.virtnet: 1 124565 2.28 MB/sec execute 224 sec latency 8.092 ms oleg140-client.virtnet: 1 125216 2.28 MB/sec execute 225 sec latency 13.082 ms oleg140-client.virtnet: 1 125979 2.29 MB/sec execute 226 sec latency 10.859 ms oleg140-client.virtnet: 1 126788 2.29 MB/sec execute 227 sec latency 8.249 ms oleg140-client.virtnet: 1 127963 2.32 MB/sec execute 228 sec latency 7.874 ms oleg140-client.virtnet: 1 128562 2.32 MB/sec execute 229 sec latency 10.489 ms oleg140-client.virtnet: 1 129313 2.33 MB/sec execute 230 sec latency 8.415 ms oleg140-client.virtnet: 1 129395 2.32 MB/sec execute 231 sec latency 872.520 ms oleg140-client.virtnet: 1 129395 2.31 MB/sec execute 232 sec latency 1872.697 ms oleg140-client.virtnet: 1 129395 2.30 MB/sec execute 233 sec latency 2872.898 ms oleg140-client.virtnet: 1 129395 2.29 MB/sec execute 234 sec latency 3873.050 ms oleg140-client.virtnet: 1 129395 2.28 MB/sec execute 235 sec latency 4873.202 ms oleg140-client.virtnet: 1 129395 2.27 MB/sec execute 236 sec latency 5873.324 ms oleg140-client.virtnet: 1 129395 2.26 MB/sec execute 237 sec latency 6873.455 ms oleg140-client.virtnet: 1 129395 2.25 MB/sec execute 238 sec latency 7873.616 ms oleg140-client.virtnet: 1 129395 2.24 MB/sec execute 239 sec latency 8873.780 ms oleg140-client.virtnet: 1 129395 2.23 MB/sec execute 240 sec latency 9873.957 ms oleg140-client.virtnet: 1 129395 2.22 MB/sec execute 241 sec latency 10874.069 ms oleg140-client.virtnet: 1 129395 2.21 MB/sec execute 242 sec latency 11874.156 ms oleg140-client.virtnet: 1 129395 2.20 MB/sec oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 19:08:01 (1718320081) targets are mounted 19:08:01 (1718320081) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec execute 243 sec latency 12874.283 ms oleg140-client.virtnet: 1 129395 2.19 MB/sec execute 244 sec latency 13874.387 ms oleg140-client.virtnet: 1 129395 2.19 MB/sec execute 245 sec latency 14874.492 ms oleg140-client.virtnet: 1 129395 2.18 MB/sec execute 246 sec latency 15874.668 ms oleg140-client.virtnet: 1 129395 2.17 MB/sec execute 247 sec latency 16874.812 ms oleg140-client.virtnet: 1 129395 2.16 MB/sec execute 248 sec latency 17875.018 ms oleg140-client.virtnet: 1 129759 2.15 MB/sec execute 249 sec latency 18238.368 ms oleg140-client.virtnet: 1 130650 2.17 MB/sec execute 250 sec latency 10.631 ms oleg140-client.virtnet: 1 131601 2.19 MB/sec execute 251 sec latency 7.493 ms oleg140-client.virtnet: 1 132253 2.18 MB/sec execute 252 sec latency 10.222 ms oleg140-client.virtnet: 1 133017 2.19 MB/sec execute 253 sec latency 9.131 ms oleg140-client.virtnet: 1 133843 2.19 MB/sec execute 254 sec latency 7.769 ms oleg140-client.virtnet: 1 134616 2.21 MB/sec execute 255 sec latency 15.669 ms oleg140-client.virtnet: 1 135098 2.22 MB/sec execute 256 sec latency 13.540 ms oleg140-client.virtnet: 1 135345 2.21 MB/sec execute 257 sec latency 15.422 ms oleg140-client.virtnet: 1 135747 2.21 MB/sec execute 258 sec latency 18.182 ms oleg140-client.virtnet: 1 136101 2.20 MB/sec execute 259 sec latency 15.881 ms oleg140-client.virtnet: 1 136505 2.21 MB/sec execute 260 sec latency 15.773 ms oleg140-client.virtnet: 1 136945 2.20 MB/sec execute 261 sec latency 17.838 ms oleg140-client.virtnet: 1 137311 2.20 MB/sec execute 262 sec latency 15.935 ms oleg140-client.virtnet: 1 138318 2.23 MB/sec execute 263 sec latency 13.229 ms oleg140-client.virtnet: 1 138671 2.22 MB/sec execute 264 sec latency 15.871 ms oleg140-client.virtnet: 1 138956 2.22 MB/sec execute 265 sec latency 16.096 ms oleg140-client.virtnet: 1 139326 2.21 MB/sec execute 266 sec latency 17.205 ms oleg140-client.virtnet: 1 139670 2.21 MB/sec execute 267 sec latency 15.878 ms oleg140-client.virtnet: 1 140065 2.21 MB/sec execute 268 sec latency 16.616 ms oleg140-client.virtnet: 1 140633 2.21 MB/sec execute 269 sec latency 10.451 ms oleg140-client.virtnet: 1 141262 2.22 MB/sec execute 270 sec latency 15.843 ms oleg140-client.virtnet: 1 141894 2.23 MB/sec execute 271 sec latency 15.819 ms oleg140-client.virtnet: 1 142238 2.23 MB/sec execute 272 sec latency 15.635 ms oleg140-client.virtnet: 1 142728 2.22 MB/sec execute 273 sec latency 15.420 ms oleg140-client.virtnet: 1 143091 2.22 MB/sec execute 274 sec latency 15.505 ms oleg140-client.virtnet: 1 143788 2.23 MB/sec execute 275 sec latency 10.090 ms oleg140-client.virtnet: 1 144684 2.24 MB/sec execute 276 sec latency 7.698 ms oleg140-client.virtnet: 1 145596 2.25 MB/sec execute 277 sec latency 10.354 ms oleg140-client.virtnet: 1 145938 2.25 MB/sec execute 278 sec latency 15.571 ms oleg140-client.virtnet: 1 146334 2.25 MB/sec execute 279 sec latency 20.122 ms oleg140-client.virtnet: 1 146695 2.24 MB/sec execute 280 sec latency 15.418 ms oleg140-client.virtnet: 1 147300 2.25 MB/sec execute 281 sec latency 16.659 ms oleg140-client.virtnet: 1 147930 2.25 MB/sec execute 282 sec latency 17.244 ms oleg140-client.virtnet: 1 148481 2.26 MB/sec execute 283 sec latency 15.975 ms oleg140-client.virtnet: 1 149110 2.27 MB/sec execute 284 sec latency 15.660 ms oleg140-client.virtnet: 1 149384 2.26 MB/sec execute 285 sec latency 15.941 ms oleg140-client.virtnet: 1 149752 2.26 MB/sec execute 286 sec latency 15.336 ms oleg140-client.virtnet: 1 150469 2.26 MB/sec execute 287 sec latency 9.733 ms oleg140-client.virtnet: 1 150845 2.26 MB/sec execute 288 sec latency 14.392 ms oleg140-client.virtnet: 1 151370 2.26 MB/sec execute 289 sec latency 11.716 ms oleg140-client.virtnet: 1 152007 2.27 MB/sec execute 290 sec latency 15.636 ms oleg140-client.virtnet: 1 152633 2.28 MB/sec execute 291 sec latency 16.219 ms oleg140-client.virtnet: 1 152922 2.27 MB/sec execute 292 sec latency 15.990 ms oleg140-client.virtnet: 1 153327 2.27 MB/sec execute 293 sec latency 18.300 ms oleg140-client.virtnet: 1 153755 2.27 MB/sec execute 294 sec latency 19.451 ms oleg140-client.virtnet: 1 154245 2.27 MB/sec execute 295 sec latency 17.360 ms oleg140-client.virtnet: 1 154703 2.26 MB/sec execute 296 sec latency 14.676 ms oleg140-client.virtnet: 1 155041 2.26 MB/sec execute 297 sec latency 16.269 ms oleg140-client.virtnet: 1 155726 2.27 MB/sec execute 298 sec latency 15.823 ms oleg140-client.virtnet: 1 156223 2.28 MB/sec execute 299 sec latency 15.647 ms oleg140-client.virtnet: 1 cleanup 300 sec oleg140-client.virtnet: 0 cleanup 300 sec oleg140-client.virtnet: oleg140-client.virtnet: Operation Count AvgLat MaxLat oleg140-client.virtnet: ---------------------------------------- oleg140-client.virtnet: NTCreateX 21862 5.964 18238.360 oleg140-client.virtnet: Close 16060 4.666 19582.921 oleg140-client.virtnet: Rename 926 7.583 17.818 oleg140-client.virtnet: Unlink 4408 6.275 18954.818 oleg140-client.virtnet: Qpathinfo 19841 1.408 19.032 oleg140-client.virtnet: Qfileinfo 3466 0.150 2.197 oleg140-client.virtnet: Qfsinfo 3636 0.234 14.099 oleg140-client.virtnet: Sfileinfo 1766 3.914 13.485 oleg140-client.virtnet: Find 7667 0.422 13.343 oleg140-client.virtnet: WriteX 10813 0.907 7.935 oleg140-client.virtnet: ReadX 34390 0.019 4.079 oleg140-client.virtnet: LockX 72 1.118 2.041 oleg140-client.virtnet: UnlockX 72 1.148 2.084 oleg140-client.virtnet: Flush 1523 6.001 647.176 oleg140-client.virtnet: oleg140-client.virtnet: Throughput 2.28032 MB/sec 1 clients 1 procs max_latency=19582.957 ms oleg140-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg140-client.virtnet at Thu Jun 13 19:08:57 EDT 2024 with return code 0 oleg140-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg140-client.virtnet oleg140-client.virtnet: /mnt/lustre/d70b.replay-single/oleg140-client.virtnet /mnt/lustre/d70b.replay-single/oleg140-client.virtnet oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg140-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg140-client.virtnet: removed directory: 'clients/client0' oleg140-client.virtnet: removed directory: 'clients' oleg140-client.virtnet: removed 'client.txt' oleg140-client.virtnet: /mnt/lustre/d70b.replay-single/oleg140-client.virtnet oleg140-client.virtnet: dbench successfully finished