== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 19:48:57 (1723247337) Starting client oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre Started clients oleg427-client.virtnet: 192.168.204.127@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg427-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=12479 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2468 1285220 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26160 3556140 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3592988 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7149128 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg427-server Stopping /mnt/lustre-mds1 (opts:) on oleg427-server 19:49:06 (1723247346) shut down Failover mds1 to oleg427-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 19:49:21 (1723247361) targets are mounted 19:49:21 (1723247361) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26416 3569800 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1728 3593040 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28144 7162840 1% /mnt/lustre oleg427-client.virtnet: looking for dbench program oleg427-client.virtnet: /usr/bin/dbench oleg427-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg427-client.virtnet oleg427-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg427-client.virtnet' oleg427-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg427-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg427-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg427-client.virtnet at Fri Aug 9 19:48:58 EDT 2024 oleg427-client.virtnet: waiting for dbench pid 12503 oleg427-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg427-client.virtnet: oleg427-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg427-client.virtnet: failed to create barrier semaphore oleg427-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg427-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg427-client.virtnet: releasing clients oleg427-client.virtnet: 1 305 11.22 MB/sec warmup 1 sec latency 24.583 ms oleg427-client.virtnet: 1 661 10.57 MB/sec warmup 2 sec latency 18.128 ms oleg427-client.virtnet: 1 933 7.26 MB/sec warmup 3 sec latency 19.262 ms oleg427-client.virtnet: 1 1268 5.78 MB/sec warmup 4 sec latency 16.919 ms oleg427-client.virtnet: 1 1668 5.24 MB/sec warmup 5 sec latency 16.204 ms oleg427-client.virtnet: 1 2016 4.42 MB/sec warmup 6 sec latency 19.256 ms oleg427-client.virtnet: 1 2396 4.00 MB/sec warmup 7 sec latency 17.393 ms oleg427-client.virtnet: 1 2917 4.18 MB/sec warmup 8 sec latency 30.260 ms oleg427-client.virtnet: 1 3543 4.27 MB/sec warmup 9 sec latency 18.138 ms oleg427-client.virtnet: 1 3850 3.98 MB/sec warmup 10 sec latency 16.252 ms oleg427-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 18.305 ms oleg427-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 1018.481 ms oleg427-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 2018.671 ms oleg427-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 3018.811 ms oleg427-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 4019.030 ms oleg427-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 5019.227 ms oleg427-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 6019.442 ms oleg427-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 7019.641 ms oleg427-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 8019.839 ms oleg427-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 9020.071 ms oleg427-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 10020.263 ms oleg427-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 11020.471 ms oleg427-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 12020.682 ms oleg427-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 13020.897 ms oleg427-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 14021.072 ms oleg427-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 15021.232 ms oleg427-client.virtnet: 1 4352 1.50 MB/sec warmup 27 sec latency 15260.250 ms oleg427-client.virtnet: 1 4663 1.50 MB/sec warmup 28 sec latency 17.533 ms oleg427-client.virtnet: 1 5030 1.56 MB/sec warmup 29 sec latency 17.691 ms oleg427-client.virtnet: 1 5296 1.51 MB/sec warmup 30 sec latency 19.005 ms oleg427-client.virtnet: 1 5620 1.48 MB/sec warmup 31 sec latency 14.665 ms oleg427-client.virtnet: 1 5977 1.48 MB/sec warmup 32 sec latency 17.738 ms oleg427-client.virtnet: 1 6138 1.46 MB/sec warmup 33 sec latency 570.914 ms test_70b fail mds2 2 times Failing mds2 on oleg427-server Stopping /mnt/lustre-mds2 (opts:) on oleg427-server 19:49:35 (1723247375) shut down Failover mds2 to oleg427-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0001 19:49:50 (1723247390) targets are mounted 19:49:50 (1723247390) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3570160 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13084 3593880 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7164040 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg427-server Stopping /mnt/lustre-mds1 (opts:) on oleg427-server 19:50:04 (1723247404) shut down Failover mds1 to oleg427-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-client.virtnet: 1 6790 1.57 MB/sec warmup 34 sec latency 18.592 ms oleg427-client.virtnet: 1 7259 1.65 MB/sec warmup 35 sec latency 13.448 ms oleg427-client.virtnet: 1 7375 1.64 MB/sec warmup 36 sec latency 590.138 ms oleg427-client.virtnet: 1 7375 1.59 MB/sec warmup 37 sec latency 1590.352 ms oleg427-client.virtnet: 1 7375 1.55 MB/sec warmup 38 sec latency 2590.547 ms oleg427-client.virtnet: 1 7375 1.51 MB/sec warmup 39 sec latency 3590.737 ms oleg427-client.virtnet: 1 7375 1.47 MB/sec warmup 40 sec latency 4590.928 ms oleg427-client.virtnet: 1 7375 1.44 MB/sec warmup 41 sec latency 5591.116 ms oleg427-client.virtnet: 1 7375 1.40 MB/sec warmup 42 sec latency 6591.301 ms oleg427-client.virtnet: 1 7375 1.37 MB/sec warmup 43 sec latency 7591.531 ms oleg427-client.virtnet: 1 7375 1.34 MB/sec warmup 44 sec latency 8591.702 ms oleg427-client.virtnet: 1 7375 1.31 MB/sec warmup 45 sec latency 9591.869 ms oleg427-client.virtnet: 1 7375 1.28 MB/sec warmup 46 sec latency 10592.045 ms oleg427-client.virtnet: 1 7375 1.25 MB/sec warmup 47 sec latency 11592.211 ms oleg427-client.virtnet: 1 7375 1.23 MB/sec warmup 48 sec latency 12592.381 ms oleg427-client.virtnet: 1 7375 1.20 MB/sec warmup 49 sec latency 13592.523 ms oleg427-client.virtnet: 1 7375 1.18 MB/sec warmup 50 sec latency 14592.715 ms oleg427-client.virtnet: 1 7375 1.15 MB/sec warmup 51 sec latency 15592.898 ms oleg427-client.virtnet: 1 7375 1.13 MB/sec warmup 52 sec latency 16593.078 ms oleg427-client.virtnet: 1 7375 1.11 MB/sec warmup 53 sec latency 17593.251 ms oleg427-client.virtnet: 1 7375 1.09 MB/sec warmup 54 sec latency 18593.455 ms oleg427-client.virtnet: 1 7375 1.07 MB/sec warmup 55 sec latency 19593.650 ms oleg427-client.virtnet: 1 7558 1.06 MB/sec warmup 56 sec latency 19822.427 ms oleg427-client.virtnet: 1 7848 1.04 MB/sec warmup 57 sec latency 20.735 ms oleg427-client.virtnet: 1 8101 1.04 MB/sec warmup 58 sec latency 18.741 ms oleg427-client.virtnet: 1 8521 1.09 MB/sec warmup 59 sec latency 18.740 ms oleg427-client.virtnet: 1 9117 0.29 MB/sec execute 1 sec latency 18.998 ms oleg427-client.virtnet: 1 9492 0.87 MB/sec execute 2 sec latency 11.965 ms oleg427-client.virtnet: 1 9996 2.20 MB/sec execute 3 sec latency 17.741 ms oleg427-client.virtnet: 1 10622 3.04 MB/sec execute 4 sec latency 17.874 ms oleg427-client.virtnet: 1 10925 2.71 MB/sec execute 5 sec latency 16.234 ms oleg427-client.virtnet: 1 11155 2.31 MB/sec execute 6 sec latency 18.970 ms oleg427-client.virtnet: 1 11475 2.05 MB/sec execute 7 sec latency 19.294 ms oleg427-client.virtnet: 1 11792 1.99 MB/sec execute 8 sec latency 17.716 ms oleg427-client.virtnet: 1 12155 2.10 MB/sec execute 9 sec latency 17.868 ms oleg427-client.virtnet: 1 12440 1.92 MB/sec execute 10 sec latency 18.039 ms oleg427-client.virtnet: 1 12767 1.78 MB/sec execute 11 sec latency 12.084 ms oleg427-client.virtnet: 1 13104 1.75 MB/sec execute 12 sec latency 17.947 ms oleg427-client.virtnet: 1 13660 2.03 MB/sec execute 13 sec latency 17.448 ms oleg427-client.virtnet: 1 14269 2.25 MB/sec execute 14 sec latency 17.467 ms oleg427-client.virtnet: 1 14544 2.20 MB/sec execute 15 sec latency 15.881 ms oleg427-client.virtnet: 1 14847 2.07 MB/sec execute 16 sec latency 15.423 ms oleg427-client.virtnet: 1 15122 1.99 MB/sec execute 17 sec latency 16.978 ms oleg427-client.virtnet: 1 15433 1.95 MB/sec execute 18 sec latency 18.235 ms oleg427-client.virtnet: 1 15798 2.01 MB/sec execute 1oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 19:50:19 (1723247419) targets are mounted 19:50:19 (1723247419) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3570028 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13412 3592592 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49816 7162620 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg427-server Stopping /mnt/lustre-mds2 (opts:) on oleg427-server 19:50:32 (1723247432) shut down Failover mds2 to oleg427-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0001 19:50:48 (1723247448) targets are mounted 19:50:48 (1723247448) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36540 3570012 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13096 3590976 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160988 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg427-server Stopping /mnt/lustre-mds1 (opts:) on oleg427-server 19:51:01 (1723247461) shut down 9 sec latency 16.480 ms oleg427-client.virtnet: 1 16067 1.92 MB/sec execute 20 sec latency 19.313 ms oleg427-client.virtnet: 1 16401 1.85 MB/sec execute 21 sec latency 12.019 ms oleg427-client.virtnet: 1 16769 1.86 MB/sec execute 22 sec latency 18.121 ms oleg427-client.virtnet: 1 17383 2.01 MB/sec execute 23 sec latency 18.221 ms oleg427-client.virtnet: 1 17852 2.11 MB/sec execute 24 sec latency 18.798 ms oleg427-client.virtnet: 1 18122 2.08 MB/sec execute 25 sec latency 16.476 ms oleg427-client.virtnet: 1 18420 2.01 MB/sec execute 26 sec latency 20.936 ms oleg427-client.virtnet: 1 18677 1.96 MB/sec execute 27 sec latency 20.220 ms oleg427-client.virtnet: 1 18996 1.94 MB/sec execute 28 sec latency 17.625 ms oleg427-client.virtnet: 1 19359 1.98 MB/sec execute 29 sec latency 14.962 ms oleg427-client.virtnet: 1 19635 1.92 MB/sec execute 30 sec latency 41.655 ms oleg427-client.virtnet: 1 19852 1.86 MB/sec execute 31 sec latency 472.215 ms oleg427-client.virtnet: 1 20192 1.85 MB/sec execute 32 sec latency 16.309 ms oleg427-client.virtnet: 1 20709 1.96 MB/sec execute 33 sec latency 17.891 ms oleg427-client.virtnet: 1 20747 1.90 MB/sec execute 34 sec latency 941.200 ms oleg427-client.virtnet: 1 20747 1.85 MB/sec execute 35 sec latency 1941.424 ms oleg427-client.virtnet: 1 20747 1.80 MB/sec execute 36 sec latency 2941.591 ms oleg427-client.virtnet: 1 20747 1.75 MB/sec execute 37 sec latency 3941.760 ms oleg427-client.virtnet: 1 20747 1.70 MB/sec execute 38 sec latency 4941.994 ms oleg427-client.virtnet: 1 20747 1.66 MB/sec execute 39 sec latency 5942.177 ms oleg427-client.virtnet: 1 20747 1.62 MB/sec execute 40 sec latency 6942.456 ms oleg427-client.virtnet: 1 20747 1.58 MB/sec execute 41 sec latency 7942.655 ms oleg427-client.virtnet: 1 20747 1.54 MB/sec execute 42 sec latency 8942.853 ms oleg427-client.virtnet: 1 20747 1.50 MB/sec execute 43 sec latency 9943.107 ms oleg427-client.virtnet: 1 20747 1.47 MB/sec execute 44 sec latency 10943.392 ms oleg427-client.virtnet: 1 20747 1.44 MB/sec execute 45 sec latency 11943.558 ms oleg427-client.virtnet: 1 20747 1.41 MB/sec execute 46 sec latency 12943.748 ms oleg427-client.virtnet: 1 20747 1.38 MB/sec execute 47 sec latency 13943.928 ms oleg427-client.virtnet: 1 20747 1.35 MB/sec execute 48 sec latency 14944.124 ms oleg427-client.virtnet: 1 20747 1.32 MB/sec execute 49 sec latency 15944.293 ms oleg427-client.virtnet: 1 20747 1.29 MB/sec execute 50 sec latency 16944.481 ms oleg427-client.virtnet: 1 20747 1.27 MB/sec execute 51 sec latency 17944.663 ms oleg427-client.virtnet: 1 20747 1.24 MB/sec execute 52 sec latency 18944.858 ms oleg427-client.virtnet: 1 21015 1.24 MB/sec execute 53 sec latency 19643.562 ms oleg427-client.virtnet: 1 21477 1.30 MB/sec execute 54 sec latency 14.408 ms oleg427-client.virtnet: 1 21761 1.30 MB/sec execute 55 sec latency 15.764 ms oleg427-client.virtnet: 1 22046 1.28 MB/sec execute 56 sec latency 24.422 ms oleg427-client.virtnet: 1 22371 1.28 MB/sec execute 57 sec latency 17.221 ms oleg427-client.virtnet: 1 22778 1.32 MB/sec execute 58 sec latency 17.109 ms oleg427-client.virtnet: 1 23049 1.30 MB/sec execute 59 sec latency 19.000 ms oleg427-client.virtnet: 1 23360 1.28 MB/sec execute 60 sec latency 11.868 ms oleg427-client.virtnet: 1 23733 1.28 MB/sec execute 61 sec latency 17.423 ms oleg427-client.virtnet: 1 24342 1.35 MB/sec execute 62 sec latency 18.459 ms oleg427-client.virtnet: 1 25247 1.43 MB/sec execute 63 sec latency 8.214 ms oleg427-client.virFailover mds1 to oleg427-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 19:51:16 (1723247476) targets are mounted 19:51:16 (1723247476) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36408 3570016 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13084 3592852 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7162868 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg427-server Stopping /mnt/lustre-mds2 (opts:) on oleg427-server 19:51:29 (1723247489) shut down Failover mds2 to oleg427-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0001 19:51:45 (1723247505) targets are mounted 19:51:45 (1723247505) facet_failover done tnet: 1 25843 1.43 MB/sec execute 64 sec latency 9.414 ms oleg427-client.virtnet: 1 26565 1.47 MB/sec execute 65 sec latency 8.476 ms oleg427-client.virtnet: 1 27270 1.48 MB/sec execute 66 sec latency 9.488 ms oleg427-client.virtnet: 1 27787 1.54 MB/sec execute 67 sec latency 17.862 ms oleg427-client.virtnet: 1 28407 1.59 MB/sec execute 68 sec latency 17.645 ms oleg427-client.virtnet: 1 28760 1.59 MB/sec execute 69 sec latency 13.137 ms oleg427-client.virtnet: 1 29025 1.56 MB/sec execute 70 sec latency 15.496 ms oleg427-client.virtnet: 1 29327 1.55 MB/sec execute 71 sec latency 18.583 ms oleg427-client.virtnet: 1 29638 1.55 MB/sec execute 72 sec latency 17.853 ms oleg427-client.virtnet: 1 29997 1.57 MB/sec execute 73 sec latency 17.979 ms oleg427-client.virtnet: 1 30274 1.55 MB/sec execute 74 sec latency 19.398 ms oleg427-client.virtnet: 1 30635 1.54 MB/sec execute 75 sec latency 11.445 ms oleg427-client.virtnet: 1 31019 1.55 MB/sec execute 76 sec latency 17.648 ms oleg427-client.virtnet: 1 31720 1.61 MB/sec execute 77 sec latency 17.416 ms oleg427-client.virtnet: 1 32130 1.63 MB/sec execute 78 sec latency 12.574 ms oleg427-client.virtnet: 1 32388 1.63 MB/sec execute 79 sec latency 16.371 ms oleg427-client.virtnet: 1 32684 1.61 MB/sec execute 80 sec latency 23.346 ms oleg427-client.virtnet: 1 32943 1.60 MB/sec execute 81 sec latency 18.544 ms oleg427-client.virtnet: 1 33358 1.63 MB/sec execute 82 sec latency 18.220 ms oleg427-client.virtnet: 1 33638 1.61 MB/sec execute 83 sec latency 16.425 ms oleg427-client.virtnet: 1 33948 1.60 MB/sec execute 84 sec latency 19.047 ms oleg427-client.virtnet: 1 34313 1.59 MB/sec execute 85 sec latency 13.942 ms oleg427-client.virtnet: 1 34731 1.62 MB/sec execute 86 sec latency 18.127 ms oleg427-client.virtnet: 1 35393 1.66 MB/sec execute 87 sec latency 17.794 ms oleg427-client.virtnet: 1 35752 1.68 MB/sec execute 88 sec latency 15.648 ms oleg427-client.virtnet: 1 35974 1.66 MB/sec execute 89 sec latency 16.891 ms oleg427-client.virtnet: 1 36288 1.65 MB/sec execute 90 sec latency 17.208 ms oleg427-client.virtnet: 1 36297 1.63 MB/sec execute 91 sec latency 978.524 ms oleg427-client.virtnet: 1 36297 1.61 MB/sec execute 92 sec latency 1978.693 ms oleg427-client.virtnet: 1 36297 1.59 MB/sec execute 93 sec latency 2978.875 ms oleg427-client.virtnet: 1 36297 1.58 MB/sec execute 94 sec latency 3979.128 ms oleg427-client.virtnet: 1 36297 1.56 MB/sec execute 95 sec latency 4979.318 ms oleg427-client.virtnet: 1 36297 1.54 MB/sec execute 96 sec latency 5979.503 ms oleg427-client.virtnet: 1 36297 1.53 MB/sec execute 97 sec latency 6979.685 ms oleg427-client.virtnet: 1 36297 1.51 MB/sec execute 98 sec latency 7979.865 ms oleg427-client.virtnet: 1 36297 1.50 MB/sec execute 99 sec latency 8980.110 ms oleg427-client.virtnet: 1 36297 1.48 MB/sec execute 100 sec latency 9980.284 ms oleg427-client.virtnet: 1 36297 1.47 MB/sec execute 101 sec latency 10980.468 ms oleg427-client.virtnet: 1 36297 1.45 MB/sec execute 102 sec latency 11980.592 ms oleg427-client.virtnet: 1 36297 1.44 MB/sec execute 103 sec latency 12980.771 ms oleg427-client.virtnet: 1 36297 1.42 MB/sec execute 104 sec latency 13980.969 ms oleg427-client.virtnet: 1 36297 1.41 MB/sec execute 105 sec latency 14981.149 ms oleg427-client.virtnet: 1 36297 1.40 MB/sec execute 106 sec latency 15981.331 ms oleg427-client.virtnet: 1 36297 1.38 MB/sec execute 107 sec latency 16981.513 ms oleg427-client.virtnet: 1 36297 1.37 MB/sec execute 108 sec oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3570220 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13084 3593024 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163244 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg427-server Stopping /mnt/lustre-mds1 (opts:) on oleg427-server 19:51:58 (1723247518) shut down Failover mds1 to oleg427-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 19:52:14 (1723247534) targets are mounted 19:52:14 (1723247534) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36412 3570040 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13084 3592880 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7162920 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg427-server Stopping /mnt/lustre-mds2 (opts:) on oleg427-server 19:52:27 (1723247547) shut down latency 17981.701 ms oleg427-client.virtnet: 1 36297 1.36 MB/sec execute 109 sec latency 18981.868 ms oleg427-client.virtnet: 1 36308 1.35 MB/sec execute 110 sec latency 19933.269 ms oleg427-client.virtnet: 1 36619 1.35 MB/sec execute 111 sec latency 18.123 ms oleg427-client.virtnet: 1 36980 1.36 MB/sec execute 112 sec latency 17.909 ms oleg427-client.virtnet: 1 37268 1.35 MB/sec execute 113 sec latency 17.021 ms oleg427-client.virtnet: 1 37580 1.35 MB/sec execute 114 sec latency 12.023 ms oleg427-client.virtnet: 1 37937 1.35 MB/sec execute 115 sec latency 17.546 ms oleg427-client.virtnet: 1 38479 1.38 MB/sec execute 116 sec latency 17.967 ms oleg427-client.virtnet: 1 39142 1.41 MB/sec execute 117 sec latency 17.638 ms oleg427-client.virtnet: 1 39415 1.41 MB/sec execute 118 sec latency 15.559 ms oleg427-client.virtnet: 1 39708 1.40 MB/sec execute 119 sec latency 15.737 ms oleg427-client.virtnet: 1 40024 1.40 MB/sec execute 120 sec latency 16.249 ms oleg427-client.virtnet: 1 40336 1.40 MB/sec execute 121 sec latency 18.853 ms oleg427-client.virtnet: 1 40694 1.41 MB/sec execute 122 sec latency 15.770 ms oleg427-client.virtnet: 1 40989 1.40 MB/sec execute 123 sec latency 20.086 ms oleg427-client.virtnet: 1 41354 1.40 MB/sec execute 124 sec latency 11.911 ms oleg427-client.virtnet: 1 41808 1.42 MB/sec execute 125 sec latency 17.522 ms oleg427-client.virtnet: 1 42418 1.44 MB/sec execute 126 sec latency 17.764 ms oleg427-client.virtnet: 1 42816 1.46 MB/sec execute 127 sec latency 16.240 ms oleg427-client.virtnet: 1 43042 1.45 MB/sec execute 128 sec latency 15.971 ms oleg427-client.virtnet: 1 43339 1.44 MB/sec execute 129 sec latency 20.772 ms oleg427-client.virtnet: 1 43592 1.44 MB/sec execute 130 sec latency 18.061 ms oleg427-client.virtnet: 1 43995 1.46 MB/sec execute 131 sec latency 18.191 ms oleg427-client.virtnet: 1 44272 1.45 MB/sec execute 132 sec latency 16.031 ms oleg427-client.virtnet: 1 44577 1.44 MB/sec execute 133 sec latency 20.579 ms oleg427-client.virtnet: 1 44948 1.44 MB/sec execute 134 sec latency 12.173 ms oleg427-client.virtnet: 1 45364 1.46 MB/sec execute 135 sec latency 18.072 ms oleg427-client.virtnet: 1 45961 1.47 MB/sec execute 136 sec latency 19.277 ms oleg427-client.virtnet: 1 46356 1.49 MB/sec execute 137 sec latency 14.451 ms oleg427-client.virtnet: 1 46582 1.49 MB/sec execute 138 sec latency 15.840 ms oleg427-client.virtnet: 1 46889 1.48 MB/sec execute 139 sec latency 29.243 ms oleg427-client.virtnet: 1 47162 1.47 MB/sec execute 140 sec latency 19.420 ms oleg427-client.virtnet: 1 47564 1.49 MB/sec execute 141 sec latency 17.664 ms oleg427-client.virtnet: 1 47844 1.48 MB/sec execute 142 sec latency 20.306 ms oleg427-client.virtnet: 1 48159 1.47 MB/sec execute 143 sec latency 11.231 ms oleg427-client.virtnet: 1 48531 1.47 MB/sec execute 144 sec latency 12.264 ms oleg427-client.virtnet: 1 48654 1.47 MB/sec execute 145 sec latency 536.132 ms oleg427-client.virtnet: 1 49327 1.49 MB/sec execute 146 sec latency 17.795 ms oleg427-client.virtnet: 1 49806 1.52 MB/sec execute 147 sec latency 17.116 ms oleg427-client.virtnet: 1 49952 1.51 MB/sec execute 148 sec latency 487.262 ms oleg427-client.virtnet: 1 49952 1.50 MB/sec execute 149 sec latency 1487.426 ms oleg427-client.virtnet: 1 49952 1.49 MB/sec execute 150 sec latency 2487.643 ms oleg427-client.virtnet: 1 49952 1.48 MB/sec execute 151 sec latency 3487.822 ms oleg427-client.virtnet: 1 49952 1.47 MB/sec execute 152 sec latency 4487.973 ms oleg427-client.virtnet: 1 49952 1.46 MB/sec Failover mds2 to oleg427-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0001 19:52:42 (1723247562) targets are mounted 19:52:42 (1723247562) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36564 3570400 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12924 3590800 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7161200 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg427-server Stopping /mnt/lustre-mds1 (opts:) on oleg427-server 19:52:56 (1723247576) shut down Failover mds1 to oleg427-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 19:53:11 (1723247591) targets are mounted 19:53:11 (1723247591) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec execute 153 sec latency 5488.131 ms oleg427-client.virtnet: 1 49952 1.45 MB/sec execute 154 sec latency 6488.282 ms oleg427-client.virtnet: 1 49952 1.44 MB/sec execute 155 sec latency 7488.452 ms oleg427-client.virtnet: 1 49952 1.44 MB/sec execute 156 sec latency 8488.651 ms oleg427-client.virtnet: 1 49952 1.43 MB/sec execute 157 sec latency 9488.835 ms oleg427-client.virtnet: 1 49952 1.42 MB/sec execute 158 sec latency 10489.026 ms oleg427-client.virtnet: 1 49952 1.41 MB/sec execute 159 sec latency 11489.190 ms oleg427-client.virtnet: 1 49952 1.40 MB/sec execute 160 sec latency 12489.375 ms oleg427-client.virtnet: 1 49952 1.39 MB/sec execute 161 sec latency 13489.571 ms oleg427-client.virtnet: 1 49952 1.38 MB/sec execute 162 sec latency 14489.768 ms oleg427-client.virtnet: 1 49952 1.37 MB/sec execute 163 sec latency 15489.939 ms oleg427-client.virtnet: 1 49952 1.37 MB/sec execute 164 sec latency 16490.112 ms oleg427-client.virtnet: 1 49952 1.36 MB/sec execute 165 sec latency 17490.296 ms oleg427-client.virtnet: 1 49952 1.35 MB/sec execute 166 sec latency 18490.467 ms oleg427-client.virtnet: 1 49952 1.34 MB/sec execute 167 sec latency 19490.653 ms oleg427-client.virtnet: 1 50130 1.33 MB/sec execute 168 sec latency 19755.942 ms oleg427-client.virtnet: 1 50404 1.33 MB/sec execute 169 sec latency 21.850 ms oleg427-client.virtnet: 1 50668 1.32 MB/sec execute 170 sec latency 17.560 ms oleg427-client.virtnet: 1 50993 1.33 MB/sec execute 171 sec latency 17.858 ms oleg427-client.virtnet: 1 51352 1.33 MB/sec execute 172 sec latency 16.790 ms oleg427-client.virtnet: 1 51697 1.33 MB/sec execute 173 sec latency 19.292 ms oleg427-client.virtnet: 1 52051 1.33 MB/sec execute 174 sec latency 13.707 ms oleg427-client.virtnet: 1 52464 1.34 MB/sec execute 175 sec latency 18.037 ms oleg427-client.virtnet: 1 53067 1.36 MB/sec execute 176 sec latency 17.638 ms oleg427-client.virtnet: 1 53490 1.37 MB/sec execute 177 sec latency 12.484 ms oleg427-client.virtnet: 1 53746 1.37 MB/sec execute 178 sec latency 15.318 ms oleg427-client.virtnet: 1 54063 1.36 MB/sec execute 179 sec latency 19.664 ms oleg427-client.virtnet: 1 54391 1.36 MB/sec execute 180 sec latency 18.209 ms oleg427-client.virtnet: 1 54755 1.37 MB/sec execute 181 sec latency 19.123 ms oleg427-client.virtnet: 1 55035 1.37 MB/sec execute 182 sec latency 18.835 ms oleg427-client.virtnet: 1 55403 1.36 MB/sec execute 183 sec latency 11.423 ms oleg427-client.virtnet: 1 55736 1.36 MB/sec execute 184 sec latency 17.500 ms oleg427-client.virtnet: 1 56424 1.39 MB/sec execute 185 sec latency 17.680 ms oleg427-client.virtnet: 1 56903 1.40 MB/sec execute 186 sec latency 18.110 ms oleg427-client.virtnet: 1 57168 1.40 MB/sec execute 187 sec latency 15.785 ms oleg427-client.virtnet: 1 57462 1.40 MB/sec execute 188 sec latency 18.736 ms oleg427-client.virtnet: 1 57742 1.39 MB/sec execute 189 sec latency 18.188 ms oleg427-client.virtnet: 1 58062 1.39 MB/sec execute 190 sec latency 17.224 ms oleg427-client.virtnet: 1 58460 1.40 MB/sec execute 191 sec latency 15.468 ms oleg427-client.virtnet: 1 58792 1.40 MB/sec execute 192 sec latency 19.156 ms oleg427-client.virtnet: 1 59159 1.40 MB/sec execute 193 sec latency 13.130 ms oleg427-client.virtnet: 1 59675 1.41 MB/sec execute 194 sec latency 17.688 ms oleg427-client.virtnet: 1 60303 1.44 MB/sec execute 195 sec latency 18.425 ms oleg427-client.virtnet: 1 60613 1.43 MB/sec execute 196 sec latency 17.363 ms oleg427-client.virtnet: 1 60858 1.43 MB/sec execute 197 sec latency 15.703 ms oleg427-clienUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36816 3569308 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12924 3593104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49740 7162412 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg427-server Stopping /mnt/lustre-mds2 (opts:) on oleg427-server 19:53:24 (1723247604) shut down Failover mds2 to oleg427-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0001 19:53:40 (1723247620) targets are mounted 19:53:40 (1723247620) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3570308 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13084 3593164 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163472 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg427-server Stopping /mnt/lustre-mds1 (opts:) on oleg427-server 19:53:53 (1723247633) shut down t.virtnet: 1 61204 1.43 MB/sec execute 198 sec latency 17.110 ms oleg427-client.virtnet: 1 61508 1.43 MB/sec execute 199 sec latency 18.347 ms oleg427-client.virtnet: 1 61888 1.43 MB/sec execute 200 sec latency 17.374 ms oleg427-client.virtnet: 1 62149 1.43 MB/sec execute 201 sec latency 19.956 ms oleg427-client.virtnet: 1 62503 1.42 MB/sec execute 202 sec latency 13.271 ms oleg427-client.virtnet: 1 62905 1.43 MB/sec execute 203 sec latency 16.846 ms oleg427-client.virtnet: 1 63584 1.45 MB/sec execute 204 sec latency 17.636 ms oleg427-client.virtnet: 1 64017 1.46 MB/sec execute 205 sec latency 46.761 ms oleg427-client.virtnet: 1 64017 1.45 MB/sec execute 206 sec latency 1046.949 ms oleg427-client.virtnet: 1 64017 1.45 MB/sec execute 207 sec latency 2047.124 ms oleg427-client.virtnet: 1 64017 1.44 MB/sec execute 208 sec latency 3047.324 ms oleg427-client.virtnet: 1 64017 1.43 MB/sec execute 209 sec latency 4047.527 ms oleg427-client.virtnet: 1 64017 1.43 MB/sec execute 210 sec latency 5047.709 ms oleg427-client.virtnet: 1 64017 1.42 MB/sec execute 211 sec latency 6047.887 ms oleg427-client.virtnet: 1 64017 1.41 MB/sec execute 212 sec latency 7048.054 ms oleg427-client.virtnet: 1 64017 1.41 MB/sec execute 213 sec latency 8048.233 ms oleg427-client.virtnet: 1 64017 1.40 MB/sec execute 214 sec latency 9048.401 ms oleg427-client.virtnet: 1 64017 1.39 MB/sec execute 215 sec latency 10048.555 ms oleg427-client.virtnet: 1 64017 1.39 MB/sec execute 216 sec latency 11048.717 ms oleg427-client.virtnet: 1 64017 1.38 MB/sec execute 217 sec latency 12048.896 ms oleg427-client.virtnet: 1 64017 1.37 MB/sec execute 218 sec latency 13049.103 ms oleg427-client.virtnet: 1 64017 1.37 MB/sec execute 219 sec latency 14049.267 ms oleg427-client.virtnet: 1 64017 1.36 MB/sec execute 220 sec latency 15049.452 ms oleg427-client.virtnet: 1 64017 1.35 MB/sec execute 221 sec latency 16049.661 ms oleg427-client.virtnet: 1 64017 1.35 MB/sec execute 222 sec latency 17049.812 ms oleg427-client.virtnet: 1 64017 1.34 MB/sec execute 223 sec latency 18050.019 ms oleg427-client.virtnet: 1 64017 1.34 MB/sec execute 224 sec latency 19050.169 ms oleg427-client.virtnet: 1 64181 1.33 MB/sec execute 225 sec latency 19556.399 ms oleg427-client.virtnet: 1 64423 1.33 MB/sec execute 226 sec latency 16.153 ms oleg427-client.virtnet: 1 64692 1.33 MB/sec execute 227 sec latency 21.124 ms oleg427-client.virtnet: 1 65005 1.33 MB/sec execute 228 sec latency 17.518 ms oleg427-client.virtnet: 1 65425 1.34 MB/sec execute 229 sec latency 17.681 ms oleg427-client.virtnet: 1 65690 1.33 MB/sec execute 230 sec latency 21.701 ms oleg427-client.virtnet: 1 66103 1.33 MB/sec execute 231 sec latency 12.839 ms oleg427-client.virtnet: 1 66466 1.33 MB/sec execute 232 sec latency 17.504 ms oleg427-client.virtnet: 1 67165 1.35 MB/sec execute 233 sec latency 18.222 ms oleg427-client.virtnet: 1 67625 1.37 MB/sec execute 234 sec latency 13.511 ms oleg427-client.virtnet: 1 67860 1.36 MB/sec execute 235 sec latency 15.993 ms oleg427-client.virtnet: 1 68165 1.36 MB/sec execute 236 sec latency 19.426 ms oleg427-client.virtnet: 1 68493 1.36 MB/sec execute 237 sec latency 17.886 ms oleg427-client.virtnet: 1 68914 1.37 MB/sec execute 238 sec latency 17.526 ms oleg427-client.virtnet: 1 69215 1.36 MB/sec execute 239 sec latency 15.727 ms oleg427-client.virtnet: 1 69558 1.36 MB/sec execute 240 sec latency 12.128 ms oleg427-client.virtnet: 1 69878 1.36 MB/sec execute 241 sec latency 18.054 ms oleg427-client.virtnet: 1 70417 1.37 MB/secFailover mds1 to oleg427-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 19:54:08 (1723247648) targets are mounted 19:54:08 (1723247648) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec execute 242 sec latency 17.715 ms oleg427-client.virtnet: 1 71036 1.39 MB/sec execute 243 sec latency 17.215 ms oleg427-client.virtnet: 1 71308 1.39 MB/sec execute 244 sec latency 15.887 ms oleg427-client.virtnet: 1 71615 1.38 MB/sec execute 245 sec latency 14.682 ms oleg427-client.virtnet: 1 71889 1.38 MB/sec execute 246 sec latency 17.554 ms oleg427-client.virtnet: 1 72213 1.38 MB/sec execute 247 sec latency 17.434 ms oleg427-client.virtnet: 1 72620 1.39 MB/sec execute 248 sec latency 17.326 ms oleg427-client.virtnet: 1 72954 1.38 MB/sec execute 249 sec latency 19.033 ms oleg427-client.virtnet: 1 73299 1.38 MB/sec execute 250 sec latency 13.198 ms oleg427-client.virtnet: 1 73746 1.39 MB/sec execute 251 sec latency 17.746 ms oleg427-client.virtnet: 1 74355 1.40 MB/sec execute 252 sec latency 18.926 ms oleg427-client.virtnet: 1 74765 1.41 MB/sec execute 253 sec latency 15.338 ms oleg427-client.virtnet: 1 74998 1.41 MB/sec execute 254 sec latency 16.285 ms oleg427-client.virtnet: 1 75317 1.41 MB/sec execute 255 sec latency 24.524 ms oleg427-client.virtnet: 1 75612 1.41 MB/sec execute 256 sec latency 17.610 ms oleg427-client.virtnet: 1 75988 1.41 MB/sec execute 257 sec latency 18.331 ms oleg427-client.virtnet: 1 76260 1.41 MB/sec execute 258 sec latency 20.059 ms oleg427-client.virtnet: 1 76565 1.40 MB/sec execute 259 sec latency 11.514 ms oleg427-client.virtnet: 1 76939 1.40 MB/sec execute 260 sec latency 16.939 ms oleg427-client.virtnet: 1 77454 1.42 MB/sec execute 261 sec latency 17.342 ms oleg427-client.virtnet: 1 78071 1.43 MB/sec execute 262 sec latency 17.755 ms oleg427-client.virtnet: 1 78368 1.43 MB/sec execute 263 sec latency 15.165 ms oleg427-client.virtnet: 1 78606 1.43 MB/sec execute 264 sec latency 15.384 ms oleg427-client.virtnet: 1 78922 1.43 MB/sec execute 265 sec latency 20.737 ms oleg427-client.virtnet: 1 79242 1.43 MB/sec execute 266 sec latency 17.862 ms oleg427-client.virtnet: 1 79606 1.43 MB/sec execute 267 sec latency 17.076 ms oleg427-client.virtnet: 1 79887 1.43 MB/sec execute 268 sec latency 18.815 ms oleg427-client.virtnet: 1 80231 1.42 MB/sec execute 269 sec latency 11.544 ms oleg427-client.virtnet: 1 80554 1.43 MB/sec execute 270 sec latency 17.806 ms oleg427-client.virtnet: 1 81095 1.44 MB/sec execute 271 sec latency 17.647 ms oleg427-client.virtnet: 1 81704 1.45 MB/sec execute 272 sec latency 17.620 ms oleg427-client.virtnet: 1 81976 1.45 MB/sec execute 273 sec latency 15.855 ms oleg427-client.virtnet: 1 82237 1.45 MB/sec execute 274 sec latency 15.671 ms oleg427-client.virtnet: 1 82534 1.44 MB/sec execute 275 sec latency 22.370 ms oleg427-client.virtnet: 1 82840 1.44 MB/sec execute 276 sec latency 17.715 ms oleg427-client.virtnet: 1 83205 1.45 MB/sec execute 277 sec latency 17.503 ms oleg427-client.virtnet: 1 83469 1.44 MB/sec execute 278 sec latency 18.987 ms oleg427-client.virtnet: 1 83807 1.44 MB/sec execute 279 sec latency 11.517 ms oleg427-client.virtnet: 1 84143 1.44 MB/sec execute 280 sec latency 17.680 ms oleg427-client.virtnet: 1 84819 1.46 MB/sec execute 281 sec latency 17.587 ms oleg427-client.virtnet: 1 85296 1.47 MB/sec execute 282 sec latency 17.883 ms oleg427-client.virtnet: 1 85560 1.47 MB/sec execute 283 sec latency 15.610 ms oleg427-client.virtnet: 1 85855 1.46 MB/sec execute 284 sec latency 20.969 ms oleg427-client.virtnet: 1 86120 1.46 MB/sec execute 285 sec latency 17.359 ms oleg427-client.virtnet: 1 86438 1.46 MB/sec execute 286 sec latency 17.752 ms oleg427-client.virtnet: 1 86801 1.47 MB/sec execute 287 sec latency 15.217 ms oleg427-client.virtnet: 1 87091 1.46 MB/sec execute 288 sec latency 18.909 ms oleg427-client.virtnet: 1 87473 1.46 MB/sec execute 289 sec latency 11.240 ms oleg427-client.virtnet: 1 87799 1.46 MB/sec execute 290 sec latency 17.495 ms oleg427-client.virtnet: 1 88527 1.48 MB/sec execute 291 sec latency 17.579 ms oleg427-client.virtnet: 1 88934 1.49 MB/sec execute 292 sec latency 13.689 ms oleg427-client.virtnet: 1 89173 1.48 MB/sec execute 293 sec latency 15.626 ms oleg427-client.virtnet: 1 89478 1.48 MB/sec execute 294 sec latency 22.476 ms oleg427-client.virtnet: 1 89776 1.48 MB/sec execute 295 sec latency 17.838 ms oleg427-client.virtnet: 1 90159 1.49 MB/sec execute 296 sec latency 17.830 ms oleg427-client.virtnet: 1 90445 1.48 MB/sec execute 297 sec latency 19.314 ms oleg427-client.virtnet: 1 90824 1.48 MB/sec execute 298 sec latency 11.741 ms oleg427-client.virtnet: 1 91164 1.48 MB/sec execute 299 sec latency 17.262 ms oleg427-client.virtnet: 1 cleanup 300 sec oleg427-client.virtnet: 0 cleanup 300 sec oleg427-client.virtnet: oleg427-client.virtnet: Operation Count AvgLat MaxLat oleg427-client.virtnet: ---------------------------------------- oleg427-client.virtnet: NTCreateX 14325 9.903 19755.922 oleg427-client.virtnet: Close 10539 1.756 5.682 oleg427-client.virtnet: Rename 606 46.196 19933.249 oleg427-client.virtnet: Unlink 2875 2.850 9.951 oleg427-client.virtnet: Qpathinfo 13020 5.373 19643.543 oleg427-client.virtnet: Qfileinfo 2282 0.199 2.643 oleg427-client.virtnet: Qfsinfo 2368 0.341 18.954 oleg427-client.virtnet: Sfileinfo 1166 8.023 18.071 oleg427-client.virtnet: Find 5015 0.597 16.076 oleg427-client.virtnet: WriteX 7092 1.314 8.391 oleg427-client.virtnet: ReadX 22528 0.024 1.254 oleg427-client.virtnet: LockX 48 1.697 2.083 oleg427-client.virtnet: UnlockX 48 1.892 2.195 oleg427-client.virtnet: Flush 999 9.122 536.067 oleg427-client.virtnet: oleg427-client.virtnet: Throughput 1.47729 MB/sec 1 clients 1 procs max_latency=19933.269 ms oleg427-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg427-client.virtnet at Fri Aug 9 19:54:59 EDT 2024 with return code 0 oleg427-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg427-client.virtnet oleg427-client.virtnet: /mnt/lustre/d70b.replay-single/oleg427-client.virtnet /mnt/lustre/d70b.replay-single/oleg427-client.virtnet oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg427-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg427-client.virtnet: removed directory: 'clients/client0' oleg427-client.virtnet: removed directory: 'clients' oleg427-client.virtnet: removed 'client.txt' oleg427-client.virtnet: /mnt/lustre/d70b.replay-single/oleg427-client.virtnet oleg427-client.virtnet: dbench successfully finished