== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 23:47:14 (1718855234) Starting client oleg341-client.virtnet: -o user_xattr,flock oleg341-server@tcp:/lustre /mnt/lustre Started clients oleg341-client.virtnet: 192.168.203.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg341-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=11720 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1285172 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3556144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3592872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7149016 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg341-server Stopping /mnt/lustre-mds1 (opts:) on oleg341-server 23:47:23 (1718855243) shut down Failover mds1 to oleg341-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 23:47:39 (1718855259) targets are mounted 23:47:39 (1718855259) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 27576 3568676 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1740 3592792 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 29316 7161468 1% /mnt/lustre oleg341-client.virtnet: looking for dbench program oleg341-client.virtnet: /usr/bin/dbench oleg341-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg341-client.virtnet oleg341-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg341-client.virtnet' oleg341-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg341-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg341-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg341-client.virtnet at Wed Jun 19 23:47:15 EDT 2024 oleg341-client.virtnet: waiting for dbench pid 11744 oleg341-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg341-client.virtnet: oleg341-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg341-client.virtnet: failed to create barrier semaphore oleg341-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg341-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg341-client.virtnet: releasing clients oleg341-client.virtnet: 1 305 11.22 MB/sec warmup 1 sec latency 23.036 ms oleg341-client.virtnet: 1 666 10.58 MB/sec warmup 2 sec latency 18.275 ms oleg341-client.virtnet: 1 955 7.27 MB/sec warmup 3 sec latency 16.002 ms oleg341-client.virtnet: 1 1410 6.50 MB/sec warmup 4 sec latency 18.995 ms oleg341-client.virtnet: 1 1698 5.24 MB/sec warmup 5 sec latency 85.463 ms oleg341-client.virtnet: 1 2087 4.44 MB/sec warmup 6 sec latency 11.403 ms oleg341-client.virtnet: 1 2441 4.01 MB/sec warmup 7 sec latency 33.630 ms oleg341-client.virtnet: 1 2441 3.51 MB/sec warmup 8 sec latency 1033.828 ms oleg341-client.virtnet: 1 2441 3.12 MB/sec warmup 9 sec latency 2034.037 ms oleg341-client.virtnet: 1 2441 2.81 MB/sec warmup 10 sec latency 3034.270 ms oleg341-client.virtnet: 1 2441 2.55 MB/sec warmup 11 sec latency 4034.484 ms oleg341-client.virtnet: 1 2441 2.34 MB/sec warmup 12 sec latency 5034.730 ms oleg341-client.virtnet: 1 2441 2.16 MB/sec warmup 13 sec latency 6034.882 ms oleg341-client.virtnet: 1 2441 2.01 MB/sec warmup 14 sec latency 7035.080 ms oleg341-client.virtnet: 1 2441 1.87 MB/sec warmup 15 sec latency 8035.330 ms oleg341-client.virtnet: 1 2441 1.76 MB/sec warmup 16 sec latency 9035.577 ms oleg341-client.virtnet: 1 2441 1.65 MB/sec warmup 17 sec latency 10035.793 ms oleg341-client.virtnet: 1 2441 1.56 MB/sec warmup 18 sec latency 11035.999 ms oleg341-client.virtnet: 1 2441 1.48 MB/sec warmup 19 sec latency 12036.201 ms oleg341-client.virtnet: 1 2441 1.41 MB/sec warmup 20 sec latency 13036.388 ms oleg341-client.virtnet: 1 2441 1.34 MB/sec warmup 21 sec latency 14036.530 ms oleg341-client.virtnet: 1 2441 1.28 MB/sec warmup 22 sec latency 15036.712 ms oleg341-client.virtnet: 1 2441 1.22 MB/sec warmup 23 sec latency 16036.861 ms oleg341-client.virtnet: 1 2441 1.17 MB/sec warmup 24 sec latency 17037.052 ms oleg341-client.virtnet: 1 2441 1.12 MB/sec warmup 25 sec latency 18037.289 ms oleg341-client.virtnet: 1 2441 1.08 MB/sec warmup 26 sec latency 19037.515 ms oleg341-client.virtnet: 1 2490 1.06 MB/sec warmup 27 sec latency 19845.587 ms oleg341-client.virtnet: 1 3176 1.22 MB/sec warmup 28 sec latency 18.792 ms oleg341-client.virtnet: 1 3685 1.34 MB/sec warmup 29 sec latency 15.685 ms oleg341-client.virtnet: 1 3949 1.34 MB/sec warmup 30 sec latency 16.665 ms oleg341-client.virtnet: 1 4243 1.30 MB/sec warmup 31 sec latency 19.497 ms oleg341-client.virtnet: 1 4526 1.28 MB/sec warmup 32 sec latency 17.345 ms oleg341-client.virtnet: 1 4955 1.37 MB/sec warmup 33 sec latencytest_70b fail mds2 2 times Failing mds2 on oleg341-server Stopping /mnt/lustre-mds2 (opts:) on oleg341-server 23:47:52 (1718855272) shut down Failover mds2 to oleg341-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0001 23:48:08 (1718855288) targets are mounted 23:48:08 (1718855288) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1285228 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36400 3569844 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13096 3592708 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7162552 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg341-server Stopping /mnt/lustre-mds1 (opts:) on oleg341-server 23:48:21 (1718855301) shut down Failover mds1 to oleg341-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 16.229 ms oleg341-client.virtnet: 1 5249 1.33 MB/sec warmup 34 sec latency 16.280 ms oleg341-client.virtnet: 1 5581 1.30 MB/sec warmup 35 sec latency 17.062 ms oleg341-client.virtnet: 1 5980 1.31 MB/sec warmup 36 sec latency 14.456 ms oleg341-client.virtnet: 1 6533 1.42 MB/sec warmup 37 sec latency 15.950 ms oleg341-client.virtnet: 1 7243 1.52 MB/sec warmup 38 sec latency 12.296 ms oleg341-client.virtnet: 1 7510 1.52 MB/sec warmup 39 sec latency 16.063 ms oleg341-client.virtnet: 1 7821 1.49 MB/sec warmup 40 sec latency 21.719 ms oleg341-client.virtnet: 1 8099 1.47 MB/sec warmup 41 sec latency 16.447 ms oleg341-client.virtnet: 1 8531 1.53 MB/sec warmup 42 sec latency 15.719 ms oleg341-client.virtnet: 1 8829 1.50 MB/sec warmup 43 sec latency 18.011 ms oleg341-client.virtnet: 1 9226 1.48 MB/sec warmup 44 sec latency 10.956 ms oleg341-client.virtnet: 1 9571 1.49 MB/sec warmup 45 sec latency 15.825 ms oleg341-client.virtnet: 1 10118 1.56 MB/sec warmup 46 sec latency 16.336 ms oleg341-client.virtnet: 1 10754 1.64 MB/sec warmup 47 sec latency 15.639 ms oleg341-client.virtnet: 1 11029 1.63 MB/sec warmup 48 sec latency 15.817 ms oleg341-client.virtnet: 1 11336 1.60 MB/sec warmup 49 sec latency 18.240 ms oleg341-client.virtnet: 1 11627 1.58 MB/sec warmup 50 sec latency 16.320 ms oleg341-client.virtnet: 1 12047 1.63 MB/sec warmup 51 sec latency 15.746 ms oleg341-client.virtnet: 1 12269 1.61 MB/sec warmup 52 sec latency 288.022 ms oleg341-client.virtnet: 1 12269 1.58 MB/sec warmup 53 sec latency 1288.282 ms oleg341-client.virtnet: 1 12269 1.55 MB/sec warmup 54 sec latency 2288.515 ms oleg341-client.virtnet: 1 12269 1.52 MB/sec warmup 55 sec latency 3288.741 ms oleg341-client.virtnet: 1 12488 1.49 MB/sec warmup 56 sec latency 3581.228 ms oleg341-client.virtnet: 1 12894 1.48 MB/sec warmup 57 sec latency 10.856 ms oleg341-client.virtnet: 1 13259 1.49 MB/sec warmup 58 sec latency 15.883 ms oleg341-client.virtnet: 1 13991 1.57 MB/sec warmup 59 sec latency 15.854 ms oleg341-client.virtnet: 1 14640 0.30 MB/sec execute 1 sec latency 15.891 ms oleg341-client.virtnet: 1 14969 0.35 MB/sec execute 2 sec latency 18.833 ms oleg341-client.virtnet: 1 15271 0.75 MB/sec execute 3 sec latency 16.511 ms oleg341-client.virtnet: 1 15670 1.35 MB/sec execute 4 sec latency 16.300 ms oleg341-client.virtnet: 1 15748 1.09 MB/sec execute 5 sec latency 779.792 ms oleg341-client.virtnet: 1 15748 0.91 MB/sec execute 6 sec latency 1780.068 ms oleg341-client.virtnet: 1 15748 0.78 MB/sec execute 7 sec latency 2780.318 ms oleg341-client.virtnet: 1 15748 0.68 MB/sec execute 8 sec latency 3780.570 ms oleg341-client.virtnet: 1 15748 0.61 MB/sec execute 9 sec latency 4780.820 ms oleg341-client.virtnet: 1 15748 0.55 MB/sec execute 10 sec latency 5781.051 ms oleg341-client.virtnet: 1 15748 0.50 MB/sec execute 11 sec latency 6781.266 ms oleg341-client.virtnet: 1 15748 0.45 MB/sec execute 12 sec latency 7781.420 ms oleg341-client.virtnet: 1 15748 0.42 MB/sec execute 13 sec latency 8781.623 ms oleg341-client.virtnet: 1 15748 0.39 MB/sec execute 14 sec latency 9781.786 ms oleg341-client.virtnet: 1 15748 0.36 MB/sec execute 15 sec latency 10781.967 ms oleg341-client.virtnet: 1 15748 0.34 MB/sec execute 16 sec latency 11782.167 ms oleg341-client.virtnet: 1 15748 0.32 MB/sec execute 17 sec latency 12782.401 ms oleg341-client.virtnet: 1 15748 0.30 MB/sec execute 18 sec latency 13782.639 ms oleg341-client.virtnet: 1 15748 0.29 MB/sec execute 19oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 23:48:36 (1718855316) targets are mounted 23:48:36 (1718855316) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36400 3569932 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13100 3592820 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162752 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg341-server Stopping /mnt/lustre-mds2 (opts:) on oleg341-server 23:48:49 (1718855329) shut down Failover mds2 to oleg341-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0001 23:49:05 (1718855345) targets are mounted 23:49:05 (1718855345) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1285228 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3570020 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13088 3592848 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7162868 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg341-server Stopping /mnt/lustre-mds1 (opts:) on oleg341-server 23:49:18 (1718855358) shut down sec latency 14782.834 ms oleg341-client.virtnet: 1 15748 0.27 MB/sec execute 20 sec latency 15782.980 ms oleg341-client.virtnet: 1 15748 0.26 MB/sec execute 21 sec latency 16783.237 ms oleg341-client.virtnet: 1 15748 0.25 MB/sec execute 22 sec latency 17783.447 ms oleg341-client.virtnet: 1 15748 0.24 MB/sec execute 23 sec latency 18783.617 ms oleg341-client.virtnet: 1 15748 0.23 MB/sec execute 24 sec latency 19783.839 ms oleg341-client.virtnet: 1 15986 0.23 MB/sec execute 25 sec latency 19974.222 ms oleg341-client.virtnet: 1 16368 0.23 MB/sec execute 26 sec latency 13.296 ms oleg341-client.virtnet: 1 16758 0.31 MB/sec execute 27 sec latency 15.396 ms oleg341-client.virtnet: 1 17404 0.49 MB/sec execute 28 sec latency 15.568 ms oleg341-client.virtnet: 1 17891 0.63 MB/sec execute 29 sec latency 12.844 ms oleg341-client.virtnet: 1 18158 0.65 MB/sec execute 30 sec latency 15.583 ms oleg341-client.virtnet: 1 18466 0.63 MB/sec execute 31 sec latency 18.132 ms oleg341-client.virtnet: 1 18741 0.64 MB/sec execute 32 sec latency 16.572 ms oleg341-client.virtnet: 1 19172 0.74 MB/sec execute 33 sec latency 15.602 ms oleg341-client.virtnet: 1 19521 0.73 MB/sec execute 34 sec latency 16.974 ms oleg341-client.virtnet: 1 19906 0.72 MB/sec execute 35 sec latency 11.757 ms oleg341-client.virtnet: 1 20262 0.76 MB/sec execute 36 sec latency 15.040 ms oleg341-client.virtnet: 1 20950 0.88 MB/sec execute 37 sec latency 15.212 ms oleg341-client.virtnet: 1 21448 0.98 MB/sec execute 38 sec latency 15.394 ms oleg341-client.virtnet: 1 21720 0.99 MB/sec execute 39 sec latency 15.994 ms oleg341-client.virtnet: 1 22028 0.97 MB/sec execute 40 sec latency 19.065 ms oleg341-client.virtnet: 1 22315 0.96 MB/sec execute 41 sec latency 16.302 ms oleg341-client.virtnet: 1 22752 1.04 MB/sec execute 42 sec latency 14.966 ms oleg341-client.virtnet: 1 23059 1.02 MB/sec execute 43 sec latency 16.968 ms oleg341-client.virtnet: 1 23448 1.01 MB/sec execute 44 sec latency 10.785 ms oleg341-client.virtnet: 1 23813 1.03 MB/sec execute 45 sec latency 14.654 ms oleg341-client.virtnet: 1 24500 1.13 MB/sec execute 46 sec latency 15.117 ms oleg341-client.virtnet: 1 25000 1.20 MB/sec execute 47 sec latency 15.756 ms oleg341-client.virtnet: 1 25270 1.20 MB/sec execute 48 sec latency 15.694 ms oleg341-client.virtnet: 1 25583 1.18 MB/sec execute 49 sec latency 47.062 ms oleg341-client.virtnet: 1 25583 1.16 MB/sec execute 50 sec latency 1047.242 ms oleg341-client.virtnet: 1 25583 1.14 MB/sec execute 51 sec latency 2047.485 ms oleg341-client.virtnet: 1 25583 1.11 MB/sec execute 52 sec latency 3047.669 ms oleg341-client.virtnet: 1 25748 1.10 MB/sec execute 53 sec latency 3546.050 ms oleg341-client.virtnet: 1 26062 1.11 MB/sec execute 54 sec latency 17.634 ms oleg341-client.virtnet: 1 26455 1.14 MB/sec execute 55 sec latency 16.357 ms oleg341-client.virtnet: 1 26750 1.13 MB/sec execute 56 sec latency 18.433 ms oleg341-client.virtnet: 1 27203 1.13 MB/sec execute 57 sec latency 10.779 ms oleg341-client.virtnet: 1 27636 1.18 MB/sec execute 58 sec latency 16.375 ms oleg341-client.virtnet: 1 27858 1.19 MB/sec execute 59 sec latency 618.014 ms oleg341-client.virtnet: 1 28478 1.26 MB/sec execute 60 sec latency 16.188 ms oleg341-client.virtnet: 1 28750 1.26 MB/sec execute 61 sec latency 15.599 ms oleg341-client.virtnet: 1 28804 1.24 MB/sec execute 62 sec latency 790.520 ms oleg341-client.virtnet: 1 28804 1.22 MB/sec execute 63 sec latency 1790.746 ms oleg341-client.virtnet: 1 28Failover mds1 to oleg341-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 23:49:33 (1718855373) targets are mounted 23:49:33 (1718855373) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36808 3569592 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12928 3591068 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49736 7160660 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg341-server Stopping /mnt/lustre-mds2 (opts:) on oleg341-server 23:49:46 (1718855386) shut down Failover mds2 to oleg341-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0001 23:50:01 (1718855401) targets are mounted 23:50:01 (1718855401) facet_failover done 804 1.20 MB/sec execute 64 sec latency 2790.990 ms oleg341-client.virtnet: 1 28804 1.18 MB/sec execute 65 sec latency 3791.243 ms oleg341-client.virtnet: 1 28804 1.16 MB/sec execute 66 sec latency 4791.573 ms oleg341-client.virtnet: 1 28804 1.15 MB/sec execute 67 sec latency 5791.856 ms oleg341-client.virtnet: 1 28804 1.13 MB/sec execute 68 sec latency 6792.160 ms oleg341-client.virtnet: 1 28804 1.11 MB/sec execute 69 sec latency 7792.408 ms oleg341-client.virtnet: 1 28804 1.10 MB/sec execute 70 sec latency 8792.657 ms oleg341-client.virtnet: 1 28804 1.08 MB/sec execute 71 sec latency 9792.951 ms oleg341-client.virtnet: 1 28804 1.07 MB/sec execute 72 sec latency 10793.245 ms oleg341-client.virtnet: 1 28804 1.05 MB/sec execute 73 sec latency 11793.448 ms oleg341-client.virtnet: 1 28804 1.04 MB/sec execute 74 sec latency 12793.706 ms oleg341-client.virtnet: 1 28804 1.02 MB/sec execute 75 sec latency 13793.950 ms oleg341-client.virtnet: 1 28804 1.01 MB/sec execute 76 sec latency 14794.100 ms oleg341-client.virtnet: 1 28804 1.00 MB/sec execute 77 sec latency 15794.252 ms oleg341-client.virtnet: 1 28804 0.98 MB/sec execute 78 sec latency 16794.471 ms oleg341-client.virtnet: 1 28804 0.97 MB/sec execute 79 sec latency 17794.779 ms oleg341-client.virtnet: 1 28804 0.96 MB/sec execute 80 sec latency 18795.087 ms oleg341-client.virtnet: 1 28804 0.95 MB/sec execute 81 sec latency 19795.305 ms oleg341-client.virtnet: 1 29022 0.94 MB/sec execute 82 sec latency 19847.812 ms oleg341-client.virtnet: 1 29331 0.93 MB/sec execute 83 sec latency 17.943 ms oleg341-client.virtnet: 1 29659 0.94 MB/sec execute 84 sec latency 15.844 ms oleg341-client.virtnet: 1 30032 0.97 MB/sec execute 85 sec latency 18.215 ms oleg341-client.virtnet: 1 30359 0.96 MB/sec execute 86 sec latency 20.749 ms oleg341-client.virtnet: 1 30808 0.96 MB/sec execute 87 sec latency 16.988 ms oleg341-client.virtnet: 1 31332 1.02 MB/sec execute 88 sec latency 15.659 ms oleg341-client.virtnet: 1 31968 1.06 MB/sec execute 89 sec latency 15.684 ms oleg341-client.virtnet: 1 32270 1.06 MB/sec execute 90 sec latency 15.663 ms oleg341-client.virtnet: 1 32489 1.05 MB/sec execute 91 sec latency 126.211 ms oleg341-client.virtnet: 1 32489 1.04 MB/sec execute 92 sec latency 1126.430 ms oleg341-client.virtnet: 1 32489 1.03 MB/sec execute 93 sec latency 2126.628 ms oleg341-client.virtnet: 1 32489 1.02 MB/sec execute 94 sec latency 3126.837 ms oleg341-client.virtnet: 1 32489 1.01 MB/sec execute 95 sec latency 4127.041 ms oleg341-client.virtnet: 1 32489 1.00 MB/sec execute 96 sec latency 5127.271 ms oleg341-client.virtnet: 1 32489 0.99 MB/sec execute 97 sec latency 6127.516 ms oleg341-client.virtnet: 1 32489 0.98 MB/sec execute 98 sec latency 7127.718 ms oleg341-client.virtnet: 1 32489 0.97 MB/sec execute 99 sec latency 8127.937 ms oleg341-client.virtnet: 1 32489 0.96 MB/sec execute 100 sec latency 9128.128 ms oleg341-client.virtnet: 1 32489 0.95 MB/sec execute 101 sec latency 10128.328 ms oleg341-client.virtnet: 1 32489 0.94 MB/sec execute 102 sec latency 11128.515 ms oleg341-client.virtnet: 1 32489 0.93 MB/sec execute 103 sec latency 12128.709 ms oleg341-client.virtnet: 1 32489 0.92 MB/sec execute 104 sec latency 13128.942 ms oleg341-client.virtnet: 1 32489 0.91 MB/sec execute 105 sec latency 14129.142 ms oleg341-client.virtnet: 1 32489 0.91 MB/sec execute 106 sec latency 15129.349 ms oleg341-client.virtnet: 1 32489 0.90 MB/sec execute 107 sec latency 16129.539 ms oleg341-client.virtnet: 1 32489oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36560 3570256 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12928 3593232 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163488 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg341-server Stopping /mnt/lustre-mds1 (opts:) on oleg341-server 23:50:14 (1718855414) shut down Failover mds1 to oleg341-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 23:50:29 (1718855429) targets are mounted 23:50:29 (1718855429) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36560 3569896 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12936 3593312 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7163208 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg341-server Stopping /mnt/lustre-mds2 (opts:) on oleg341-server 23:50:43 (1718855443) shut down 0.89 MB/sec execute 108 sec latency 17129.726 ms oleg341-client.virtnet: 1 32489 0.88 MB/sec execute 109 sec latency 18129.946 ms oleg341-client.virtnet: 1 32823 0.88 MB/sec execute 110 sec latency 18266.254 ms oleg341-client.virtnet: 1 33156 0.88 MB/sec execute 111 sec latency 15.226 ms oleg341-client.virtnet: 1 33557 0.90 MB/sec execute 112 sec latency 16.623 ms oleg341-client.virtnet: 1 33935 0.90 MB/sec execute 113 sec latency 17.632 ms oleg341-client.virtnet: 1 34342 0.90 MB/sec execute 114 sec latency 12.975 ms oleg341-client.virtnet: 1 34731 0.93 MB/sec execute 115 sec latency 159.565 ms oleg341-client.virtnet: 1 35326 0.95 MB/sec execute 116 sec latency 415.829 ms oleg341-client.virtnet: 1 35791 0.98 MB/sec execute 117 sec latency 16.219 ms oleg341-client.virtnet: 1 35989 0.98 MB/sec execute 118 sec latency 175.038 ms oleg341-client.virtnet: 1 35989 0.97 MB/sec execute 119 sec latency 1175.305 ms oleg341-client.virtnet: 1 35989 0.96 MB/sec execute 120 sec latency 2175.519 ms oleg341-client.virtnet: 1 35989 0.95 MB/sec execute 121 sec latency 3175.736 ms oleg341-client.virtnet: 1 35989 0.94 MB/sec execute 122 sec latency 4175.930 ms oleg341-client.virtnet: 1 35989 0.94 MB/sec execute 123 sec latency 5176.146 ms oleg341-client.virtnet: 1 35989 0.93 MB/sec execute 124 sec latency 6176.409 ms oleg341-client.virtnet: 1 35989 0.92 MB/sec execute 125 sec latency 7176.634 ms oleg341-client.virtnet: 1 35989 0.91 MB/sec execute 126 sec latency 8176.865 ms oleg341-client.virtnet: 1 35989 0.91 MB/sec execute 127 sec latency 9177.069 ms oleg341-client.virtnet: 1 35989 0.90 MB/sec execute 128 sec latency 10177.336 ms oleg341-client.virtnet: 1 35989 0.89 MB/sec execute 129 sec latency 11177.529 ms oleg341-client.virtnet: 1 35989 0.89 MB/sec execute 130 sec latency 12177.727 ms oleg341-client.virtnet: 1 35989 0.88 MB/sec execute 131 sec latency 13177.956 ms oleg341-client.virtnet: 1 35989 0.87 MB/sec execute 132 sec latency 14178.223 ms oleg341-client.virtnet: 1 35989 0.87 MB/sec execute 133 sec latency 15178.440 ms oleg341-client.virtnet: 1 35989 0.86 MB/sec execute 134 sec latency 16178.603 ms oleg341-client.virtnet: 1 35989 0.85 MB/sec execute 135 sec latency 17178.780 ms oleg341-client.virtnet: 1 35989 0.85 MB/sec execute 136 sec latency 18179.028 ms oleg341-client.virtnet: 1 35989 0.84 MB/sec execute 137 sec latency 19179.244 ms oleg341-client.virtnet: 1 36097 0.83 MB/sec execute 138 sec latency 19776.981 ms oleg341-client.virtnet: 1 36407 0.83 MB/sec execute 139 sec latency 17.132 ms oleg341-client.virtnet: 1 36715 0.84 MB/sec execute 140 sec latency 16.317 ms oleg341-client.virtnet: 1 37094 0.85 MB/sec execute 141 sec latency 15.672 ms oleg341-client.virtnet: 1 37416 0.85 MB/sec execute 142 sec latency 16.763 ms oleg341-client.virtnet: 1 37854 0.85 MB/sec execute 143 sec latency 11.654 ms oleg341-client.virtnet: 1 38275 0.88 MB/sec execute 144 sec latency 16.685 ms oleg341-client.virtnet: 1 38881 0.89 MB/sec execute 145 sec latency 16.034 ms oleg341-client.virtnet: 1 39296 0.92 MB/sec execute 146 sec latency 15.435 ms oleg341-client.virtnet: 1 39547 0.91 MB/sec execute 147 sec latency 16.343 ms oleg341-client.virtnet: 1 39887 0.91 MB/sec execute 148 sec latency 18.456 ms oleg341-client.virtnet: 1 40225 0.91 MB/sec execute 149 sec latency 15.967 ms oleg341-client.virtnet: 1 40624 0.93 MB/sec execute 150 sec latency 15.699 ms oleg341-client.virtnet: 1 40933 0.92 MB/sec execute 151 sec latency 18.039 ms oleg341-client.virtnet: 1 41360 0.93 MB/sec execute 152Failover mds2 to oleg341-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0001 23:50:58 (1718855458) targets are mounted 23:50:58 (1718855458) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1285228 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36728 3569176 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12928 3593164 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49656 7162340 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg341-server Stopping /mnt/lustre-mds1 (opts:) on oleg341-server 23:51:10 (1718855470) shut down Failover mds1 to oleg341-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 23:51:26 (1718855486) targets are mounted 23:51:26 (1718855486) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec sec latency 12.886 ms oleg341-client.virtnet: 1 41814 0.95 MB/sec execute 153 sec latency 16.514 ms oleg341-client.virtnet: 1 42432 0.97 MB/sec execute 154 sec latency 15.329 ms oleg341-client.virtnet: 1 42851 0.99 MB/sec execute 155 sec latency 15.619 ms oleg341-client.virtnet: 1 43085 0.98 MB/sec execute 156 sec latency 15.455 ms oleg341-client.virtnet: 1 43419 0.98 MB/sec execute 157 sec latency 18.396 ms oleg341-client.virtnet: 1 43765 0.98 MB/sec execute 158 sec latency 15.546 ms oleg341-client.virtnet: 1 44173 1.00 MB/sec execute 159 sec latency 14.208 ms oleg341-client.virtnet: 1 44489 0.99 MB/sec execute 160 sec latency 16.634 ms oleg341-client.virtnet: 1 44967 1.00 MB/sec execute 161 sec latency 12.125 ms oleg341-client.virtnet: 1 45499 1.02 MB/sec execute 162 sec latency 15.531 ms oleg341-client.virtnet: 1 45784 1.02 MB/sec execute 163 sec latency 579.958 ms oleg341-client.virtnet: 1 45784 1.02 MB/sec execute 164 sec latency 1580.159 ms oleg341-client.virtnet: 1 45784 1.01 MB/sec execute 165 sec latency 2580.344 ms oleg341-client.virtnet: 1 45937 1.01 MB/sec execute 166 sec latency 3446.616 ms oleg341-client.virtnet: 1 46388 1.03 MB/sec execute 167 sec latency 15.519 ms oleg341-client.virtnet: 1 46616 1.03 MB/sec execute 168 sec latency 15.637 ms oleg341-client.virtnet: 1 46962 1.02 MB/sec execute 169 sec latency 16.884 ms oleg341-client.virtnet: 1 47297 1.03 MB/sec execute 170 sec latency 16.664 ms oleg341-client.virtnet: 1 47706 1.04 MB/sec execute 171 sec latency 15.652 ms oleg341-client.virtnet: 1 48206 1.04 MB/sec execute 172 sec latency 10.724 ms oleg341-client.virtnet: 1 48587 1.04 MB/sec execute 173 sec latency 15.776 ms oleg341-client.virtnet: 1 49138 1.06 MB/sec execute 174 sec latency 15.164 ms oleg341-client.virtnet: 1 49313 1.06 MB/sec execute 175 sec latency 857.139 ms oleg341-client.virtnet: 1 49313 1.05 MB/sec execute 176 sec latency 1857.373 ms oleg341-client.virtnet: 1 49313 1.05 MB/sec execute 177 sec latency 2857.608 ms oleg341-client.virtnet: 1 49313 1.04 MB/sec execute 178 sec latency 3857.803 ms oleg341-client.virtnet: 1 49313 1.04 MB/sec execute 179 sec latency 4858.003 ms oleg341-client.virtnet: 1 49313 1.03 MB/sec execute 180 sec latency 5858.282 ms oleg341-client.virtnet: 1 49313 1.03 MB/sec execute 181 sec latency 6858.607 ms oleg341-client.virtnet: 1 49313 1.02 MB/sec execute 182 sec latency 7858.832 ms oleg341-client.virtnet: 1 49313 1.01 MB/sec execute 183 sec latency 8859.183 ms oleg341-client.virtnet: 1 49313 1.01 MB/sec execute 184 sec latency 9859.483 ms oleg341-client.virtnet: 1 49313 1.00 MB/sec execute 185 sec latency 10859.693 ms oleg341-client.virtnet: 1 49313 1.00 MB/sec execute 186 sec latency 11859.899 ms oleg341-client.virtnet: 1 49313 0.99 MB/sec execute 187 sec latency 12860.158 ms oleg341-client.virtnet: 1 49313 0.99 MB/sec execute 188 sec latency 13860.382 ms oleg341-client.virtnet: 1 49313 0.98 MB/sec execute 189 sec latency 14860.616 ms oleg341-client.virtnet: 1 49313 0.98 MB/sec execute 190 sec latency 15860.827 ms oleg341-client.virtnet: 1 49313 0.97 MB/sec execute 191 sec latency 16861.138 ms oleg341-client.virtnet: 1 49313 0.97 MB/sec execute 192 sec latency 17861.380 ms oleg341-client.virtnet: 1 49313 0.96 MB/sec execute 193 sec latency 18861.658 ms oleg341-client.virtnet: 1 49315 0.96 MB/sec execute 194 sec latency 19851.796 ms oleg341-client.virtnet: 1 49798 0.98 MB/sec execute 195 sec latency 13.414 ms oleg341-client.virtnet: 1 50063 0.98 MB/sec execute 196 sec latency 15.901 ms oleg341-clieUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36592 3567880 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13044 3592380 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160260 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg341-server Stopping /mnt/lustre-mds2 (opts:) on oleg341-server 23:51:39 (1718855499) shut down Failover mds2 to oleg341-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0001 23:51:55 (1718855515) targets are mounted 23:51:55 (1718855515) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1285228 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36536 3569560 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13100 3589744 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7159304 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg341-server Stopping /mnt/lustre-mds1 (opts:) on oleg341-server 23:52:08 (1718855528) shut down nt.virtnet: 1 50378 0.97 MB/sec execute 197 sec latency 16.157 ms oleg341-client.virtnet: 1 50773 0.98 MB/sec execute 198 sec latency 13.098 ms oleg341-client.virtnet: 1 51183 0.99 MB/sec execute 199 sec latency 15.331 ms oleg341-client.virtnet: 1 51485 0.98 MB/sec execute 200 sec latency 17.702 ms oleg341-client.virtnet: 1 51860 0.98 MB/sec execute 201 sec latency 11.712 ms oleg341-client.virtnet: 1 52255 0.99 MB/sec execute 202 sec latency 15.281 ms oleg341-client.virtnet: 1 52918 1.01 MB/sec execute 203 sec latency 17.956 ms oleg341-client.virtnet: 1 53393 1.03 MB/sec execute 204 sec latency 12.961 ms oleg341-client.virtnet: 1 53656 1.03 MB/sec execute 205 sec latency 15.804 ms oleg341-client.virtnet: 1 53974 1.02 MB/sec execute 206 sec latency 16.546 ms oleg341-client.virtnet: 1 54296 1.03 MB/sec execute 207 sec latency 16.143 ms oleg341-client.virtnet: 1 54700 1.04 MB/sec execute 208 sec latency 16.694 ms oleg341-client.virtnet: 1 54999 1.03 MB/sec execute 209 sec latency 17.961 ms oleg341-client.virtnet: 1 55370 1.03 MB/sec execute 210 sec latency 13.655 ms oleg341-client.virtnet: 1 55719 1.04 MB/sec execute 211 sec latency 15.826 ms oleg341-client.virtnet: 1 56404 1.06 MB/sec execute 212 sec latency 15.643 ms oleg341-client.virtnet: 1 56899 1.07 MB/sec execute 213 sec latency 15.676 ms oleg341-client.virtnet: 1 57178 1.07 MB/sec execute 214 sec latency 16.364 ms oleg341-client.virtnet: 1 57502 1.07 MB/sec execute 215 sec latency 15.886 ms oleg341-client.virtnet: 1 57799 1.07 MB/sec execute 216 sec latency 16.074 ms oleg341-client.virtnet: 1 58234 1.08 MB/sec execute 217 sec latency 15.610 ms oleg341-client.virtnet: 1 58565 1.08 MB/sec execute 218 sec latency 18.938 ms oleg341-client.virtnet: 1 58947 1.08 MB/sec execute 219 sec latency 10.948 ms oleg341-client.virtnet: 1 58947 1.07 MB/sec execute 220 sec latency 1006.263 ms oleg341-client.virtnet: 1 58947 1.07 MB/sec execute 221 sec latency 2006.458 ms oleg341-client.virtnet: 1 58947 1.06 MB/sec execute 222 sec latency 3006.648 ms oleg341-client.virtnet: 1 59189 1.06 MB/sec execute 223 sec latency 3395.546 ms oleg341-client.virtnet: 1 59710 1.08 MB/sec execute 224 sec latency 16.636 ms oleg341-client.virtnet: 1 60336 1.10 MB/sec execute 225 sec latency 16.152 ms oleg341-client.virtnet: 1 60623 1.10 MB/sec execute 226 sec latency 16.440 ms oleg341-client.virtnet: 1 60864 1.10 MB/sec execute 227 sec latency 16.209 ms oleg341-client.virtnet: 1 61212 1.09 MB/sec execute 228 sec latency 66.909 ms oleg341-client.virtnet: 1 61321 1.09 MB/sec execute 229 sec latency 638.935 ms oleg341-client.virtnet: 1 61740 1.10 MB/sec execute 230 sec latency 16.169 ms oleg341-client.virtnet: 1 62048 1.10 MB/sec execute 231 sec latency 16.882 ms oleg341-client.virtnet: 1 62118 1.10 MB/sec execute 232 sec latency 794.383 ms oleg341-client.virtnet: 1 62118 1.09 MB/sec execute 233 sec latency 1794.633 ms oleg341-client.virtnet: 1 62118 1.09 MB/sec execute 234 sec latency 2794.823 ms oleg341-client.virtnet: 1 62118 1.08 MB/sec execute 235 sec latency 3795.052 ms oleg341-client.virtnet: 1 62118 1.08 MB/sec execute 236 sec latency 4795.281 ms oleg341-client.virtnet: 1 62118 1.07 MB/sec execute 237 sec latency 5795.561 ms oleg341-client.virtnet: 1 62118 1.07 MB/sec execute 238 sec latency 6795.821 ms oleg341-client.virtnet: 1 62118 1.06 MB/sec execute 239 sec latency 7796.082 ms oleg341-client.virtnet: 1 62118 1.06 MB/sec execute 240 sec latency 8796.320 ms oleg341-client.virtnet: 1 62118 1.05 MB/sec execute 241 sec latenFailover mds1 to oleg341-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 23:52:23 (1718855543) targets are mounted 23:52:23 (1718855543) facet_failover done oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec cy 9796.622 ms oleg341-client.virtnet: 1 62118 1.05 MB/sec execute 242 sec latency 10796.871 ms oleg341-client.virtnet: 1 62118 1.05 MB/sec execute 243 sec latency 11797.085 ms oleg341-client.virtnet: 1 62118 1.04 MB/sec execute 244 sec latency 12797.297 ms oleg341-client.virtnet: 1 62118 1.04 MB/sec execute 245 sec latency 13797.532 ms oleg341-client.virtnet: 1 62118 1.03 MB/sec execute 246 sec latency 14797.766 ms oleg341-client.virtnet: 1 62118 1.03 MB/sec execute 247 sec latency 15797.975 ms oleg341-client.virtnet: 1 62118 1.03 MB/sec execute 248 sec latency 16798.169 ms oleg341-client.virtnet: 1 62118 1.02 MB/sec execute 249 sec latency 17798.463 ms oleg341-client.virtnet: 1 62118 1.02 MB/sec execute 250 sec latency 18798.655 ms oleg341-client.virtnet: 1 62130 1.01 MB/sec execute 251 sec latency 19726.839 ms oleg341-client.virtnet: 1 62517 1.01 MB/sec execute 252 sec latency 11.217 ms oleg341-client.virtnet: 1 62901 1.02 MB/sec execute 253 sec latency 15.859 ms oleg341-client.virtnet: 1 63558 1.03 MB/sec execute 254 sec latency 15.833 ms oleg341-client.virtnet: 1 64017 1.05 MB/sec execute 255 sec latency 13.221 ms oleg341-client.virtnet: 1 64275 1.05 MB/sec execute 256 sec latency 16.226 ms oleg341-client.virtnet: 1 64591 1.04 MB/sec execute 257 sec latency 16.201 ms oleg341-client.virtnet: 1 64869 1.04 MB/sec execute 258 sec latency 16.119 ms oleg341-client.virtnet: 1 65297 1.05 MB/sec execute 259 sec latency 15.609 ms oleg341-client.virtnet: 1 65599 1.05 MB/sec execute 260 sec latency 16.970 ms oleg341-client.virtnet: 1 65977 1.05 MB/sec execute 261 sec latency 11.122 ms oleg341-client.virtnet: 1 66345 1.05 MB/sec execute 262 sec latency 15.124 ms oleg341-client.virtnet: 1 67039 1.07 MB/sec execute 263 sec latency 15.915 ms oleg341-client.virtnet: 1 67538 1.08 MB/sec execute 264 sec latency 14.901 ms oleg341-client.virtnet: 1 67805 1.08 MB/sec execute 265 sec latency 15.830 ms oleg341-client.virtnet: 1 68127 1.08 MB/sec execute 266 sec latency 22.007 ms oleg341-client.virtnet: 1 68410 1.08 MB/sec execute 267 sec latency 16.126 ms oleg341-client.virtnet: 1 68830 1.09 MB/sec execute 268 sec latency 16.026 ms oleg341-client.virtnet: 1 69139 1.09 MB/sec execute 269 sec latency 17.698 ms oleg341-client.virtnet: 1 69506 1.08 MB/sec execute 270 sec latency 10.778 ms oleg341-client.virtnet: 1 69894 1.09 MB/sec execute 271 sec latency 15.677 ms oleg341-client.virtnet: 1 70672 1.11 MB/sec execute 272 sec latency 15.726 ms oleg341-client.virtnet: 1 71122 1.12 MB/sec execute 273 sec latency 13.065 ms oleg341-client.virtnet: 1 71388 1.12 MB/sec execute 274 sec latency 16.427 ms oleg341-client.virtnet: 1 71697 1.11 MB/sec execute 275 sec latency 20.363 ms oleg341-client.virtnet: 1 72022 1.12 MB/sec execute 276 sec latency 16.371 ms oleg341-client.virtnet: 1 72493 1.12 MB/sec execute 277 sec latency 13.196 ms oleg341-client.virtnet: 1 72804 1.12 MB/sec execute 278 sec latency 17.337 ms oleg341-client.virtnet: 1 73179 1.12 MB/sec execute 279 sec latency 11.116 ms oleg341-client.virtnet: 1 73562 1.12 MB/sec execute 280 sec latency 16.378 ms oleg341-client.virtnet: 1 74202 1.14 MB/sec execute 281 sec latency 16.023 ms oleg341-client.virtnet: 1 74704 1.15 MB/sec execute 282 sec latency 15.487 ms oleg341-client.virtnet: 1 75010 1.15 MB/sec execute 283 sec latency 14.393 ms oleg341-client.virtnet: 1 75341 1.15 MB/sec execute 284 sec latency 18.061 ms oleg341-client.virtnet: 1 75675 1.15 MB/sec execute 285 sec latency 15.794 ms oleg341-client.virtnet: 1 76054 1.16 MB/sec execute 286 sec latency 15.352 ms oleg341-client.virtnet: 1 76370 1.15 MB/sec execute 287 sec latency 18.789 ms oleg341-client.virtnet: 1 76751 1.15 MB/sec execute 288 sec latency 12.312 ms oleg341-client.virtnet: 1 77137 1.15 MB/sec execute 289 sec latency 15.955 ms oleg341-client.virtnet: 1 77816 1.17 MB/sec execute 290 sec latency 16.408 ms oleg341-client.virtnet: 1 78255 1.18 MB/sec execute 291 sec latency 13.322 ms oleg341-client.virtnet: 1 78509 1.18 MB/sec execute 292 sec latency 16.317 ms oleg341-client.virtnet: 1 78819 1.18 MB/sec execute 293 sec latency 22.727 ms oleg341-client.virtnet: 1 79105 1.18 MB/sec execute 294 sec latency 16.229 ms oleg341-client.virtnet: 1 79527 1.19 MB/sec execute 295 sec latency 15.986 ms oleg341-client.virtnet: 1 79832 1.18 MB/sec execute 296 sec latency 17.498 ms oleg341-client.virtnet: 1 80224 1.18 MB/sec execute 297 sec latency 11.830 ms oleg341-client.virtnet: 1 80567 1.18 MB/sec execute 298 sec latency 15.568 ms oleg341-client.virtnet: 1 81102 1.19 MB/sec execute 299 sec latency 20.779 ms oleg341-client.virtnet: 1 cleanup 300 sec oleg341-client.virtnet: 0 cleanup 300 sec oleg341-client.virtnet: oleg341-client.virtnet: Operation Count AvgLat MaxLat oleg341-client.virtnet: ---------------------------------------- oleg341-client.virtnet: NTCreateX 11653 13.746 19851.773 oleg341-client.virtnet: Close 8557 6.473 19974.207 oleg341-client.virtnet: Rename 492 11.603 18.922 oleg341-client.virtnet: Unlink 2353 2.976 19.374 oleg341-client.virtnet: Qpathinfo 10554 3.915 18266.230 oleg341-client.virtnet: Qfileinfo 1841 0.202 2.718 oleg341-client.virtnet: Qfsinfo 1935 0.336 7.022 oleg341-client.virtnet: Sfileinfo 950 9.955 3546.036 oleg341-client.virtnet: Find 4082 0.623 12.222 oleg341-client.virtnet: WriteX 5735 1.347 7.810 oleg341-client.virtnet: ReadX 18288 0.024 2.264 oleg341-client.virtnet: LockX 38 1.897 2.317 oleg341-client.virtnet: UnlockX 38 1.974 2.194 oleg341-client.virtnet: Flush 817 10.387 638.915 oleg341-client.virtnet: oleg341-client.virtnet: Throughput 1.19437 MB/sec 1 clients 1 procs max_latency=19974.222 ms oleg341-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg341-client.virtnet at Wed Jun 19 23:53:16 EDT 2024 with return code 0 oleg341-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg341-client.virtnet oleg341-client.virtnet: /mnt/lustre/d70b.replay-single/oleg341-client.virtnet /mnt/lustre/d70b.replay-single/oleg341-client.virtnet oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg341-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg341-client.virtnet: removed directory: 'clients/client0' oleg341-client.virtnet: removed directory: 'clients' oleg341-client.virtnet: removed 'client.txt' oleg341-client.virtnet: /mnt/lustre/d70b.replay-single/oleg341-client.virtnet oleg341-client.virtnet: dbench successfully finished