== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 04:59:46 (1715590786) Starting client oleg413-client.virtnet: -o user_xattr,flock oleg413-server@tcp:/lustre /mnt/lustre Started clients oleg413-client.virtnet: 192.168.204.113@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg413-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=21766 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1744 3592560 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26136 3556164 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27880 7148724 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 04:59:54 (1715590794) shut down Failover mds1 to oleg413-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 05:00:08 (1715590808) targets are mounted 05:00:08 (1715590808) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3593220 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26556 3569508 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28140 7162728 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg413-server Stopping /mnt/lustre-mds2 (opts:) on oleg413-server 05:00:19 (1715590819) shut down oleg413-client.virtnet: looking for dbench program oleg413-client.virtnet: /usr/bin/dbench oleg413-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg413-client.virtnet oleg413-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg413-client.virtnet' oleg413-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg413-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg413-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg413-client.virtnet at Mon May 13 04:59:47 EDT 2024 oleg413-client.virtnet: waiting for dbench pid 21790 oleg413-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg413-client.virtnet: oleg413-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg413-client.virtnet: failed to create barrier semaphore oleg413-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg413-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg413-client.virtnet: releasing clients oleg413-client.virtnet: 1 493 17.21 MB/sec warmup 1 sec latency 13.561 ms oleg413-client.virtnet: 1 998 10.96 MB/sec warmup 2 sec latency 12.734 ms oleg413-client.virtnet: 1 1616 8.72 MB/sec warmup 3 sec latency 11.833 ms oleg413-client.virtnet: 1 2063 6.66 MB/sec warmup 4 sec latency 211.204 ms oleg413-client.virtnet: 1 2929 6.70 MB/sec warmup 5 sec latency 9.104 ms oleg413-client.virtnet: 1 3804 6.63 MB/sec warmup 6 sec latency 70.756 ms oleg413-client.virtnet: 1 3804 5.68 MB/sec warmup 7 sec latency 1070.913 ms oleg413-client.virtnet: 1 3804 4.97 MB/sec warmup 8 sec latency 2071.125 ms oleg413-client.virtnet: 1 3804 4.42 MB/sec warmup 9 sec latency 3071.311 ms oleg413-client.virtnet: 1 3804 3.98 MB/sec warmup 10 sec latency 4071.627 ms oleg413-client.virtnet: 1 3804 3.61 MB/sec warmup 11 sec latency 5071.963 ms oleg413-client.virtnet: 1 3804 3.31 MB/sec warmup 12 sec latency 6072.171 ms oleg413-client.virtnet: 1 3804 3.06 MB/sec warmup 13 sec latency 7072.367 ms oleg413-client.virtnet: 1 3804 2.84 MB/sec warmup 14 sec latency 8072.548 ms oleg413-client.virtnet: 1 3804 2.65 MB/sec warmup 15 sec latency 9072.729 ms oleg413-client.virtnet: 1 3804 2.48 MB/sec warmup 16 sec latency 10072.889 ms oleg413-client.virtnet: 1 3804 2.34 MB/sec warmup 17 sec latency 11073.082 ms oleg413-client.virtnet: 1 3804 2.21 MB/sec warmup 18 sec latency 12073.212 ms oleg413-client.virtnet: 1 3804 2.09 MB/sec warmup 19 sec latency 13073.368 ms oleg413-client.virtnet: 1 3804 1.99 MB/sec warmup 20 sec latency 14073.547 ms oleg413-client.virtnet: 1 3804 1.89 MB/sec warmup 21 sec latency 15073.759 ms oleg413-client.virtnet: 1 3804 1.81 MB/sec warmup 22 sec latency 16073.964 ms oleg413-client.virtnet: 1 3804 1.73 MB/sec warmup 23 sec latency 17074.137 ms oleg413-client.virtnet: 1 3804 1.66 MB/sec warmup 24 sec latency 18074.268 ms oleg413-client.virtnet: 1 4154 1.60 MB/sec warmup 25 sec latency 18287.386 ms oleg413-client.virtnet: 1 4787 1.62 MB/sec warmup 26 sec latency 10.292 ms oleg413-client.virtnet: 1 5469 1.69 MB/sec warmup 27 sec latency 9.238 ms oleg413-client.virtnet: 1 6134 1.72 MB/sec warmup 28 sec latency 11.883 ms oleg413-client.virtnet: 1 7209 1.99 MB/sec warmup 29 sec latency 11.722 ms oleg413-client.virtnet: 1 7730 1.97 MB/sec warmup 30 sec latency 10.504 ms oleg413-client.virtnet: 1 8281 1.98 MB/sec warmup 31 sec latency 12.071 ms oleg413-client.virtnet: 1 8893 2.02 MB/sec warmup 32 sec latency 10.980 ms oleg413-client.virtnet: 1 9591 2.03 MB/sec warmup 33 sec latency 8.5Failover mds2 to oleg413-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0001 05:00:33 (1715590833) targets are mounted 05:00:33 (1715590833) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12952 3593180 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36548 3569820 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7163000 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 05:00:45 (1715590845) shut down Failover mds1 to oleg413-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 05:00:59 (1715590859) targets are mounted 05:00:59 (1715590859) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 91 ms oleg413-client.virtnet: 1 10679 2.25 MB/sec warmup 34 sec latency 9.513 ms oleg413-client.virtnet: 1 11232 2.24 MB/sec warmup 35 sec latency 10.261 ms oleg413-client.virtnet: 1 11848 2.23 MB/sec warmup 36 sec latency 10.186 ms oleg413-client.virtnet: 1 12499 2.26 MB/sec warmup 37 sec latency 10.409 ms oleg413-client.virtnet: 1 13239 2.27 MB/sec warmup 38 sec latency 11.186 ms oleg413-client.virtnet: 1 14399 2.49 MB/sec warmup 39 sec latency 7.854 ms oleg413-client.virtnet: 1 14879 2.44 MB/sec warmup 40 sec latency 12.449 ms oleg413-client.virtnet: 1 15418 2.43 MB/sec warmup 41 sec latency 9.655 ms oleg413-client.virtnet: 1 16089 2.45 MB/sec warmup 42 sec latency 10.058 ms oleg413-client.virtnet: 1 16963 2.52 MB/sec warmup 43 sec latency 8.343 ms oleg413-client.virtnet: 1 17964 2.64 MB/sec warmup 44 sec latency 9.124 ms oleg413-client.virtnet: 1 18495 2.60 MB/sec warmup 45 sec latency 10.004 ms oleg413-client.virtnet: 1 18607 2.55 MB/sec warmup 46 sec latency 825.280 ms oleg413-client.virtnet: 1 18607 2.49 MB/sec warmup 47 sec latency 1825.396 ms oleg413-client.virtnet: 1 18607 2.44 MB/sec warmup 48 sec latency 2825.601 ms oleg413-client.virtnet: 1 18607 2.39 MB/sec warmup 49 sec latency 3825.790 ms oleg413-client.virtnet: 1 18814 2.37 MB/sec warmup 50 sec latency 4407.363 ms oleg413-client.virtnet: 1 19403 2.39 MB/sec warmup 51 sec latency 10.157 ms oleg413-client.virtnet: 1 20113 2.37 MB/sec warmup 52 sec latency 8.621 ms oleg413-client.virtnet: 1 21057 2.47 MB/sec warmup 53 sec latency 12.768 ms oleg413-client.virtnet: 1 21676 2.51 MB/sec warmup 54 sec latency 10.920 ms oleg413-client.virtnet: 1 22199 2.48 MB/sec warmup 55 sec latency 93.645 ms oleg413-client.virtnet: 1 22912 2.52 MB/sec warmup 56 sec latency 7.609 ms oleg413-client.virtnet: 1 23435 2.48 MB/sec warmup 57 sec latency 100.114 ms oleg413-client.virtnet: 1 23435 2.44 MB/sec warmup 58 sec latency 1100.317 ms oleg413-client.virtnet: 1 23435 2.40 MB/sec warmup 59 sec latency 2100.532 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 1 sec latency 4100.816 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 2 sec latency 5101.024 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 3 sec latency 6101.240 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 4 sec latency 7101.419 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 5 sec latency 8101.594 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 6 sec latency 9101.833 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 7 sec latency 10102.019 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 8 sec latency 11102.162 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 9 sec latency 12102.316 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 10 sec latency 13102.503 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 11 sec latency 14102.670 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 12 sec latency 15102.894 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 13 sec latency 16103.064 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 14 sec latency 17103.244 ms oleg413-client.virtnet: 1 23435 0.00 MB/sec execute 15 sec latency 18103.432 ms oleg413-client.virtnet: 1 23779 0.09 MB/sec execute 16 sec latency 18513.588 ms oleg413-client.virtnet: 1 24854 0.69 MB/sec execute 17 sec latency 12.296 ms oleg413-client.virtnet: 1 25329 0.74 MB/sec execute 18 sec latency 10.132 ms oleg413-client.virtnet: 1 25921 0.81 MB/sec executeUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12960 3591304 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36656 3569592 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160896 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg413-server Stopping /mnt/lustre-mds2 (opts:) on oleg413-server 05:01:11 (1715590871) shut down Failover mds2 to oleg413-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0001 05:01:25 (1715590885) targets are mounted 05:01:25 (1715590885) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13096 3592896 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36516 3569972 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7162868 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 05:01:37 (1715590897) shut down Failover mds1 to oleg413-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 05:01:50 (1715590910) targets are mounted 05:01:50 (1715590910) facet_failover done 19 sec latency 11.037 ms oleg413-client.virtnet: 1 26576 0.93 MB/sec execute 20 sec latency 9.984 ms oleg413-client.virtnet: 1 27235 0.97 MB/sec execute 21 sec latency 10.441 ms oleg413-client.virtnet: 1 28209 1.27 MB/sec execute 22 sec latency 9.959 ms oleg413-client.virtnet: 1 28769 1.41 MB/sec execute 23 sec latency 10.075 ms oleg413-client.virtnet: 1 29222 1.38 MB/sec execute 24 sec latency 13.896 ms oleg413-client.virtnet: 1 29827 1.50 MB/sec execute 25 sec latency 11.357 ms oleg413-client.virtnet: 1 30294 1.46 MB/sec execute 26 sec latency 11.498 ms oleg413-client.virtnet: 1 30897 1.49 MB/sec execute 27 sec latency 9.789 ms oleg413-client.virtnet: 1 31976 1.79 MB/sec execute 28 sec latency 9.313 ms oleg413-client.virtnet: 1 32491 1.78 MB/sec execute 29 sec latency 9.202 ms oleg413-client.virtnet: 1 33138 1.79 MB/sec execute 30 sec latency 9.124 ms oleg413-client.virtnet: 1 33772 1.84 MB/sec execute 31 sec latency 9.662 ms oleg413-client.virtnet: 1 34432 1.85 MB/sec execute 32 sec latency 9.383 ms oleg413-client.virtnet: 1 35525 2.09 MB/sec execute 33 sec latency 9.161 ms oleg413-client.virtnet: 1 36030 2.08 MB/sec execute 34 sec latency 9.406 ms oleg413-client.virtnet: 1 36647 2.08 MB/sec execute 35 sec latency 10.757 ms oleg413-client.virtnet: 1 37245 2.11 MB/sec execute 36 sec latency 11.701 ms oleg413-client.virtnet: 1 37915 2.10 MB/sec execute 37 sec latency 8.482 ms oleg413-client.virtnet: 1 38497 2.19 MB/sec execute 38 sec latency 351.439 ms oleg413-client.virtnet: 1 38497 2.14 MB/sec execute 39 sec latency 1351.643 ms oleg413-client.virtnet: 1 38497 2.08 MB/sec execute 40 sec latency 2351.893 ms oleg413-client.virtnet: 1 38497 2.03 MB/sec execute 41 sec latency 3352.110 ms oleg413-client.virtnet: 1 39197 2.11 MB/sec execute 42 sec latency 3629.019 ms oleg413-client.virtnet: 1 39746 2.10 MB/sec execute 43 sec latency 14.444 ms oleg413-client.virtnet: 1 40317 2.09 MB/sec execute 44 sec latency 9.521 ms oleg413-client.virtnet: 1 41045 2.12 MB/sec execute 45 sec latency 9.207 ms oleg413-client.virtnet: 1 41823 2.19 MB/sec execute 46 sec latency 67.473 ms oleg413-client.virtnet: 1 42607 2.28 MB/sec execute 47 sec latency 410.292 ms oleg413-client.virtnet: 1 43088 2.27 MB/sec execute 48 sec latency 9.868 ms oleg413-client.virtnet: 1 43425 2.24 MB/sec execute 49 sec latency 437.058 ms oleg413-client.virtnet: 1 43425 2.19 MB/sec execute 50 sec latency 1437.254 ms oleg413-client.virtnet: 1 43425 2.15 MB/sec execute 51 sec latency 2437.449 ms oleg413-client.virtnet: 1 43425 2.11 MB/sec execute 52 sec latency 3437.660 ms oleg413-client.virtnet: 1 43425 2.07 MB/sec execute 53 sec latency 4437.861 ms oleg413-client.virtnet: 1 43425 2.03 MB/sec execute 54 sec latency 5438.031 ms oleg413-client.virtnet: 1 43425 1.99 MB/sec execute 55 sec latency 6438.183 ms oleg413-client.virtnet: 1 43425 1.96 MB/sec execute 56 sec latency 7438.351 ms oleg413-client.virtnet: 1 43425 1.92 MB/sec execute 57 sec latency 8438.583 ms oleg413-client.virtnet: 1 43425 1.89 MB/sec execute 58 sec latency 9438.760 ms oleg413-client.virtnet: 1 43425 1.86 MB/sec execute 59 sec latency 10438.962 ms oleg413-client.virtnet: 1 43425 1.83 MB/sec execute 60 sec latency 11439.126 ms oleg413-client.virtnet: 1 43425 1.80 MB/sec execute 61 sec latency 12439.286 ms oleg413-client.virtnet: 1 43425 1.77 MB/sec execute 62 sec latency 13439.420 ms oleg413-client.virtnet: 1 43425 1.74 MB/sec execute 63 sec latency 14439.576 ms oleg413-client.virtnet: 1 4oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13104 3592852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36396 3570120 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162972 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg413-server Stopping /mnt/lustre-mds2 (opts:) on oleg413-server 05:02:02 (1715590922) shut down Failover mds2 to oleg413-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0001 05:02:16 (1715590936) targets are mounted 05:02:16 (1715590936) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13096 3591908 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36504 3569928 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49600 7161836 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 05:02:28 (1715590948) shut down 3425 1.71 MB/sec execute 64 sec latency 15439.706 ms oleg413-client.virtnet: 1 43425 1.68 MB/sec execute 65 sec latency 16439.857 ms oleg413-client.virtnet: 1 43425 1.66 MB/sec execute 66 sec latency 17440.028 ms oleg413-client.virtnet: 1 43425 1.63 MB/sec execute 67 sec latency 18440.148 ms oleg413-client.virtnet: 1 44090 1.68 MB/sec execute 68 sec latency 18474.863 ms oleg413-client.virtnet: 1 44779 1.66 MB/sec execute 69 sec latency 9.703 ms oleg413-client.virtnet: 1 45804 1.75 MB/sec execute 70 sec latency 10.074 ms oleg413-client.virtnet: 1 46542 1.80 MB/sec execute 71 sec latency 9.221 ms oleg413-client.virtnet: 1 47134 1.79 MB/sec execute 72 sec latency 10.046 ms oleg413-client.virtnet: 1 47868 1.83 MB/sec execute 73 sec latency 9.512 ms oleg413-client.virtnet: 1 48605 1.83 MB/sec execute 74 sec latency 7.603 ms oleg413-client.virtnet: 1 49761 1.94 MB/sec execute 75 sec latency 8.373 ms oleg413-client.virtnet: 1 50343 1.94 MB/sec execute 76 sec latency 9.144 ms oleg413-client.virtnet: 1 50948 1.94 MB/sec execute 77 sec latency 9.808 ms oleg413-client.virtnet: 1 51719 1.96 MB/sec execute 78 sec latency 8.569 ms oleg413-client.virtnet: 1 52668 2.02 MB/sec execute 79 sec latency 8.738 ms oleg413-client.virtnet: 1 53563 2.08 MB/sec execute 80 sec latency 8.730 ms oleg413-client.virtnet: 1 54165 2.06 MB/sec execute 81 sec latency 10.722 ms oleg413-client.virtnet: 1 54897 2.09 MB/sec execute 82 sec latency 9.408 ms oleg413-client.virtnet: 1 55664 2.09 MB/sec execute 83 sec latency 9.100 ms oleg413-client.virtnet: 1 56817 2.19 MB/sec execute 84 sec latency 8.527 ms oleg413-client.virtnet: 1 57409 2.18 MB/sec execute 85 sec latency 8.339 ms oleg413-client.virtnet: 1 58039 2.18 MB/sec execute 86 sec latency 10.595 ms oleg413-client.virtnet: 1 58786 2.20 MB/sec execute 87 sec latency 9.266 ms oleg413-client.virtnet: 1 59743 2.25 MB/sec execute 88 sec latency 8.063 ms oleg413-client.virtnet: 1 60141 2.25 MB/sec execute 89 sec latency 679.715 ms oleg413-client.virtnet: 1 60141 2.22 MB/sec execute 90 sec latency 1679.945 ms oleg413-client.virtnet: 1 60141 2.20 MB/sec execute 91 sec latency 2680.138 ms oleg413-client.virtnet: 1 60141 2.17 MB/sec execute 92 sec latency 3680.307 ms oleg413-client.virtnet: 1 60607 2.20 MB/sec execute 93 sec latency 4017.697 ms oleg413-client.virtnet: 1 61207 2.18 MB/sec execute 94 sec latency 8.825 ms oleg413-client.virtnet: 1 61915 2.21 MB/sec execute 95 sec latency 10.502 ms oleg413-client.virtnet: 1 62673 2.20 MB/sec execute 96 sec latency 8.708 ms oleg413-client.virtnet: 1 63682 2.26 MB/sec execute 97 sec latency 12.874 ms oleg413-client.virtnet: 1 64298 2.28 MB/sec execute 98 sec latency 21.259 ms oleg413-client.virtnet: 1 64879 2.27 MB/sec execute 99 sec latency 10.512 ms oleg413-client.virtnet: 1 65340 2.29 MB/sec execute 100 sec latency 429.685 ms oleg413-client.virtnet: 1 65340 2.27 MB/sec execute 101 sec latency 1429.881 ms oleg413-client.virtnet: 1 65340 2.24 MB/sec execute 102 sec latency 2430.086 ms oleg413-client.virtnet: 1 65340 2.22 MB/sec execute 103 sec latency 3430.335 ms oleg413-client.virtnet: 1 65340 2.20 MB/sec execute 104 sec latency 4430.534 ms oleg413-client.virtnet: 1 65340 2.18 MB/sec execute 105 sec latency 5430.720 ms oleg413-client.virtnet: 1 65340 2.16 MB/sec execute 106 sec latency 6430.961 ms oleg413-client.virtnet: 1 65340 2.14 MB/sec execute 107 sec latency 7431.170 ms oleg413-client.virtnet: 1 65340 2.12 MB/sec execute 108 sec latency 8431.397 ms oleg413-clieFailover mds1 to oleg413-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 05:02:42 (1715590962) targets are mounted 05:02:42 (1715590962) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13112 3590884 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36516 3569916 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49628 7160800 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg413-server Stopping /mnt/lustre-mds2 (opts:) on oleg413-server 05:02:53 (1715590973) shut down Failover mds2 to oleg413-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0001 05:03:07 (1715590987) targets are mounted 05:03:07 (1715590987) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13576 3592556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36504 3570004 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50080 7162560 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server nt.virtnet: 1 65340 2.10 MB/sec execute 109 sec latency 9431.590 ms oleg413-client.virtnet: 1 65340 2.08 MB/sec execute 110 sec latency 10431.816 ms oleg413-client.virtnet: 1 65340 2.06 MB/sec execute 111 sec latency 11431.939 ms oleg413-client.virtnet: 1 65340 2.04 MB/sec execute 112 sec latency 12432.055 ms oleg413-client.virtnet: 1 65340 2.02 MB/sec execute 113 sec latency 13432.245 ms oleg413-client.virtnet: 1 65340 2.01 MB/sec execute 114 sec latency 14432.429 ms oleg413-client.virtnet: 1 65340 1.99 MB/sec execute 115 sec latency 15432.632 ms oleg413-client.virtnet: 1 65340 1.97 MB/sec execute 116 sec latency 16432.872 ms oleg413-client.virtnet: 1 65340 1.96 MB/sec execute 117 sec latency 17433.058 ms oleg413-client.virtnet: 1 65340 1.94 MB/sec execute 118 sec latency 18433.185 ms oleg413-client.virtnet: 1 65741 1.93 MB/sec execute 119 sec latency 18617.189 ms oleg413-client.virtnet: 1 66467 1.93 MB/sec execute 120 sec latency 8.750 ms oleg413-client.virtnet: 1 67558 2.00 MB/sec execute 121 sec latency 8.889 ms oleg413-client.virtnet: 1 68094 1.99 MB/sec execute 122 sec latency 9.870 ms oleg413-client.virtnet: 1 68678 1.99 MB/sec execute 123 sec latency 13.559 ms oleg413-client.virtnet: 1 69434 2.00 MB/sec execute 124 sec latency 8.569 ms oleg413-client.virtnet: 1 70320 2.04 MB/sec execute 125 sec latency 8.561 ms oleg413-client.virtnet: 1 71226 2.08 MB/sec execute 126 sec latency 9.346 ms oleg413-client.virtnet: 1 71765 2.07 MB/sec execute 127 sec latency 11.230 ms oleg413-client.virtnet: 1 72482 2.09 MB/sec execute 128 sec latency 8.743 ms oleg413-client.virtnet: 1 73198 2.08 MB/sec execute 129 sec latency 8.884 ms oleg413-client.virtnet: 1 74252 2.12 MB/sec execute 130 sec latency 8.096 ms oleg413-client.virtnet: 1 74952 2.15 MB/sec execute 131 sec latency 8.627 ms oleg413-client.virtnet: 1 75581 2.14 MB/sec execute 132 sec latency 12.523 ms oleg413-client.virtnet: 1 76278 2.15 MB/sec execute 133 sec latency 9.370 ms oleg413-client.virtnet: 1 77034 2.16 MB/sec execute 134 sec latency 7.722 ms oleg413-client.virtnet: 1 78208 2.21 MB/sec execute 135 sec latency 9.416 ms oleg413-client.virtnet: 1 78777 2.21 MB/sec execute 136 sec latency 9.828 ms oleg413-client.virtnet: 1 79484 2.23 MB/sec execute 137 sec latency 9.835 ms oleg413-client.virtnet: 1 80146 2.22 MB/sec execute 138 sec latency 9.103 ms oleg413-client.virtnet: 1 81063 2.25 MB/sec execute 139 sec latency 8.601 ms oleg413-client.virtnet: 1 81752 2.27 MB/sec execute 140 sec latency 388.250 ms oleg413-client.virtnet: 1 81752 2.26 MB/sec execute 141 sec latency 1388.380 ms oleg413-client.virtnet: 1 81752 2.24 MB/sec execute 142 sec latency 2388.529 ms oleg413-client.virtnet: 1 81752 2.22 MB/sec execute 143 sec latency 3388.686 ms oleg413-client.virtnet: 1 81993 2.22 MB/sec execute 144 sec latency 3951.140 ms oleg413-client.virtnet: 1 82569 2.21 MB/sec execute 145 sec latency 9.338 ms oleg413-client.virtnet: 1 83296 2.22 MB/sec execute 146 sec latency 12.654 ms oleg413-client.virtnet: 1 84044 2.22 MB/sec execute 147 sec latency 10.640 ms oleg413-client.virtnet: 1 85150 2.28 MB/sec execute 148 sec latency 9.890 ms oleg413-client.virtnet: 1 85649 2.27 MB/sec execute 149 sec latency 11.235 ms oleg413-client.virtnet: 1 86265 2.27 MB/sec execute 150 sec latency 13.126 ms oleg413-client.virtnet: 1 86791 2.28 MB/sec execute 151 sec latency 238.454 ms oleg413-client.virtnet: 1 86791 2.26 MB/sec execute 152 sec latency 1238.666 ms oleg413-client.virtnet: 1 86791 2.25 MB/sec execute 153 sec latenc05:03:25 (1715591005) shut down Failover mds1 to oleg413-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 05:03:39 (1715591019) targets are mounted 05:03:39 (1715591019) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13236 3590780 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36520 3569724 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49756 7160504 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg413-server Stopping /mnt/lustre-mds2 (opts:) on oleg413-server 05:03:51 (1715591031) shut down Failover mds2 to oleg413-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0001 05:04:05 (1715591045) targets are mounted 05:04:05 (1715591045) facet_failover done y 2238.882 ms oleg413-client.virtnet: 1 86791 2.23 MB/sec execute 154 sec latency 3239.114 ms oleg413-client.virtnet: 1 86791 2.22 MB/sec execute 155 sec latency 4239.293 ms oleg413-client.virtnet: 1 86791 2.20 MB/sec execute 156 sec latency 5239.457 ms oleg413-client.virtnet: 1 86791 2.19 MB/sec execute 157 sec latency 6239.624 ms oleg413-client.virtnet: 1 86791 2.18 MB/sec execute 158 sec latency 7239.797 ms oleg413-client.virtnet: 1 86791 2.16 MB/sec execute 159 sec latency 8239.952 ms oleg413-client.virtnet: 1 86791 2.15 MB/sec execute 160 sec latency 9240.144 ms oleg413-client.virtnet: 1 86791 2.13 MB/sec execute 161 sec latency 10240.325 ms oleg413-client.virtnet: 1 86791 2.12 MB/sec execute 162 sec latency 11240.509 ms oleg413-client.virtnet: 1 86791 2.11 MB/sec execute 163 sec latency 12240.676 ms oleg413-client.virtnet: 1 86791 2.10 MB/sec execute 164 sec latency 13240.868 ms oleg413-client.virtnet: 1 86791 2.08 MB/sec execute 165 sec latency 14241.062 ms oleg413-client.virtnet: 1 86791 2.07 MB/sec execute 166 sec latency 15241.213 ms oleg413-client.virtnet: 1 86791 2.06 MB/sec execute 167 sec latency 16241.423 ms oleg413-client.virtnet: 1 86791 2.05 MB/sec execute 168 sec latency 17241.550 ms oleg413-client.virtnet: 1 86791 2.03 MB/sec execute 169 sec latency 18241.664 ms oleg413-client.virtnet: 1 86791 2.02 MB/sec execute 170 sec latency 19241.854 ms oleg413-client.virtnet: 1 86791 2.01 MB/sec execute 171 sec latency 20242.070 ms oleg413-client.virtnet: 1 86791 2.00 MB/sec execute 172 sec latency 21242.312 ms oleg413-client.virtnet: 1 86791 1.99 MB/sec execute 173 sec latency 22242.506 ms oleg413-client.virtnet: 1 86791 1.98 MB/sec execute 174 sec latency 23242.735 ms oleg413-client.virtnet: 1 86791 1.96 MB/sec execute 175 sec latency 24242.928 ms oleg413-client.virtnet: 1 87188 1.95 MB/sec execute 176 sec latency 24635.337 ms oleg413-client.virtnet: 1 88101 1.98 MB/sec execute 177 sec latency 8.283 ms oleg413-client.virtnet: 1 89017 2.01 MB/sec execute 178 sec latency 8.267 ms oleg413-client.virtnet: 1 89593 2.00 MB/sec execute 179 sec latency 12.256 ms oleg413-client.virtnet: 1 90249 2.02 MB/sec execute 180 sec latency 10.887 ms oleg413-client.virtnet: 1 90897 2.01 MB/sec execute 181 sec latency 10.103 ms oleg413-client.virtnet: 1 91753 2.03 MB/sec execute 182 sec latency 10.933 ms oleg413-client.virtnet: 1 92638 2.06 MB/sec execute 183 sec latency 9.516 ms oleg413-client.virtnet: 1 93198 2.05 MB/sec execute 184 sec latency 11.181 ms oleg413-client.virtnet: 1 93841 2.06 MB/sec execute 185 sec latency 11.899 ms oleg413-client.virtnet: 1 94523 2.06 MB/sec execute 186 sec latency 9.584 ms oleg413-client.virtnet: 1 95499 2.09 MB/sec execute 187 sec latency 9.439 ms oleg413-client.virtnet: 1 96212 2.11 MB/sec execute 188 sec latency 9.029 ms oleg413-client.virtnet: 1 96774 2.10 MB/sec execute 189 sec latency 15.482 ms oleg413-client.virtnet: 1 97513 2.11 MB/sec execute 190 sec latency 8.871 ms oleg413-client.virtnet: 1 98272 2.11 MB/sec execute 191 sec latency 8.186 ms oleg413-client.virtnet: 1 99452 2.15 MB/sec execute 192 sec latency 8.099 ms oleg413-client.virtnet: 1 100050 2.15 MB/sec execute 193 sec latency 10.888 ms oleg413-client.virtnet: 1 100614 2.15 MB/sec execute 194 sec latency 9.657 ms oleg413-client.virtnet: 1 101391 2.16 MB/sec execute 195 sec latency 9.025 ms oleg413-client.virtnet: 1 102339 2.18 MB/sec execute 196 sec latency 8.242 ms oleg413-client.virtnet: 1 103174 2.20 MB/sec execute 197 sec latency 124.565 ms oleg413-client.virtnet: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13416 3592576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36400 3570080 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49816 7162656 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 05:04:16 (1715591056) shut down Failover mds1 to oleg413-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 05:04:30 (1715591070) targets are mounted 05:04:30 (1715591070) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13196 3590856 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36496 3569824 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49692 7160680 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg413-server Stopping /mnt/lustre-mds2 (opts:) on oleg413-server 05:04:42 (1715591082) shut down 1 103174 2.19 MB/sec execute 198 sec latency 1124.758 ms oleg413-client.virtnet: 1 103174 2.18 MB/sec execute 199 sec latency 2124.943 ms oleg413-client.virtnet: 1 103174 2.17 MB/sec execute 200 sec latency 3125.132 ms oleg413-client.virtnet: 1 103247 2.16 MB/sec execute 201 sec latency 3956.235 ms oleg413-client.virtnet: 1 103836 2.15 MB/sec execute 202 sec latency 9.340 ms oleg413-client.virtnet: 1 104593 2.16 MB/sec execute 203 sec latency 9.441 ms oleg413-client.virtnet: 1 105348 2.16 MB/sec execute 204 sec latency 11.372 ms oleg413-client.virtnet: 1 106483 2.20 MB/sec execute 205 sec latency 8.470 ms oleg413-client.virtnet: 1 107039 2.20 MB/sec execute 206 sec latency 9.003 ms oleg413-client.virtnet: 1 107688 2.20 MB/sec execute 207 sec latency 9.726 ms oleg413-client.virtnet: 1 108342 2.20 MB/sec execute 208 sec latency 82.239 ms oleg413-client.virtnet: 1 108342 2.19 MB/sec execute 209 sec latency 1082.638 ms oleg413-client.virtnet: 1 108342 2.18 MB/sec execute 210 sec latency 2082.838 ms oleg413-client.virtnet: 1 108342 2.17 MB/sec execute 211 sec latency 3083.066 ms oleg413-client.virtnet: 1 108342 2.16 MB/sec execute 212 sec latency 4083.274 ms oleg413-client.virtnet: 1 108342 2.15 MB/sec execute 213 sec latency 5083.495 ms oleg413-client.virtnet: 1 108342 2.14 MB/sec execute 214 sec latency 6083.704 ms oleg413-client.virtnet: 1 108342 2.13 MB/sec execute 215 sec latency 7083.898 ms oleg413-client.virtnet: 1 108342 2.12 MB/sec execute 216 sec latency 8084.067 ms oleg413-client.virtnet: 1 108342 2.11 MB/sec execute 217 sec latency 9084.294 ms oleg413-client.virtnet: 1 108342 2.10 MB/sec execute 218 sec latency 10084.485 ms oleg413-client.virtnet: 1 108342 2.09 MB/sec execute 219 sec latency 11084.605 ms oleg413-client.virtnet: 1 108342 2.08 MB/sec execute 220 sec latency 12084.746 ms oleg413-client.virtnet: 1 108342 2.08 MB/sec execute 221 sec latency 13084.913 ms oleg413-client.virtnet: 1 108342 2.07 MB/sec execute 222 sec latency 14085.096 ms oleg413-client.virtnet: 1 108342 2.06 MB/sec execute 223 sec latency 15085.273 ms oleg413-client.virtnet: 1 108342 2.05 MB/sec execute 224 sec latency 16085.425 ms oleg413-client.virtnet: 1 108342 2.04 MB/sec execute 225 sec latency 17085.630 ms oleg413-client.virtnet: 1 108342 2.03 MB/sec execute 226 sec latency 18085.802 ms oleg413-client.virtnet: 1 108779 2.03 MB/sec execute 227 sec latency 18498.340 ms oleg413-client.virtnet: 1 109731 2.05 MB/sec execute 228 sec latency 8.446 ms oleg413-client.virtnet: 1 110417 2.06 MB/sec execute 229 sec latency 10.695 ms oleg413-client.virtnet: 1 110967 2.06 MB/sec execute 230 sec latency 11.302 ms oleg413-client.virtnet: 1 111603 2.07 MB/sec execute 231 sec latency 10.378 ms oleg413-client.virtnet: 1 112341 2.07 MB/sec execute 232 sec latency 9.064 ms oleg413-client.virtnet: 1 113281 2.09 MB/sec execute 233 sec latency 10.928 ms oleg413-client.virtnet: 1 113958 2.10 MB/sec execute 234 sec latency 8.868 ms oleg413-client.virtnet: 1 114534 2.10 MB/sec execute 235 sec latency 10.994 ms oleg413-client.virtnet: 1 115258 2.10 MB/sec execute 236 sec latency 9.870 ms oleg413-client.virtnet: 1 115985 2.10 MB/sec execute 237 sec latency 21.862 ms oleg413-client.virtnet: 1 117116 2.14 MB/sec execute 238 sec latency 8.346 ms oleg413-client.virtnet: 1 117672 2.14 MB/sec execute 239 sec latency 8.771 ms oleg413-client.virtnet: 1 118312 2.14 MB/sec execute 240 sec latency 10.297 ms oleg413-client.virtnet: 1 118984 2.14 MB/sec execute 241 sec latency 10.253 ms oleg413-client.virtnet: 1 119687 2.14 MB/sec execute 242 sec lFailover mds2 to oleg413-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0001 05:04:56 (1715591096) targets are mounted 05:04:56 (1715591096) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec atency 9.681 ms oleg413-client.virtnet: 1 120793 2.17 MB/sec execute 243 sec latency 8.103 ms oleg413-client.virtnet: 1 121307 2.17 MB/sec execute 244 sec latency 9.492 ms oleg413-client.virtnet: 1 121820 2.17 MB/sec execute 245 sec latency 10.407 ms oleg413-client.virtnet: 1 122428 2.17 MB/sec execute 246 sec latency 10.939 ms oleg413-client.virtnet: 1 123112 2.18 MB/sec execute 247 sec latency 8.815 ms oleg413-client.virtnet: 1 123642 2.19 MB/sec execute 248 sec latency 451.387 ms oleg413-client.virtnet: 1 123642 2.18 MB/sec execute 249 sec latency 1451.645 ms oleg413-client.virtnet: 1 123642 2.17 MB/sec execute 250 sec latency 2451.836 ms oleg413-client.virtnet: 1 123642 2.16 MB/sec execute 251 sec latency 3452.032 ms oleg413-client.virtnet: 1 123642 2.15 MB/sec execute 252 sec latency 4452.236 ms oleg413-client.virtnet: 1 124481 2.17 MB/sec execute 253 sec latency 4579.213 ms oleg413-client.virtnet: 1 125057 2.16 MB/sec execute 254 sec latency 12.725 ms oleg413-client.virtnet: 1 125752 2.17 MB/sec execute 255 sec latency 9.728 ms oleg413-client.virtnet: 1 126535 2.17 MB/sec execute 256 sec latency 9.079 ms oleg413-client.virtnet: 1 127572 2.19 MB/sec execute 257 sec latency 8.379 ms oleg413-client.virtnet: 1 128194 2.20 MB/sec execute 258 sec latency 8.822 ms oleg413-client.virtnet: 1 128851 2.20 MB/sec execute 259 sec latency 10.387 ms oleg413-client.virtnet: 1 129553 2.20 MB/sec execute 260 sec latency 9.556 ms oleg413-client.virtnet: 1 130324 2.21 MB/sec execute 261 sec latency 7.820 ms oleg413-client.virtnet: 1 131422 2.24 MB/sec execute 262 sec latency 11.475 ms oleg413-client.virtnet: 1 131985 2.23 MB/sec execute 263 sec latency 11.428 ms oleg413-client.virtnet: 1 132692 2.24 MB/sec execute 264 sec latency 8.314 ms oleg413-client.virtnet: 1 133379 2.24 MB/sec execute 265 sec latency 9.593 ms oleg413-client.virtnet: 1 134286 2.25 MB/sec execute 266 sec latency 7.838 ms oleg413-client.virtnet: 1 135177 2.27 MB/sec execute 267 sec latency 8.461 ms oleg413-client.virtnet: 1 135779 2.26 MB/sec execute 268 sec latency 11.021 ms oleg413-client.virtnet: 1 136543 2.27 MB/sec execute 269 sec latency 8.056 ms oleg413-client.virtnet: 1 137275 2.27 MB/sec execute 270 sec latency 8.694 ms oleg413-client.virtnet: 1 138427 2.30 MB/sec execute 271 sec latency 9.477 ms oleg413-client.virtnet: 1 138953 2.30 MB/sec execute 272 sec latency 9.330 ms oleg413-client.virtnet: 1 139598 2.30 MB/sec execute 273 sec latency 11.716 ms oleg413-client.virtnet: 1 140282 2.30 MB/sec execute 274 sec latency 10.354 ms oleg413-client.virtnet: 1 141137 2.31 MB/sec execute 275 sec latency 8.130 ms oleg413-client.virtnet: 1 142114 2.33 MB/sec execute 276 sec latency 9.068 ms oleg413-client.virtnet: 1 142594 2.33 MB/sec execute 277 sec latency 12.496 ms oleg413-client.virtnet: 1 143180 2.33 MB/sec execute 278 sec latency 9.788 ms oleg413-client.virtnet: 1 143897 2.33 MB/sec execute 279 sec latency 8.823 ms oleg413-client.virtnet: 1 144687 2.34 MB/sec execute 280 sec latency 9.516 ms oleg413-client.virtnet: 1 145679 2.36 MB/sec execute 281 sec latency 9.258 ms oleg413-client.virtnet: 1 146177 2.35 MB/sec execute 282 sec latency 14.652 ms oleg413-client.virtnet: 1 146691 2.35 MB/sec execute 283 sec latency 11.026 ms oleg413-client.virtnet: 1 147375 2.36 MB/sec execute 284 sec latency 10.463 ms oleg413-client.virtnet: 1 148042 2.36 MB/sec execute 285 sec latency 9.634 ms oleg413-client.virtnet: 1 149088 2.38 MB/sec execute 286 sec latency 11.148 ms oleg413-client.virtnet: 1 149569 2.38 MB/sec execute 287 sec latency 10.394 ms oleg413-client.virtnet: 1 150167 2.38 MB/sec execute 288 sec latency 15.960 ms oleg413-client.virtnet: 1 150810 2.38 MB/sec execute 289 sec latency 13.064 ms oleg413-client.virtnet: 1 151456 2.38 MB/sec execute 290 sec latency 9.180 ms oleg413-client.virtnet: 1 152580 2.41 MB/sec execute 291 sec latency 9.462 ms oleg413-client.virtnet: 1 153118 2.40 MB/sec execute 292 sec latency 9.101 ms oleg413-client.virtnet: 1 153755 2.40 MB/sec execute 293 sec latency 12.295 ms oleg413-client.virtnet: 1 154480 2.41 MB/sec execute 294 sec latency 9.099 ms oleg413-client.virtnet: 1 155328 2.42 MB/sec execute 295 sec latency 8.366 ms oleg413-client.virtnet: 1 156346 2.43 MB/sec execute 296 sec latency 11.271 ms oleg413-client.virtnet: 1 156882 2.43 MB/sec execute 297 sec latency 12.577 ms oleg413-client.virtnet: 1 157534 2.44 MB/sec execute 298 sec latency 9.977 ms oleg413-client.virtnet: 1 158195 2.43 MB/sec execute 299 sec latency 13.448 ms oleg413-client.virtnet: 1 cleanup 300 sec oleg413-client.virtnet: 0 cleanup 300 sec oleg413-client.virtnet: oleg413-client.virtnet: Operation Count AvgLat MaxLat oleg413-client.virtnet: ---------------------------------------- oleg413-client.virtnet: NTCreateX 23395 8.095 24635.286 oleg413-client.virtnet: Close 17202 0.864 6.814 oleg413-client.virtnet: Rename 995 10.774 4579.201 oleg413-client.virtnet: Unlink 4726 1.727 5.652 oleg413-client.virtnet: Qpathinfo 21241 1.519 3956.215 oleg413-client.virtnet: Qfileinfo 3716 0.140 1.635 oleg413-client.virtnet: Qfsinfo 3897 0.202 3.802 oleg413-client.virtnet: Sfileinfo 1916 3.172 12.283 oleg413-client.virtnet: Find 8212 0.376 8.915 oleg413-client.virtnet: WriteX 11581 0.788 8.008 oleg413-client.virtnet: ReadX 36856 0.018 1.527 oleg413-client.virtnet: LockX 78 0.945 1.994 oleg413-client.virtnet: UnlockX 78 0.979 2.631 oleg413-client.virtnet: Flush 1639 16.093 18498.329 oleg413-client.virtnet: oleg413-client.virtnet: Throughput 2.43 MB/sec 1 clients 1 procs max_latency=24635.337 ms oleg413-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg413-client.virtnet at Mon May 13 05:05:48 EDT 2024 with return code 0 oleg413-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg413-client.virtnet oleg413-client.virtnet: /mnt/lustre/d70b.replay-single/oleg413-client.virtnet /mnt/lustre/d70b.replay-single/oleg413-client.virtnet oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg413-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg413-client.virtnet: removed directory: 'clients/client0' oleg413-client.virtnet: removed directory: 'clients' oleg413-client.virtnet: removed 'client.txt' oleg413-client.virtnet: /mnt/lustre/d70b.replay-single/oleg413-client.virtnet oleg413-client.virtnet: dbench successfully finished