== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 19:48:37 (1723247317) Starting client oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre Started clients oleg453-client.virtnet: 192.168.204.153@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg453-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=12460 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3594184 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26144 3580876 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7175060 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 19:48:46 (1723247326) shut down Failover mds1 to oleg453-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 19:49:01 (1723247341) targets are mounted 19:49:01 (1723247341) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1740 3594244 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26400 3568880 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28140 7163124 1% /mnt/lustre test_70b fail mds2 2 times oleg453-client.virtnet: looking for dbench program oleg453-client.virtnet: /usr/bin/dbench oleg453-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg453-client.virtnet oleg453-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg453-client.virtnet' oleg453-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg453-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg453-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg453-client.virtnet at Fri Aug 9 19:48:38 EDT 2024 oleg453-client.virtnet: waiting for dbench pid 12485 oleg453-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg453-client.virtnet: oleg453-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg453-client.virtnet: failed to create barrier semaphore oleg453-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg453-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg453-client.virtnet: releasing clients oleg453-client.virtnet: 1 305 11.22 MB/sec warmup 1 sec latency 26.569 ms oleg453-client.virtnet: 1 660 10.57 MB/sec warmup 2 sec latency 18.152 ms oleg453-client.virtnet: 1 919 7.26 MB/sec warmup 3 sec latency 19.650 ms oleg453-client.virtnet: 1 1243 5.77 MB/sec warmup 4 sec latency 18.235 ms oleg453-client.virtnet: 1 1632 5.23 MB/sec warmup 5 sec latency 17.439 ms oleg453-client.virtnet: 1 1984 4.41 MB/sec warmup 6 sec latency 19.411 ms oleg453-client.virtnet: 1 2373 3.99 MB/sec warmup 7 sec latency 15.766 ms oleg453-client.virtnet: 1 2887 4.05 MB/sec warmup 8 sec latency 17.583 ms oleg453-client.virtnet: 1 3513 4.26 MB/sec warmup 9 sec latency 18.736 ms oleg453-client.virtnet: 1 3836 3.98 MB/sec warmup 10 sec latency 15.541 ms oleg453-client.virtnet: 1 4070 3.64 MB/sec warmup 11 sec latency 15.876 ms oleg453-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 951.758 ms oleg453-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 1951.911 ms oleg453-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 2952.098 ms oleg453-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 3952.269 ms oleg453-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 4952.490 ms oleg453-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 5952.655 ms oleg453-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 6952.778 ms oleg453-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 7952.923 ms oleg453-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 8953.111 ms oleg453-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 9953.283 ms oleg453-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 10953.540 ms oleg453-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 11953.732 ms oleg453-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 12953.909 ms oleg453-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 13954.094 ms oleg453-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 14954.308 ms oleg453-client.virtnet: 1 4433 1.51 MB/sec warmup 27 sec latency 14978.755 ms oleg453-client.virtnet: 1 4743 1.50 MB/sec warmup 28 sec latency 18.078 ms oleg453-client.virtnet: 1 5103 1.56 MB/sec warmup 29 sec latency 17.512 ms oleg453-client.virtnet: 1 5362 1.51 MB/sec warmup 30 sec latency 19.012 ms oleg453-client.virtnet: 1 5707 1.48 MB/sec warmup 31 sec latency 12.455 ms oleg453-client.virtnet: 1 6026 1.49 MB/sec warmup 32 sec latency 18.493 ms oleg453-client.virtnet: 1 6321 1.55 MB/sec warmup 33 sec latency 527.363 ms olFailing mds2 on oleg453-server Stopping /mnt/lustre-mds2 (opts:) on oleg453-server 19:49:14 (1723247354) shut down Failover mds2 to oleg453-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0001 19:49:30 (1723247370) targets are mounted 19:49:30 (1723247370) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14484 3591848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37412 3569460 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51896 7161308 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 19:49:43 (1723247383) shut down Failover mds1 to oleg453-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 eg453-client.virtnet: 1 6927 1.60 MB/sec warmup 34 sec latency 18.712 ms oleg453-client.virtnet: 1 7341 1.68 MB/sec warmup 35 sec latency 16.285 ms oleg453-client.virtnet: 1 7363 1.64 MB/sec warmup 36 sec latency 890.539 ms oleg453-client.virtnet: 1 7363 1.59 MB/sec warmup 37 sec latency 1890.775 ms oleg453-client.virtnet: 1 7363 1.55 MB/sec warmup 38 sec latency 2891.001 ms oleg453-client.virtnet: 1 7363 1.51 MB/sec warmup 39 sec latency 3891.222 ms oleg453-client.virtnet: 1 7363 1.47 MB/sec warmup 40 sec latency 4891.431 ms oleg453-client.virtnet: 1 7363 1.44 MB/sec warmup 41 sec latency 5891.645 ms oleg453-client.virtnet: 1 7363 1.40 MB/sec warmup 42 sec latency 6891.887 ms oleg453-client.virtnet: 1 7363 1.37 MB/sec warmup 43 sec latency 7892.085 ms oleg453-client.virtnet: 1 7363 1.34 MB/sec warmup 44 sec latency 8892.264 ms oleg453-client.virtnet: 1 7363 1.31 MB/sec warmup 45 sec latency 9892.467 ms oleg453-client.virtnet: 1 7363 1.28 MB/sec warmup 46 sec latency 10892.659 ms oleg453-client.virtnet: 1 7363 1.25 MB/sec warmup 47 sec latency 11892.819 ms oleg453-client.virtnet: 1 7363 1.23 MB/sec warmup 48 sec latency 12892.978 ms oleg453-client.virtnet: 1 7363 1.20 MB/sec warmup 49 sec latency 13893.168 ms oleg453-client.virtnet: 1 7363 1.18 MB/sec warmup 50 sec latency 14893.323 ms oleg453-client.virtnet: 1 7363 1.15 MB/sec warmup 51 sec latency 15893.538 ms oleg453-client.virtnet: 1 7363 1.13 MB/sec warmup 52 sec latency 16893.799 ms oleg453-client.virtnet: 1 7363 1.11 MB/sec warmup 53 sec latency 17894.024 ms oleg453-client.virtnet: 1 7363 1.09 MB/sec warmup 54 sec latency 18894.332 ms oleg453-client.virtnet: 1 7401 1.07 MB/sec warmup 55 sec latency 19727.112 ms oleg453-client.virtnet: 1 7627 1.06 MB/sec warmup 56 sec latency 16.598 ms oleg453-client.virtnet: 1 7899 1.05 MB/sec warmup 57 sec latency 19.551 ms oleg453-client.virtnet: 1 8186 1.05 MB/sec warmup 58 sec latency 18.345 ms oleg453-client.virtnet: 1 8555 1.09 MB/sec warmup 59 sec latency 18.447 ms oleg453-client.virtnet: 1 9127 0.30 MB/sec execute 1 sec latency 20.061 ms oleg453-client.virtnet: 1 9493 0.86 MB/sec execute 2 sec latency 17.872 ms oleg453-client.virtnet: 1 9993 2.19 MB/sec execute 3 sec latency 18.171 ms oleg453-client.virtnet: 1 10611 3.03 MB/sec execute 4 sec latency 18.203 ms oleg453-client.virtnet: 1 10911 2.70 MB/sec execute 5 sec latency 16.232 ms oleg453-client.virtnet: 1 11134 2.30 MB/sec execute 6 sec latency 15.999 ms oleg453-client.virtnet: 1 11439 2.03 MB/sec execute 7 sec latency 25.884 ms oleg453-client.virtnet: 1 11720 1.97 MB/sec execute 8 sec latency 18.900 ms oleg453-client.virtnet: 1 12085 2.10 MB/sec execute 9 sec latency 18.808 ms oleg453-client.virtnet: 1 12353 1.91 MB/sec execute 10 sec latency 19.635 ms oleg453-client.virtnet: 1 12658 1.76 MB/sec execute 11 sec latency 12.196 ms oleg453-client.virtnet: 1 13012 1.73 MB/sec execute 12 sec latency 12.347 ms oleg453-client.virtnet: 1 13432 1.91 MB/sec execute 13 sec latency 18.236 ms oleg453-client.virtnet: 1 14035 2.03 MB/sec execute 14 sec latency 18.133 ms oleg453-client.virtnet: 1 14436 2.18 MB/sec execute 15 sec latency 16.740 ms oleg453-client.virtnet: 1 14678 2.06 MB/sec execute 16 sec latency 16.130 ms oleg453-client.virtnet: 1 14986 1.96 MB/sec execute 17 sec latency 18.506 ms oleg453-client.virtnet: 1 15281 1.94 MB/sec execute 18 sec latency 18.316 ms oleg453-client.virtnet: 1 15661 2.00 MB/sec execute 19 secoleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 19:49:59 (1723247399) targets are mounted 19:49:59 (1723247399) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12228 3593904 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37412 3569020 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49640 7162924 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg453-server Stopping /mnt/lustre-mds2 (opts:) on oleg453-server 19:50:12 (1723247412) shut down Failover mds2 to oleg453-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0001 19:50:27 (1723247427) targets are mounted 19:50:27 (1723247427) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12200 3594576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37424 3566920 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7161496 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 19:50:40 (1723247440) shut down latency 17.290 ms oleg453-client.virtnet: 1 15937 1.91 MB/sec execute 20 sec latency 19.515 ms oleg453-client.virtnet: 1 16253 1.84 MB/sec execute 21 sec latency 11.741 ms oleg453-client.virtnet: 1 16604 1.82 MB/sec execute 22 sec latency 17.847 ms oleg453-client.virtnet: 1 17111 1.98 MB/sec execute 23 sec latency 17.945 ms oleg453-client.virtnet: 1 17740 2.10 MB/sec execute 24 sec latency 17.632 ms oleg453-client.virtnet: 1 18029 2.07 MB/sec execute 25 sec latency 15.624 ms oleg453-client.virtnet: 1 18262 2.00 MB/sec execute 26 sec latency 16.394 ms oleg453-client.virtnet: 1 18602 1.95 MB/sec execute 27 sec latency 19.851 ms oleg453-client.virtnet: 1 18897 1.93 MB/sec execute 28 sec latency 18.008 ms oleg453-client.virtnet: 1 19263 1.97 MB/sec execute 29 sec latency 17.586 ms oleg453-client.virtnet: 1 19401 1.91 MB/sec execute 30 sec latency 536.017 ms oleg453-client.virtnet: 1 19705 1.86 MB/sec execute 31 sec latency 19.246 ms oleg453-client.virtnet: 1 20037 1.81 MB/sec execute 32 sec latency 13.268 ms oleg453-client.virtnet: 1 20220 1.80 MB/sec execute 33 sec latency 451.772 ms oleg453-client.virtnet: 1 20220 1.74 MB/sec execute 34 sec latency 1451.945 ms oleg453-client.virtnet: 1 20220 1.69 MB/sec execute 35 sec latency 2452.163 ms oleg453-client.virtnet: 1 20220 1.65 MB/sec execute 36 sec latency 3452.353 ms oleg453-client.virtnet: 1 20220 1.60 MB/sec execute 37 sec latency 4452.524 ms oleg453-client.virtnet: 1 20220 1.56 MB/sec execute 38 sec latency 5452.690 ms oleg453-client.virtnet: 1 20220 1.52 MB/sec execute 39 sec latency 6452.852 ms oleg453-client.virtnet: 1 20220 1.48 MB/sec execute 40 sec latency 7453.095 ms oleg453-client.virtnet: 1 20220 1.45 MB/sec execute 41 sec latency 8453.321 ms oleg453-client.virtnet: 1 20220 1.41 MB/sec execute 42 sec latency 9453.541 ms oleg453-client.virtnet: 1 20220 1.38 MB/sec execute 43 sec latency 10453.745 ms oleg453-client.virtnet: 1 20220 1.35 MB/sec execute 44 sec latency 11453.990 ms oleg453-client.virtnet: 1 20220 1.32 MB/sec execute 45 sec latency 12454.180 ms oleg453-client.virtnet: 1 20220 1.29 MB/sec execute 46 sec latency 13454.342 ms oleg453-client.virtnet: 1 20220 1.26 MB/sec execute 47 sec latency 14454.507 ms oleg453-client.virtnet: 1 20220 1.23 MB/sec execute 48 sec latency 15454.692 ms oleg453-client.virtnet: 1 20220 1.21 MB/sec execute 49 sec latency 16454.903 ms oleg453-client.virtnet: 1 20220 1.19 MB/sec execute 50 sec latency 17455.104 ms oleg453-client.virtnet: 1 20220 1.16 MB/sec execute 51 sec latency 18455.293 ms oleg453-client.virtnet: 1 20223 1.14 MB/sec execute 52 sec latency 19447.900 ms oleg453-client.virtnet: 1 20759 1.22 MB/sec execute 53 sec latency 17.714 ms oleg453-client.virtnet: 1 21346 1.29 MB/sec execute 54 sec latency 22.140 ms oleg453-client.virtnet: 1 21625 1.29 MB/sec execute 55 sec latency 18.412 ms oleg453-client.virtnet: 1 21861 1.27 MB/sec execute 56 sec latency 15.712 ms oleg453-client.virtnet: 1 22160 1.26 MB/sec execute 57 sec latency 20.153 ms oleg453-client.virtnet: 1 22470 1.26 MB/sec execute 58 sec latency 18.598 ms oleg453-client.virtnet: 1 22828 1.29 MB/sec execute 59 sec latency 17.894 ms oleg453-client.virtnet: 1 23100 1.28 MB/sec execute 60 sec latency 19.521 ms oleg453-client.virtnet: 1 23409 1.26 MB/sec execute 61 sec latency 12.416 ms oleg453-client.virtnet: 1 23754 1.26 MB/sec execute 62 sec latency 18.130 ms oleg453-client.virtnet: 1 24260 1.33 MB/sec execute 63 sec latency 18.733 ms oleg453-client.virtnetFailover mds1 to oleg453-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 19:50:56 (1723247456) targets are mounted 19:50:56 (1723247456) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12104 3593924 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37584 3566504 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49688 7160428 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg453-server Stopping /mnt/lustre-mds2 (opts:) on oleg453-server 19:51:07 (1723247467) shut down Failover mds2 to oleg453-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0001 19:51:23 (1723247483) targets are mounted 19:51:23 (1723247483) facet_failover done : 1 24888 1.39 MB/sec execute 64 sec latency 17.903 ms oleg453-client.virtnet: 1 25162 1.39 MB/sec execute 65 sec latency 16.298 ms oleg453-client.virtnet: 1 25388 1.37 MB/sec execute 66 sec latency 15.935 ms oleg453-client.virtnet: 1 25701 1.36 MB/sec execute 67 sec latency 17.618 ms oleg453-client.virtnet: 1 26008 1.36 MB/sec execute 68 sec latency 18.721 ms oleg453-client.virtnet: 1 26353 1.38 MB/sec execute 69 sec latency 19.040 ms oleg453-client.virtnet: 1 26612 1.37 MB/sec execute 70 sec latency 18.685 ms oleg453-client.virtnet: 1 26911 1.35 MB/sec execute 71 sec latency 11.968 ms oleg453-client.virtnet: 1 27271 1.35 MB/sec execute 72 sec latency 17.310 ms oleg453-client.virtnet: 1 27796 1.41 MB/sec execute 73 sec latency 18.764 ms oleg453-client.virtnet: 1 28400 1.46 MB/sec execute 74 sec latency 18.290 ms oleg453-client.virtnet: 1 28687 1.46 MB/sec execute 75 sec latency 15.869 ms oleg453-client.virtnet: 1 28932 1.44 MB/sec execute 76 sec latency 15.944 ms oleg453-client.virtnet: 1 29300 1.43 MB/sec execute 77 sec latency 18.613 ms oleg453-client.virtnet: 1 29580 1.43 MB/sec execute 78 sec latency 19.279 ms oleg453-client.virtnet: 1 29932 1.45 MB/sec execute 79 sec latency 18.522 ms oleg453-client.virtnet: 1 30206 1.44 MB/sec execute 80 sec latency 20.716 ms oleg453-client.virtnet: 1 30569 1.42 MB/sec execute 81 sec latency 16.444 ms oleg453-client.virtnet: 1 31338 1.49 MB/sec execute 82 sec latency 11.915 ms oleg453-client.virtnet: 1 32234 1.55 MB/sec execute 83 sec latency 9.220 ms oleg453-client.virtnet: 1 32782 1.54 MB/sec execute 84 sec latency 10.774 ms oleg453-client.virtnet: 1 33480 1.57 MB/sec execute 85 sec latency 8.945 ms oleg453-client.virtnet: 1 33708 1.56 MB/sec execute 86 sec latency 390.814 ms oleg453-client.virtnet: 1 34013 1.54 MB/sec execute 87 sec latency 12.191 ms oleg453-client.virtnet: 1 34392 1.54 MB/sec execute 88 sec latency 16.931 ms oleg453-client.virtnet: 1 34445 1.53 MB/sec execute 89 sec latency 848.632 ms oleg453-client.virtnet: 1 34445 1.52 MB/sec execute 90 sec latency 1848.849 ms oleg453-client.virtnet: 1 34445 1.50 MB/sec execute 91 sec latency 2849.061 ms oleg453-client.virtnet: 1 34445 1.48 MB/sec execute 92 sec latency 3849.271 ms oleg453-client.virtnet: 1 34445 1.47 MB/sec execute 93 sec latency 4849.459 ms oleg453-client.virtnet: 1 34445 1.45 MB/sec execute 94 sec latency 5849.740 ms oleg453-client.virtnet: 1 34445 1.44 MB/sec execute 95 sec latency 6849.932 ms oleg453-client.virtnet: 1 34445 1.42 MB/sec execute 96 sec latency 7850.145 ms oleg453-client.virtnet: 1 34445 1.41 MB/sec execute 97 sec latency 8850.355 ms oleg453-client.virtnet: 1 34445 1.39 MB/sec execute 98 sec latency 9850.544 ms oleg453-client.virtnet: 1 34445 1.38 MB/sec execute 99 sec latency 10850.758 ms oleg453-client.virtnet: 1 34445 1.36 MB/sec execute 100 sec latency 11850.963 ms oleg453-client.virtnet: 1 34445 1.35 MB/sec execute 101 sec latency 12851.148 ms oleg453-client.virtnet: 1 34445 1.34 MB/sec execute 102 sec latency 13851.327 ms oleg453-client.virtnet: 1 34445 1.32 MB/sec execute 103 sec latency 14851.536 ms oleg453-client.virtnet: 1 34445 1.31 MB/sec execute 104 sec latency 15851.754 ms oleg453-client.virtnet: 1 34445 1.30 MB/sec execute 105 sec latency 16851.946 ms oleg453-client.virtnet: 1 34445 1.29 MB/sec execute 106 sec latency 17852.160 ms oleg453-client.virtnet: 1 34445 1.27 MB/sec execute 107 sec latency 18852.366 ms oleg453-client.virtnet: 1 34731 1.29 MB/sec execute 108 soleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12016 3593768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37608 3568452 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7162220 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 19:51:36 (1723247496) shut down Failover mds1 to oleg453-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 19:51:51 (1723247511) targets are mounted 19:51:51 (1723247511) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12200 3591696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37500 3568684 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49700 7160380 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg453-server Stopping /mnt/lustre-mds2 (opts:) on oleg453-server 19:52:04 (1723247524) shut down ec latency 19382.517 ms oleg453-client.virtnet: 1 35331 1.31 MB/sec execute 109 sec latency 18.697 ms oleg453-client.virtnet: 1 35727 1.34 MB/sec execute 110 sec latency 16.581 ms oleg453-client.virtnet: 1 35953 1.33 MB/sec execute 111 sec latency 16.054 ms oleg453-client.virtnet: 1 36240 1.32 MB/sec execute 112 sec latency 18.885 ms oleg453-client.virtnet: 1 36504 1.32 MB/sec execute 113 sec latency 18.144 ms oleg453-client.virtnet: 1 36906 1.34 MB/sec execute 114 sec latency 17.420 ms oleg453-client.virtnet: 1 37176 1.33 MB/sec execute 115 sec latency 16.271 ms oleg453-client.virtnet: 1 37494 1.32 MB/sec execute 116 sec latency 19.445 ms oleg453-client.virtnet: 1 37856 1.32 MB/sec execute 117 sec latency 13.833 ms oleg453-client.virtnet: 1 38267 1.35 MB/sec execute 118 sec latency 18.258 ms oleg453-client.virtnet: 1 38867 1.36 MB/sec execute 119 sec latency 18.245 ms oleg453-client.virtnet: 1 39264 1.39 MB/sec execute 120 sec latency 15.693 ms oleg453-client.virtnet: 1 39490 1.38 MB/sec execute 121 sec latency 16.454 ms oleg453-client.virtnet: 1 39788 1.37 MB/sec execute 122 sec latency 18.218 ms oleg453-client.virtnet: 1 40046 1.36 MB/sec execute 123 sec latency 19.775 ms oleg453-client.virtnet: 1 40447 1.39 MB/sec execute 124 sec latency 18.140 ms oleg453-client.virtnet: 1 40715 1.38 MB/sec execute 125 sec latency 15.952 ms oleg453-client.virtnet: 1 41012 1.37 MB/sec execute 126 sec latency 19.250 ms oleg453-client.virtnet: 1 41356 1.37 MB/sec execute 127 sec latency 12.081 ms oleg453-client.virtnet: 1 41679 1.37 MB/sec execute 128 sec latency 17.961 ms oleg453-client.virtnet: 1 42394 1.41 MB/sec execute 129 sec latency 17.944 ms oleg453-client.virtnet: 1 42796 1.43 MB/sec execute 130 sec latency 12.580 ms oleg453-client.virtnet: 1 43034 1.42 MB/sec execute 131 sec latency 16.162 ms oleg453-client.virtnet: 1 43349 1.41 MB/sec execute 132 sec latency 16.778 ms oleg453-client.virtnet: 1 43663 1.41 MB/sec execute 133 sec latency 17.528 ms oleg453-client.virtnet: 1 44032 1.43 MB/sec execute 134 sec latency 18.431 ms oleg453-client.virtnet: 1 44301 1.42 MB/sec execute 135 sec latency 19.828 ms oleg453-client.virtnet: 1 44603 1.41 MB/sec execute 136 sec latency 13.482 ms oleg453-client.virtnet: 1 45027 1.41 MB/sec execute 137 sec latency 17.018 ms oleg453-client.virtnet: 1 45563 1.44 MB/sec execute 138 sec latency 18.517 ms oleg453-client.virtnet: 1 46171 1.46 MB/sec execute 139 sec latency 17.982 ms oleg453-client.virtnet: 1 46459 1.46 MB/sec execute 140 sec latency 15.254 ms oleg453-client.virtnet: 1 46721 1.45 MB/sec execute 141 sec latency 15.909 ms oleg453-client.virtnet: 1 47031 1.45 MB/sec execute 142 sec latency 41.231 ms oleg453-client.virtnet: 1 47166 1.44 MB/sec execute 143 sec latency 503.945 ms oleg453-client.virtnet: 1 47564 1.46 MB/sec execute 144 sec latency 17.990 ms oleg453-client.virtnet: 1 47859 1.45 MB/sec execute 145 sec latency 20.281 ms oleg453-client.virtnet: 1 47859 1.44 MB/sec execute 146 sec latency 1019.289 ms oleg453-client.virtnet: 1 47859 1.43 MB/sec execute 147 sec latency 2019.472 ms oleg453-client.virtnet: 1 47859 1.42 MB/sec execute 148 sec latency 3019.655 ms oleg453-client.virtnet: 1 47859 1.41 MB/sec execute 149 sec latency 4019.822 ms oleg453-client.virtnet: 1 47859 1.40 MB/sec execute 150 sec latency 5020.031 ms oleg453-client.virtnet: 1 47859 1.39 MB/sec execute 151 sec latency 6020.252 ms oleg453-client.virtnet: 1 47859 1.38 MB/sec execute 152 sec latency 7020.456 ms oleg453-client.virtnet: 1 47859 1.37 MB/sFailover mds2 to oleg453-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0001 19:52:20 (1723247540) targets are mounted 19:52:20 (1723247540) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12200 3593712 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37424 3566752 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7160464 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 19:52:33 (1723247553) shut down Failover mds1 to oleg453-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 19:52:48 (1723247568) targets are mounted 19:52:48 (1723247568) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11924 3594076 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37576 3568668 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162744 1% /mnt/lustre ec execute 153 sec latency 8020.643 ms oleg453-client.virtnet: 1 47859 1.37 MB/sec execute 154 sec latency 9020.876 ms oleg453-client.virtnet: 1 47859 1.36 MB/sec execute 155 sec latency 10021.105 ms oleg453-client.virtnet: 1 47859 1.35 MB/sec execute 156 sec latency 11021.295 ms oleg453-client.virtnet: 1 47859 1.34 MB/sec execute 157 sec latency 12021.472 ms oleg453-client.virtnet: 1 47859 1.33 MB/sec execute 158 sec latency 13021.653 ms oleg453-client.virtnet: 1 47859 1.32 MB/sec execute 159 sec latency 14021.873 ms oleg453-client.virtnet: 1 47859 1.31 MB/sec execute 160 sec latency 15022.075 ms oleg453-client.virtnet: 1 47859 1.31 MB/sec execute 161 sec latency 16022.242 ms oleg453-client.virtnet: 1 47859 1.30 MB/sec execute 162 sec latency 17022.410 ms oleg453-client.virtnet: 1 47859 1.29 MB/sec execute 163 sec latency 18022.619 ms oleg453-client.virtnet: 1 47859 1.28 MB/sec execute 164 sec latency 19022.805 ms oleg453-client.virtnet: 1 47978 1.28 MB/sec execute 165 sec latency 19567.114 ms oleg453-client.virtnet: 1 48314 1.27 MB/sec execute 166 sec latency 13.595 ms oleg453-client.virtnet: 1 48637 1.27 MB/sec execute 167 sec latency 17.932 ms oleg453-client.virtnet: 1 49163 1.30 MB/sec execute 168 sec latency 18.621 ms oleg453-client.virtnet: 1 49767 1.32 MB/sec execute 169 sec latency 17.675 ms oleg453-client.virtnet: 1 50030 1.32 MB/sec execute 170 sec latency 16.749 ms oleg453-client.virtnet: 1 50283 1.31 MB/sec execute 171 sec latency 16.086 ms oleg453-client.virtnet: 1 50575 1.31 MB/sec execute 172 sec latency 18.125 ms oleg453-client.virtnet: 1 50873 1.31 MB/sec execute 173 sec latency 17.905 ms oleg453-client.virtnet: 1 51250 1.32 MB/sec execute 174 sec latency 17.460 ms oleg453-client.virtnet: 1 51512 1.31 MB/sec execute 175 sec latency 19.809 ms oleg453-client.virtnet: 1 51878 1.31 MB/sec execute 176 sec latency 11.909 ms oleg453-client.virtnet: 1 52198 1.31 MB/sec execute 177 sec latency 18.425 ms oleg453-client.virtnet: 1 52856 1.33 MB/sec execute 178 sec latency 18.590 ms oleg453-client.virtnet: 1 53323 1.35 MB/sec execute 179 sec latency 18.368 ms oleg453-client.virtnet: 1 53588 1.35 MB/sec execute 180 sec latency 15.592 ms oleg453-client.virtnet: 1 53857 1.34 MB/sec execute 181 sec latency 16.023 ms oleg453-client.virtnet: 1 54147 1.34 MB/sec execute 182 sec latency 16.821 ms oleg453-client.virtnet: 1 54451 1.34 MB/sec execute 183 sec latency 18.151 ms oleg453-client.virtnet: 1 54810 1.35 MB/sec execute 184 sec latency 17.825 ms oleg453-client.virtnet: 1 55074 1.34 MB/sec execute 185 sec latency 19.773 ms oleg453-client.virtnet: 1 55420 1.34 MB/sec execute 186 sec latency 12.262 ms oleg453-client.virtnet: 1 55803 1.34 MB/sec execute 187 sec latency 18.093 ms oleg453-client.virtnet: 1 56447 1.37 MB/sec execute 188 sec latency 17.860 ms oleg453-client.virtnet: 1 56932 1.38 MB/sec execute 189 sec latency 12.799 ms oleg453-client.virtnet: 1 57199 1.38 MB/sec execute 190 sec latency 15.683 ms oleg453-client.virtnet: 1 57500 1.38 MB/sec execute 191 sec latency 18.515 ms oleg453-client.virtnet: 1 57761 1.37 MB/sec execute 192 sec latency 17.368 ms oleg453-client.virtnet: 1 58176 1.39 MB/sec execute 193 sec latency 17.652 ms oleg453-client.virtnet: 1 58472 1.38 MB/sec execute 194 sec latency 18.590 ms oleg453-client.virtnet: 1 58772 1.37 MB/sec execute 195 sec latency 19.218 ms oleg453-client.virtnet: 1 59134 1.37 MB/sec execute 196 sec latency 12.742 ms oleg453-client.virtnet: 1 59693 1.40 MB/sec execute 197 sec latency 17.701 ms oleg453-client.vtest_70b fail mds2 10 times Failing mds2 on oleg453-server Stopping /mnt/lustre-mds2 (opts:) on oleg453-server 19:53:02 (1723247582) shut down Failover mds2 to oleg453-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0001 19:53:17 (1723247597) targets are mounted 19:53:17 (1723247597) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12040 3593288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37728 3568744 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49768 7162032 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 19:53:31 (1723247611) shut down irtnet: 1 60295 1.41 MB/sec execute 198 sec latency 18.248 ms oleg453-client.virtnet: 1 60588 1.41 MB/sec execute 199 sec latency 16.051 ms oleg453-client.virtnet: 1 60830 1.41 MB/sec execute 200 sec latency 17.176 ms oleg453-client.virtnet: 1 61131 1.40 MB/sec execute 201 sec latency 20.164 ms oleg453-client.virtnet: 1 61455 1.40 MB/sec execute 202 sec latency 18.102 ms oleg453-client.virtnet: 1 61530 1.40 MB/sec execute 203 sec latency 713.847 ms oleg453-client.virtnet: 1 61530 1.39 MB/sec execute 204 sec latency 1714.080 ms oleg453-client.virtnet: 1 61530 1.38 MB/sec execute 205 sec latency 2714.355 ms oleg453-client.virtnet: 1 61530 1.38 MB/sec execute 206 sec latency 3714.584 ms oleg453-client.virtnet: 1 61530 1.37 MB/sec execute 207 sec latency 4714.874 ms oleg453-client.virtnet: 1 61530 1.36 MB/sec execute 208 sec latency 5715.222 ms oleg453-client.virtnet: 1 61530 1.36 MB/sec execute 209 sec latency 6715.540 ms oleg453-client.virtnet: 1 61530 1.35 MB/sec execute 210 sec latency 7715.892 ms oleg453-client.virtnet: 1 61530 1.34 MB/sec execute 211 sec latency 8716.223 ms oleg453-client.virtnet: 1 61530 1.34 MB/sec execute 212 sec latency 9716.508 ms oleg453-client.virtnet: 1 61530 1.33 MB/sec execute 213 sec latency 10716.784 ms oleg453-client.virtnet: 1 61530 1.33 MB/sec execute 214 sec latency 11717.002 ms oleg453-client.virtnet: 1 61530 1.32 MB/sec execute 215 sec latency 12717.226 ms oleg453-client.virtnet: 1 61530 1.31 MB/sec execute 216 sec latency 13717.438 ms oleg453-client.virtnet: 1 61530 1.31 MB/sec execute 217 sec latency 14717.710 ms oleg453-client.virtnet: 1 61530 1.30 MB/sec execute 218 sec latency 15717.922 ms oleg453-client.virtnet: 1 61530 1.30 MB/sec execute 219 sec latency 16718.150 ms oleg453-client.virtnet: 1 61530 1.29 MB/sec execute 220 sec latency 17718.326 ms oleg453-client.virtnet: 1 61530 1.28 MB/sec execute 221 sec latency 18718.509 ms oleg453-client.virtnet: 1 61530 1.28 MB/sec execute 222 sec latency 19718.620 ms oleg453-client.virtnet: 1 61800 1.29 MB/sec execute 223 sec latency 19989.359 ms oleg453-client.virtnet: 1 62071 1.28 MB/sec execute 224 sec latency 19.695 ms oleg453-client.virtnet: 1 62398 1.28 MB/sec execute 225 sec latency 12.254 ms oleg453-client.virtnet: 1 62749 1.28 MB/sec execute 226 sec latency 18.089 ms oleg453-client.virtnet: 1 63257 1.30 MB/sec execute 227 sec latency 17.962 ms oleg453-client.virtnet: 1 63855 1.31 MB/sec execute 228 sec latency 18.692 ms oleg453-client.virtnet: 1 64138 1.31 MB/sec execute 229 sec latency 16.557 ms oleg453-client.virtnet: 1 64360 1.31 MB/sec execute 230 sec latency 16.420 ms oleg453-client.virtnet: 1 64661 1.30 MB/sec execute 231 sec latency 21.466 ms oleg453-client.virtnet: 1 64955 1.30 MB/sec execute 232 sec latency 18.013 ms oleg453-client.virtnet: 1 65323 1.31 MB/sec execute 233 sec latency 19.758 ms oleg453-client.virtnet: 1 65585 1.31 MB/sec execute 234 sec latency 20.237 ms oleg453-client.virtnet: 1 65880 1.30 MB/sec execute 235 sec latency 12.272 ms oleg453-client.virtnet: 1 66231 1.30 MB/sec execute 236 sec latency 12.500 ms oleg453-client.virtnet: 1 66847 1.32 MB/sec execute 237 sec latency 18.648 ms oleg453-client.virtnet: 1 67452 1.34 MB/sec execute 238 sec latency 17.979 ms oleg453-client.virtnet: 1 67761 1.34 MB/sec execute 239 sec latency 15.687 ms oleg453-client.virtnet: 1 68013 1.33 MB/sec execute 240 sec latency 16.156 ms oleg453-client.virtnet: 1 68307 1.33 MB/sec execute 241 sec latency 23.709 ms oleg453-client.virtnet: 1 68600 1.33 MB/sec Failover mds1 to oleg453-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 19:53:46 (1723247626) targets are mounted 19:53:46 (1723247626) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec execute 242 sec latency 18.219 ms oleg453-client.virtnet: 1 68961 1.34 MB/sec execute 243 sec latency 17.999 ms oleg453-client.virtnet: 1 69382 1.33 MB/sec execute 244 sec latency 11.987 ms oleg453-client.virtnet: 1 69780 1.33 MB/sec execute 245 sec latency 12.566 ms oleg453-client.virtnet: 1 70293 1.35 MB/sec execute 246 sec latency 17.771 ms oleg453-client.virtnet: 1 70917 1.37 MB/sec execute 247 sec latency 17.898 ms oleg453-client.virtnet: 1 71216 1.37 MB/sec execute 248 sec latency 16.821 ms oleg453-client.virtnet: 1 71443 1.36 MB/sec execute 249 sec latency 17.438 ms oleg453-client.virtnet: 1 71752 1.36 MB/sec execute 250 sec latency 21.362 ms oleg453-client.virtnet: 1 72054 1.36 MB/sec execute 251 sec latency 17.779 ms oleg453-client.virtnet: 1 72427 1.36 MB/sec execute 252 sec latency 17.431 ms oleg453-client.virtnet: 1 72698 1.36 MB/sec execute 253 sec latency 19.772 ms oleg453-client.virtnet: 1 73020 1.36 MB/sec execute 254 sec latency 11.930 ms oleg453-client.virtnet: 1 73386 1.36 MB/sec execute 255 sec latency 11.748 ms oleg453-client.virtnet: 1 73888 1.37 MB/sec execute 256 sec latency 18.068 ms oleg453-client.virtnet: 1 74530 1.39 MB/sec execute 257 sec latency 18.689 ms oleg453-client.virtnet: 1 74815 1.39 MB/sec execute 258 sec latency 15.763 ms oleg453-client.virtnet: 1 75047 1.38 MB/sec execute 259 sec latency 16.305 ms oleg453-client.virtnet: 1 75358 1.38 MB/sec execute 260 sec latency 20.683 ms oleg453-client.virtnet: 1 75675 1.38 MB/sec execute 261 sec latency 17.818 ms oleg453-client.virtnet: 1 76050 1.39 MB/sec execute 262 sec latency 17.945 ms oleg453-client.virtnet: 1 76330 1.38 MB/sec execute 263 sec latency 19.957 ms oleg453-client.virtnet: 1 76694 1.38 MB/sec execute 264 sec latency 12.069 ms oleg453-client.virtnet: 1 77032 1.38 MB/sec execute 265 sec latency 17.231 ms oleg453-client.virtnet: 1 77567 1.39 MB/sec execute 266 sec latency 17.871 ms oleg453-client.virtnet: 1 78195 1.41 MB/sec execute 267 sec latency 18.000 ms oleg453-client.virtnet: 1 78457 1.41 MB/sec execute 268 sec latency 16.080 ms oleg453-client.virtnet: 1 78747 1.40 MB/sec execute 269 sec latency 15.494 ms oleg453-client.virtnet: 1 79007 1.40 MB/sec execute 270 sec latency 21.770 ms oleg453-client.virtnet: 1 79325 1.40 MB/sec execute 271 sec latency 17.373 ms oleg453-client.virtnet: 1 79680 1.41 MB/sec execute 272 sec latency 18.252 ms oleg453-client.virtnet: 1 79966 1.40 MB/sec execute 273 sec latency 19.975 ms oleg453-client.virtnet: 1 80335 1.40 MB/sec execute 274 sec latency 11.330 ms oleg453-client.virtnet: 1 80708 1.40 MB/sec execute 275 sec latency 17.550 ms oleg453-client.virtnet: 1 81432 1.42 MB/sec execute 276 sec latency 17.864 ms oleg453-client.virtnet: 1 81842 1.43 MB/sec execute 277 sec latency 13.403 ms oleg453-client.virtnet: 1 82069 1.43 MB/sec execute 278 sec latency 15.552 ms oleg453-client.virtnet: 1 82370 1.42 MB/sec execute 279 sec latency 20.180 ms oleg453-client.virtnet: 1 82630 1.42 MB/sec execute 280 sec latency 18.696 ms oleg453-client.virtnet: 1 83031 1.43 MB/sec execute 281 sec latency 18.430 ms oleg453-client.virtnet: 1 83301 1.42 MB/sec execute 282 sec latency 15.575 ms oleg453-client.virtnet: 1 83609 1.42 MB/sec execute 283 sec latency 19.031 ms oleg453-client.virtnet: 1 83973 1.42 MB/sec execute 284 sec latency 11.963 ms oleg453-client.virtnet: 1 84354 1.43 MB/sec execute 285 sec latency 17.751 ms oleg453-client.virtnet: 1 84985 1.44 MB/sec execute 286 sec latency 17.923 ms oleg453-client.virtnet: 1 85381 1.45 MB/sec execute 287 sec latency 13.102 ms oleg453-client.virtnet: 1 85612 1.44 MB/sec execute 288 sec latency 16.070 ms oleg453-client.virtnet: 1 85909 1.44 MB/sec execute 289 sec latency 21.292 ms oleg453-client.virtnet: 1 86167 1.44 MB/sec execute 290 sec latency 18.032 ms oleg453-client.virtnet: 1 86570 1.44 MB/sec execute 291 sec latency 18.055 ms oleg453-client.virtnet: 1 86842 1.44 MB/sec execute 292 sec latency 15.910 ms oleg453-client.virtnet: 1 87155 1.44 MB/sec execute 293 sec latency 20.624 ms oleg453-client.virtnet: 1 87515 1.44 MB/sec execute 294 sec latency 11.639 ms oleg453-client.virtnet: 1 87928 1.45 MB/sec execute 295 sec latency 18.199 ms oleg453-client.virtnet: 1 88534 1.45 MB/sec execute 296 sec latency 18.160 ms oleg453-client.virtnet: 1 88934 1.46 MB/sec execute 297 sec latency 13.415 ms oleg453-client.virtnet: 1 89164 1.46 MB/sec execute 298 sec latency 15.909 ms oleg453-client.virtnet: 1 89468 1.45 MB/sec execute 299 sec latency 22.198 ms oleg453-client.virtnet: 1 cleanup 300 sec oleg453-client.virtnet: 0 cleanup 300 sec oleg453-client.virtnet: oleg453-client.virtnet: Operation Count AvgLat MaxLat oleg453-client.virtnet: ---------------------------------------- oleg453-client.virtnet: NTCreateX 13998 11.537 19567.087 oleg453-client.virtnet: Close 10272 3.761 19989.330 oleg453-client.virtnet: Rename 591 13.612 22.065 oleg453-client.virtnet: Unlink 2839 2.908 8.161 oleg453-client.virtnet: Qpathinfo 12672 2.391 15.295 oleg453-client.virtnet: Qfileinfo 2208 0.200 2.424 oleg453-client.virtnet: Qfsinfo 2330 0.351 12.161 oleg453-client.virtnet: Sfileinfo 1134 25.265 19382.501 oleg453-client.virtnet: Find 4905 0.606 15.905 oleg453-client.virtnet: WriteX 6905 1.334 10.446 oleg453-client.virtnet: ReadX 21972 0.026 6.380 oleg453-client.virtnet: LockX 46 1.723 2.363 oleg453-client.virtnet: UnlockX 46 1.945 2.441 oleg453-client.virtnet: Flush 986 9.859 536.000 oleg453-client.virtnet: oleg453-client.virtnet: Throughput 1.45447 MB/sec 1 clients 1 procs max_latency=19989.359 ms oleg453-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg453-client.virtnet at Fri Aug 9 19:54:39 EDT 2024 with return code 0 oleg453-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg453-client.virtnet oleg453-client.virtnet: /mnt/lustre/d70b.replay-single/oleg453-client.virtnet /mnt/lustre/d70b.replay-single/oleg453-client.virtnet oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg453-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg453-client.virtnet: removed directory: 'clients/client0' oleg453-client.virtnet: removed directory: 'clients' oleg453-client.virtnet: removed 'client.txt' oleg453-client.virtnet: /mnt/lustre/d70b.replay-single/oleg453-client.virtnet oleg453-client.virtnet: dbench successfully finished