== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 13:15:44 (1709057744) Starting client oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre Started clients oleg210-client.virtnet: 192.168.202.110@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg210-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=22087 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2508 1285180 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1852 3605168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26404 3555896 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28256 7161064 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 13:15:54 (1709057754) shut down Failover mds1 to oleg210-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 13:16:10 (1709057770) targets are mounted 13:16:10 (1709057770) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1996 1285692 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2032 3592716 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26416 3569564 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28448 7162280 1% /mnt/lustre oleg210-client.virtnet: looking for dbench program oleg210-client.virtnet: /usr/bin/dbench oleg210-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg210-client.virtnet oleg210-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg210-client.virtnet' oleg210-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg210-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg210-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg210-client.virtnet at Tue Feb 27 13:15:45 EST 2024 oleg210-client.virtnet: waiting for dbench pid 22111 oleg210-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg210-client.virtnet: oleg210-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg210-client.virtnet: failed to create barrier semaphore oleg210-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg210-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg210-client.virtnet: releasing clients oleg210-client.virtnet: 1 215 8.36 MB/sec warmup 1 sec latency 40.304 ms oleg210-client.virtnet: 1 445 7.82 MB/sec warmup 2 sec latency 41.340 ms oleg210-client.virtnet: 1 708 7.07 MB/sec warmup 3 sec latency 37.317 ms oleg210-client.virtnet: 1 893 5.42 MB/sec warmup 4 sec latency 41.415 ms oleg210-client.virtnet: 1 980 4.38 MB/sec warmup 5 sec latency 494.054 ms oleg210-client.virtnet: 1 1244 3.85 MB/sec warmup 6 sec latency 23.398 ms oleg210-client.virtnet: 1 1538 3.73 MB/sec warmup 7 sec latency 29.935 ms oleg210-client.virtnet: 1 1650 3.27 MB/sec warmup 8 sec latency 524.466 ms oleg210-client.virtnet: 1 1650 2.91 MB/sec warmup 9 sec latency 1526.139 ms oleg210-client.virtnet: 1 1650 2.62 MB/sec warmup 10 sec latency 2526.384 ms oleg210-client.virtnet: 1 1650 2.38 MB/sec warmup 11 sec latency 3526.575 ms oleg210-client.virtnet: 1 1650 2.18 MB/sec warmup 12 sec latency 4527.383 ms oleg210-client.virtnet: 1 1650 2.01 MB/sec warmup 13 sec latency 5527.710 ms oleg210-client.virtnet: 1 1650 1.87 MB/sec warmup 14 sec latency 6527.886 ms oleg210-client.virtnet: 1 1650 1.75 MB/sec warmup 15 sec latency 7528.567 ms oleg210-client.virtnet: 1 1650 1.64 MB/sec warmup 16 sec latency 8528.794 ms oleg210-client.virtnet: 1 1650 1.54 MB/sec warmup 17 sec latency 9529.014 ms oleg210-client.virtnet: 1 1650 1.45 MB/sec warmup 18 sec latency 10530.750 ms oleg210-client.virtnet: 1 1650 1.38 MB/sec warmup 19 sec latency 11531.787 ms oleg210-client.virtnet: 1 1650 1.31 MB/sec warmup 20 sec latency 12532.016 ms oleg210-client.virtnet: 1 1650 1.25 MB/sec warmup 21 sec latency 13532.295 ms oleg210-client.virtnet: 1 1650 1.19 MB/sec warmup 22 sec latency 14532.455 ms oleg210-client.virtnet: 1 1650 1.14 MB/sec warmup 23 sec latency 15532.630 ms oleg210-client.virtnet: 1 1650 1.09 MB/sec warmup 24 sec latency 16532.744 ms oleg210-client.virtnet: 1 1650 1.05 MB/sec warmup 25 sec latency 17532.926 ms oleg210-client.virtnet: 1 1650 1.01 MB/sec warmup 26 sec latency 18533.303 ms oleg210-client.virtnet: 1 1650 0.97 MB/sec warmup 27 sec latency 19533.448 ms oleg210-client.virtnet: 1 1832 0.94 MB/sec warmup 28 sec latency 19772.893 ms oleg210-client.virtnet: 1 2094 0.92 MB/sec warmup 29 sec latency 93.450 ms oleg210-client.virtnet: 1 2386 0.93 MB/sec warmup 30 sec latency 32.743 ms oleg210-client.virtnet: 1 2793 1.03 MB/sec warmup 31 sec latency 25.930 ms oleg210-client.virtnet: 1 3275 1.10 MB/sec warmup 32 sec latency 68.153 ms oleg210-client.virtnet: 1 3570 1.16 MB/sec warmup 33 sec latentest_70b fail mds2 2 times Failing mds2 on oleg210-server Stopping /mnt/lustre-mds2 (opts:) on oleg210-server 13:16:23 (1709057783) shut down Failover mds2 to oleg210-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0001 13:16:38 (1709057798) targets are mounted 13:16:38 (1709057798) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13212 3592816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36808 3569480 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50020 7162296 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 13:16:51 (1709057811) shut down Failover mds1 to oleg210-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 cy 39.054 ms oleg210-client.virtnet: 1 3797 1.17 MB/sec warmup 34 sec latency 29.453 ms oleg210-client.virtnet: 1 3975 1.14 MB/sec warmup 35 sec latency 28.330 ms oleg210-client.virtnet: 1 4262 1.12 MB/sec warmup 36 sec latency 28.601 ms oleg210-client.virtnet: 1 4491 1.10 MB/sec warmup 37 sec latency 22.858 ms oleg210-client.virtnet: 1 4810 1.11 MB/sec warmup 38 sec latency 25.534 ms oleg210-client.virtnet: 1 5170 1.16 MB/sec warmup 39 sec latency 26.075 ms oleg210-client.virtnet: 1 5475 1.14 MB/sec warmup 40 sec latency 20.164 ms oleg210-client.virtnet: 1 5988 1.15 MB/sec warmup 41 sec latency 15.079 ms oleg210-client.virtnet: 1 6731 1.27 MB/sec warmup 42 sec latency 16.430 ms oleg210-client.virtnet: 1 7360 1.37 MB/sec warmup 43 sec latency 13.807 ms oleg210-client.virtnet: 1 7632 1.35 MB/sec warmup 44 sec latency 19.964 ms oleg210-client.virtnet: 1 7977 1.33 MB/sec warmup 45 sec latency 23.394 ms oleg210-client.virtnet: 1 8315 1.33 MB/sec warmup 46 sec latency 21.090 ms oleg210-client.virtnet: 1 8690 1.37 MB/sec warmup 47 sec latency 19.504 ms oleg210-client.virtnet: 1 8969 1.35 MB/sec warmup 48 sec latency 24.205 ms oleg210-client.virtnet: 1 9428 1.35 MB/sec warmup 49 sec latency 14.885 ms oleg210-client.virtnet: 1 9699 1.34 MB/sec warmup 50 sec latency 21.158 ms oleg210-client.virtnet: 1 10288 1.42 MB/sec warmup 51 sec latency 31.805 ms oleg210-client.virtnet: 1 10742 1.48 MB/sec warmup 52 sec latency 137.223 ms oleg210-client.virtnet: 1 10742 1.45 MB/sec warmup 53 sec latency 1137.616 ms oleg210-client.virtnet: 1 10742 1.42 MB/sec warmup 54 sec latency 2137.827 ms oleg210-client.virtnet: 1 10742 1.40 MB/sec warmup 55 sec latency 3138.392 ms oleg210-client.virtnet: 1 10786 1.37 MB/sec warmup 56 sec latency 3959.654 ms oleg210-client.virtnet: 1 10970 1.37 MB/sec warmup 57 sec latency 30.423 ms oleg210-client.virtnet: 1 11165 1.35 MB/sec warmup 58 sec latency 31.461 ms oleg210-client.virtnet: 1 11529 1.34 MB/sec warmup 59 sec latency 20.211 ms oleg210-client.virtnet: 1 12090 3.03 MB/sec execute 1 sec latency 31.589 ms oleg210-client.virtnet: 1 12287 1.59 MB/sec execute 2 sec latency 258.344 ms oleg210-client.virtnet: 1 12542 1.13 MB/sec execute 3 sec latency 340.486 ms oleg210-client.virtnet: 1 12999 1.23 MB/sec execute 4 sec latency 14.180 ms oleg210-client.virtnet: 1 13144 1.15 MB/sec execute 5 sec latency 432.692 ms oleg210-client.virtnet: 1 13144 0.96 MB/sec execute 6 sec latency 1432.878 ms oleg210-client.virtnet: 1 13144 0.82 MB/sec execute 7 sec latency 2433.061 ms oleg210-client.virtnet: 1 13144 0.72 MB/sec execute 8 sec latency 3433.278 ms oleg210-client.virtnet: 1 13144 0.64 MB/sec execute 9 sec latency 4434.860 ms oleg210-client.virtnet: 1 13144 0.57 MB/sec execute 10 sec latency 5435.694 ms oleg210-client.virtnet: 1 13144 0.52 MB/sec execute 11 sec latency 6435.918 ms oleg210-client.virtnet: 1 13144 0.48 MB/sec execute 12 sec latency 7436.142 ms oleg210-client.virtnet: 1 13144 0.44 MB/sec execute 13 sec latency 8436.452 ms oleg210-client.virtnet: 1 13144 0.41 MB/sec execute 14 sec latency 9436.633 ms oleg210-client.virtnet: 1 13144 0.38 MB/sec execute 15 sec latency 10436.837 ms oleg210-client.virtnet: 1 13144 0.36 MB/sec execute 16 sec latency 11436.994 ms oleg210-client.virtnet: 1 13144 0.34 MB/sec execute 17 sec latency 12437.438 ms oleg210-client.virtnet: 1 13144 0.32 MB/sec execute 18 sec latency 13437.802 ms oleg210-client.virtnet: 1 13144 0.30 MB/sec executeoleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 13:17:07 (1709057827) targets are mounted 13:17:07 (1709057827) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13312 3592116 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36844 3569160 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50156 7161276 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg210-server Stopping /mnt/lustre-mds2 (opts:) on oleg210-server 13:17:19 (1709057839) shut down Failover mds2 to oleg210-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0001 13:17:36 (1709057856) targets are mounted 13:17:36 (1709057856) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13540 3591960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36648 3569784 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50188 7161744 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 13:17:48 (1709057868) shut down 19 sec latency 14437.969 ms oleg210-client.virtnet: 1 13144 0.29 MB/sec execute 20 sec latency 15438.346 ms oleg210-client.virtnet: 1 13144 0.27 MB/sec execute 21 sec latency 16439.195 ms oleg210-client.virtnet: 1 13144 0.26 MB/sec execute 22 sec latency 17440.142 ms oleg210-client.virtnet: 1 13144 0.25 MB/sec execute 23 sec latency 18440.376 ms oleg210-client.virtnet: 1 13144 0.24 MB/sec execute 24 sec latency 19440.609 ms oleg210-client.virtnet: 1 13576 0.42 MB/sec execute 25 sec latency 19734.712 ms oleg210-client.virtnet: 1 14194 0.59 MB/sec execute 26 sec latency 21.223 ms oleg210-client.virtnet: 1 14453 0.62 MB/sec execute 27 sec latency 25.127 ms oleg210-client.virtnet: 1 14694 0.61 MB/sec execute 28 sec latency 25.294 ms oleg210-client.virtnet: 1 14986 0.60 MB/sec execute 29 sec latency 25.356 ms oleg210-client.virtnet: 1 15271 0.63 MB/sec execute 30 sec latency 22.306 ms oleg210-client.virtnet: 1 15676 0.72 MB/sec execute 31 sec latency 20.600 ms oleg210-client.virtnet: 1 15920 0.70 MB/sec execute 32 sec latency 32.194 ms oleg210-client.virtnet: 1 16205 0.69 MB/sec execute 33 sec latency 25.093 ms oleg210-client.virtnet: 1 16576 0.71 MB/sec execute 34 sec latency 17.223 ms oleg210-client.virtnet: 1 16958 0.80 MB/sec execute 35 sec latency 24.786 ms oleg210-client.virtnet: 1 17428 0.85 MB/sec execute 36 sec latency 35.139 ms oleg210-client.virtnet: 1 17876 0.94 MB/sec execute 37 sec latency 20.912 ms oleg210-client.virtnet: 1 18093 0.95 MB/sec execute 38 sec latency 22.993 ms oleg210-client.virtnet: 1 18346 0.93 MB/sec execute 39 sec latency 22.057 ms oleg210-client.virtnet: 1 18654 0.92 MB/sec execute 40 sec latency 22.141 ms oleg210-client.virtnet: 1 18983 0.93 MB/sec execute 41 sec latency 25.381 ms oleg210-client.virtnet: 1 19322 0.98 MB/sec execute 42 sec latency 25.635 ms oleg210-client.virtnet: 1 19549 0.97 MB/sec execute 43 sec latency 24.189 ms oleg210-client.virtnet: 1 19777 0.95 MB/sec execute 44 sec latency 35.292 ms oleg210-client.virtnet: 1 19955 0.93 MB/sec execute 45 sec latency 67.661 ms oleg210-client.virtnet: 1 20181 0.94 MB/sec execute 46 sec latency 40.228 ms oleg210-client.virtnet: 1 20359 0.94 MB/sec execute 47 sec latency 55.759 ms oleg210-client.virtnet: 1 20769 1.02 MB/sec execute 48 sec latency 33.845 ms oleg210-client.virtnet: 1 21290 1.09 MB/sec execute 49 sec latency 53.394 ms oleg210-client.virtnet: 1 21416 1.08 MB/sec execute 50 sec latency 474.274 ms oleg210-client.virtnet: 1 21416 1.06 MB/sec execute 51 sec latency 1474.791 ms oleg210-client.virtnet: 1 21416 1.04 MB/sec execute 52 sec latency 2474.951 ms oleg210-client.virtnet: 1 21422 1.02 MB/sec execute 53 sec latency 3452.734 ms oleg210-client.virtnet: 1 21679 1.02 MB/sec execute 54 sec latency 32.444 ms oleg210-client.virtnet: 1 22010 1.01 MB/sec execute 55 sec latency 21.112 ms oleg210-client.virtnet: 1 22354 1.02 MB/sec execute 56 sec latency 16.793 ms oleg210-client.virtnet: 1 22793 1.06 MB/sec execute 57 sec latency 23.354 ms oleg210-client.virtnet: 1 23135 1.05 MB/sec execute 58 sec latency 15.603 ms oleg210-client.virtnet: 1 23312 1.03 MB/sec execute 59 sec latency 422.469 ms oleg210-client.virtnet: 1 23671 1.04 MB/sec execute 60 sec latency 25.115 ms oleg210-client.virtnet: 1 24044 1.09 MB/sec execute 61 sec latency 20.533 ms oleg210-client.virtnet: 1 24220 1.08 MB/sec execute 62 sec latency 738.661 ms oleg210-client.virtnet: 1 24220 1.07 MB/sec execute 63 sec latency 1738.932 ms oleg210-client.virtnet: 1 Failover mds1 to oleg210-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 13:18:04 (1709057884) targets are mounted 13:18:04 (1709057884) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13464 3591268 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36784 3569768 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50248 7161036 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg210-server Stopping /mnt/lustre-mds2 (opts:) on oleg210-server 13:18:17 (1709057897) shut down Failover mds2 to oleg210-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0001 13:18:33 (1709057913) targets are mounted 13:18:33 (1709057913) facet_failover done 24220 1.05 MB/sec execute 64 sec latency 2739.231 ms oleg210-client.virtnet: 1 24220 1.03 MB/sec execute 65 sec latency 3739.394 ms oleg210-client.virtnet: 1 24220 1.02 MB/sec execute 66 sec latency 4739.904 ms oleg210-client.virtnet: 1 24220 1.00 MB/sec execute 67 sec latency 5742.190 ms oleg210-client.virtnet: 1 24220 0.99 MB/sec execute 68 sec latency 6742.366 ms oleg210-client.virtnet: 1 24220 0.97 MB/sec execute 69 sec latency 7742.922 ms oleg210-client.virtnet: 1 24220 0.96 MB/sec execute 70 sec latency 8743.554 ms oleg210-client.virtnet: 1 24220 0.95 MB/sec execute 71 sec latency 9743.881 ms oleg210-client.virtnet: 1 24220 0.93 MB/sec execute 72 sec latency 10744.124 ms oleg210-client.virtnet: 1 24220 0.92 MB/sec execute 73 sec latency 11744.505 ms oleg210-client.virtnet: 1 24220 0.91 MB/sec execute 74 sec latency 12745.684 ms oleg210-client.virtnet: 1 24220 0.90 MB/sec execute 75 sec latency 13745.866 ms oleg210-client.virtnet: 1 24220 0.88 MB/sec execute 76 sec latency 14746.439 ms oleg210-client.virtnet: 1 24220 0.87 MB/sec execute 77 sec latency 15748.943 ms oleg210-client.virtnet: 1 24220 0.86 MB/sec execute 78 sec latency 16750.296 ms oleg210-client.virtnet: 1 24220 0.85 MB/sec execute 79 sec latency 17751.023 ms oleg210-client.virtnet: 1 24220 0.84 MB/sec execute 80 sec latency 18751.522 ms oleg210-client.virtnet: 1 24220 0.83 MB/sec execute 81 sec latency 19751.769 ms oleg210-client.virtnet: 1 24336 0.83 MB/sec execute 82 sec latency 20225.609 ms oleg210-client.virtnet: 1 24844 0.88 MB/sec execute 83 sec latency 33.195 ms oleg210-client.virtnet: 1 25112 0.88 MB/sec execute 84 sec latency 24.265 ms oleg210-client.virtnet: 1 25329 0.88 MB/sec execute 85 sec latency 24.484 ms oleg210-client.virtnet: 1 25588 0.87 MB/sec execute 86 sec latency 31.367 ms oleg210-client.virtnet: 1 25806 0.87 MB/sec execute 87 sec latency 29.697 ms oleg210-client.virtnet: 1 26042 0.87 MB/sec execute 88 sec latency 37.826 ms oleg210-client.virtnet: 1 26337 0.89 MB/sec execute 89 sec latency 26.840 ms oleg210-client.virtnet: 1 26497 0.89 MB/sec execute 90 sec latency 30.831 ms oleg210-client.virtnet: 1 26668 0.88 MB/sec execute 91 sec latency 70.360 ms oleg210-client.virtnet: 1 26893 0.87 MB/sec execute 92 sec latency 30.972 ms oleg210-client.virtnet: 1 27218 0.87 MB/sec execute 93 sec latency 22.453 ms oleg210-client.virtnet: 1 27445 0.88 MB/sec execute 94 sec latency 36.436 ms oleg210-client.virtnet: 1 27861 0.92 MB/sec execute 95 sec latency 29.187 ms oleg210-client.virtnet: 1 28362 0.96 MB/sec execute 96 sec latency 40.847 ms oleg210-client.virtnet: 1 28581 0.95 MB/sec execute 97 sec latency 59.859 ms oleg210-client.virtnet: 1 28769 0.95 MB/sec execute 98 sec latency 42.826 ms oleg210-client.virtnet: 1 28977 0.95 MB/sec execute 99 sec latency 24.558 ms oleg210-client.virtnet: 1 29227 0.94 MB/sec execute 100 sec latency 31.345 ms oleg210-client.virtnet: 1 29421 0.94 MB/sec execute 101 sec latency 25.171 ms oleg210-client.virtnet: 1 29691 0.94 MB/sec execute 102 sec latency 27.184 ms oleg210-client.virtnet: 1 29983 0.96 MB/sec execute 103 sec latency 33.931 ms oleg210-client.virtnet: 1 30146 0.95 MB/sec execute 104 sec latency 38.692 ms oleg210-client.virtnet: 1 30406 0.94 MB/sec execute 105 sec latency 34.901 ms oleg210-client.virtnet: 1 30767 0.95 MB/sec execute 106 sec latency 25.504 ms oleg210-client.virtnet: 1 30993 0.95 MB/sec execute 107 sec latency 186.447 ms oleg210-client.virtnet: 1 30993 0.94 MB/sec execute 108 sec loleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13384 3590584 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36784 3569648 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50168 7160232 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 13:18:47 (1709057927) shut down Failover mds1 to oleg210-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 13:19:03 (1709057943) targets are mounted 13:19:03 (1709057943) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13380 3592756 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36656 3569940 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50036 7162696 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg210-server Stopping /mnt/lustre-mds2 (opts:) on oleg210-server 13:19:17 (1709057957) shut down atency 1186.711 ms oleg210-client.virtnet: 1 30993 0.93 MB/sec execute 109 sec latency 2186.907 ms oleg210-client.virtnet: 1 30993 0.92 MB/sec execute 110 sec latency 3187.125 ms oleg210-client.virtnet: 1 31039 0.92 MB/sec execute 111 sec latency 3986.922 ms oleg210-client.virtnet: 1 31652 0.96 MB/sec execute 112 sec latency 33.318 ms oleg210-client.virtnet: 1 32037 0.98 MB/sec execute 113 sec latency 26.548 ms oleg210-client.virtnet: 1 32234 0.99 MB/sec execute 114 sec latency 29.567 ms oleg210-client.virtnet: 1 32369 0.98 MB/sec execute 115 sec latency 99.469 ms oleg210-client.virtnet: 1 32567 0.97 MB/sec execute 116 sec latency 27.695 ms oleg210-client.virtnet: 1 32756 0.97 MB/sec execute 117 sec latency 85.431 ms oleg210-client.virtnet: 1 32851 0.96 MB/sec execute 118 sec latency 464.339 ms oleg210-client.virtnet: 1 33098 0.96 MB/sec execute 119 sec latency 27.538 ms oleg210-client.virtnet: 1 33351 0.98 MB/sec execute 120 sec latency 48.837 ms oleg210-client.virtnet: 1 33434 0.97 MB/sec execute 121 sec latency 401.464 ms oleg210-client.virtnet: 1 33434 0.97 MB/sec execute 122 sec latency 1402.290 ms oleg210-client.virtnet: 1 33434 0.96 MB/sec execute 123 sec latency 2402.516 ms oleg210-client.virtnet: 1 33434 0.95 MB/sec execute 124 sec latency 3403.045 ms oleg210-client.virtnet: 1 33434 0.94 MB/sec execute 125 sec latency 4404.338 ms oleg210-client.virtnet: 1 33434 0.93 MB/sec execute 126 sec latency 5405.140 ms oleg210-client.virtnet: 1 33434 0.93 MB/sec execute 127 sec latency 6405.924 ms oleg210-client.virtnet: 1 33434 0.92 MB/sec execute 128 sec latency 7406.177 ms oleg210-client.virtnet: 1 33434 0.91 MB/sec execute 129 sec latency 8406.772 ms oleg210-client.virtnet: 1 33434 0.91 MB/sec execute 130 sec latency 9407.059 ms oleg210-client.virtnet: 1 33434 0.90 MB/sec execute 131 sec latency 10407.253 ms oleg210-client.virtnet: 1 33434 0.89 MB/sec execute 132 sec latency 11408.074 ms oleg210-client.virtnet: 1 33434 0.89 MB/sec execute 133 sec latency 12409.899 ms oleg210-client.virtnet: 1 33434 0.88 MB/sec execute 134 sec latency 13410.159 ms oleg210-client.virtnet: 1 33434 0.87 MB/sec execute 135 sec latency 14410.989 ms oleg210-client.virtnet: 1 33434 0.87 MB/sec execute 136 sec latency 15411.179 ms oleg210-client.virtnet: 1 33434 0.86 MB/sec execute 137 sec latency 16412.008 ms oleg210-client.virtnet: 1 33434 0.85 MB/sec execute 138 sec latency 17412.329 ms oleg210-client.virtnet: 1 33434 0.85 MB/sec execute 139 sec latency 18412.513 ms oleg210-client.virtnet: 1 33434 0.84 MB/sec execute 140 sec latency 19412.693 ms oleg210-client.virtnet: 1 33445 0.84 MB/sec execute 141 sec latency 20341.049 ms oleg210-client.virtnet: 1 33693 0.83 MB/sec execute 142 sec latency 23.090 ms oleg210-client.virtnet: 1 33996 0.83 MB/sec execute 143 sec latency 22.478 ms oleg210-client.virtnet: 1 34315 0.83 MB/sec execute 144 sec latency 27.526 ms oleg210-client.virtnet: 1 34572 0.83 MB/sec execute 145 sec latency 29.378 ms oleg210-client.virtnet: 1 34977 0.86 MB/sec execute 146 sec latency 57.501 ms oleg210-client.virtnet: 1 35322 0.87 MB/sec execute 147 sec latency 131.131 ms oleg210-client.virtnet: 1 35592 0.88 MB/sec execute 148 sec latency 43.246 ms oleg210-client.virtnet: 1 35798 0.88 MB/sec execute 149 sec latency 30.958 ms oleg210-client.virtnet: 1 35926 0.88 MB/sec execute 150 sec latency 114.966 ms oleg210-client.virtnet: 1 36024 0.87 MB/sec execute 151 sec latency 53.071 ms oleg210-client.virtnet: 1 36259 0.87 MB/sec execute 152 sec latency 36.377 ms oleg210-clieFailover mds2 to oleg210-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0001 13:19:33 (1709057973) targets are mounted 13:19:33 (1709057973) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13396 3590572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36764 3569640 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50160 7160212 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 13:19:46 (1709057986) shut down Failover mds1 to oleg210-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 13:20:01 (1709058001) targets are mounted 13:20:01 (1709058001) facet_failover done nt.virtnet: 1 36452 0.87 MB/sec execute 153 sec latency 28.646 ms oleg210-client.virtnet: 1 36688 0.87 MB/sec execute 154 sec latency 55.046 ms oleg210-client.virtnet: 1 36985 0.88 MB/sec execute 155 sec latency 23.134 ms oleg210-client.virtnet: 1 37205 0.88 MB/sec execute 156 sec latency 24.064 ms oleg210-client.virtnet: 1 37492 0.87 MB/sec execute 157 sec latency 20.143 ms oleg210-client.virtnet: 1 37866 0.88 MB/sec execute 158 sec latency 16.857 ms oleg210-client.virtnet: 1 38117 0.88 MB/sec execute 159 sec latency 44.440 ms oleg210-client.virtnet: 1 38736 0.91 MB/sec execute 160 sec latency 24.388 ms oleg210-client.virtnet: 1 39145 0.93 MB/sec execute 161 sec latency 24.820 ms oleg210-client.virtnet: 1 39379 0.93 MB/sec execute 162 sec latency 29.643 ms oleg210-client.virtnet: 1 39608 0.93 MB/sec execute 163 sec latency 20.365 ms oleg210-client.virtnet: 1 39887 0.92 MB/sec execute 164 sec latency 23.078 ms oleg210-client.virtnet: 1 40124 0.93 MB/sec execute 165 sec latency 53.867 ms oleg210-client.virtnet: 1 40459 0.94 MB/sec execute 166 sec latency 30.600 ms oleg210-client.virtnet: 1 40722 0.94 MB/sec execute 167 sec latency 19.392 ms oleg210-client.virtnet: 1 40753 0.93 MB/sec execute 168 sec latency 895.007 ms oleg210-client.virtnet: 1 40753 0.92 MB/sec execute 169 sec latency 1896.622 ms oleg210-client.virtnet: 1 40753 0.92 MB/sec execute 170 sec latency 2896.936 ms oleg210-client.virtnet: 1 40857 0.91 MB/sec execute 171 sec latency 3381.385 ms oleg210-client.virtnet: 1 41116 0.91 MB/sec execute 172 sec latency 46.227 ms oleg210-client.virtnet: 1 41437 0.91 MB/sec execute 173 sec latency 22.749 ms oleg210-client.virtnet: 1 41804 0.93 MB/sec execute 174 sec latency 29.194 ms oleg210-client.virtnet: 1 42358 0.94 MB/sec execute 175 sec latency 20.756 ms oleg210-client.virtnet: 1 42720 0.96 MB/sec execute 176 sec latency 24.989 ms oleg210-client.virtnet: 1 42913 0.96 MB/sec execute 177 sec latency 26.982 ms oleg210-client.virtnet: 1 43105 0.96 MB/sec execute 178 sec latency 21.915 ms oleg210-client.virtnet: 1 43344 0.95 MB/sec execute 179 sec latency 40.571 ms oleg210-client.virtnet: 1 43482 0.95 MB/sec execute 180 sec latency 439.638 ms oleg210-client.virtnet: 1 43482 0.94 MB/sec execute 181 sec latency 1439.823 ms oleg210-client.virtnet: 1 43482 0.94 MB/sec execute 182 sec latency 2440.714 ms oleg210-client.virtnet: 1 43482 0.93 MB/sec execute 183 sec latency 3441.305 ms oleg210-client.virtnet: 1 43482 0.93 MB/sec execute 184 sec latency 4442.517 ms oleg210-client.virtnet: 1 43482 0.92 MB/sec execute 185 sec latency 5442.749 ms oleg210-client.virtnet: 1 43482 0.92 MB/sec execute 186 sec latency 6442.922 ms oleg210-client.virtnet: 1 43482 0.91 MB/sec execute 187 sec latency 7443.171 ms oleg210-client.virtnet: 1 43482 0.91 MB/sec execute 188 sec latency 8443.470 ms oleg210-client.virtnet: 1 43482 0.90 MB/sec execute 189 sec latency 9443.642 ms oleg210-client.virtnet: 1 43482 0.90 MB/sec execute 190 sec latency 10444.006 ms oleg210-client.virtnet: 1 43482 0.89 MB/sec execute 191 sec latency 11444.212 ms oleg210-client.virtnet: 1 43482 0.89 MB/sec execute 192 sec latency 12444.678 ms oleg210-client.virtnet: 1 43482 0.88 MB/sec execute 193 sec latency 13445.226 ms oleg210-client.virtnet: 1 43482 0.88 MB/sec execute 194 sec latency 14445.754 ms oleg210-client.virtnet: 1 43482 0.88 MB/sec execute 195 sec latency 15446.123 ms oleg210-client.virtnet: 1 43482 0.87 MB/sec execute 196 sec latency 16446.361 ms oleg210-client.virtnet: 1 43482 0.87 MB/sec eoleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13372 3593620 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36648 3570064 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50020 7163684 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg210-server Stopping /mnt/lustre-mds2 (opts:) on oleg210-server 13:20:15 (1709058015) shut down Failover mds2 to oleg210-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0001 13:20:32 (1709058032) targets are mounted 13:20:32 (1709058032) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1976 1285712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13448 3593456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36648 3569808 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50096 7163264 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server xecute 197 sec latency 17446.537 ms oleg210-client.virtnet: 1 43482 0.86 MB/sec execute 198 sec latency 18446.772 ms oleg210-client.virtnet: 1 43482 0.86 MB/sec execute 199 sec latency 19447.175 ms oleg210-client.virtnet: 1 43522 0.85 MB/sec execute 200 sec latency 20144.269 ms oleg210-client.virtnet: 1 43765 0.86 MB/sec execute 201 sec latency 30.789 ms oleg210-client.virtnet: 1 44091 0.87 MB/sec execute 202 sec latency 25.032 ms oleg210-client.virtnet: 1 44322 0.86 MB/sec execute 203 sec latency 29.142 ms oleg210-client.virtnet: 1 44617 0.86 MB/sec execute 204 sec latency 20.050 ms oleg210-client.virtnet: 1 44941 0.86 MB/sec execute 205 sec latency 34.709 ms oleg210-client.virtnet: 1 45235 0.87 MB/sec execute 206 sec latency 30.398 ms oleg210-client.virtnet: 1 45796 0.89 MB/sec execute 207 sec latency 24.220 ms oleg210-client.virtnet: 1 46179 0.90 MB/sec execute 208 sec latency 25.864 ms oleg210-client.virtnet: 1 46389 0.90 MB/sec execute 209 sec latency 52.424 ms oleg210-client.virtnet: 1 46536 0.90 MB/sec execute 210 sec latency 46.094 ms oleg210-client.virtnet: 1 46671 0.90 MB/sec execute 211 sec latency 291.266 ms oleg210-client.virtnet: 1 46671 0.89 MB/sec execute 212 sec latency 1291.757 ms oleg210-client.virtnet: 1 46671 0.89 MB/sec execute 213 sec latency 2291.953 ms oleg210-client.virtnet: 1 46671 0.88 MB/sec execute 214 sec latency 3292.128 ms oleg210-client.virtnet: 1 46671 0.88 MB/sec execute 215 sec latency 4292.303 ms oleg210-client.virtnet: 1 46671 0.88 MB/sec execute 216 sec latency 5292.504 ms oleg210-client.virtnet: 1 46671 0.87 MB/sec execute 217 sec latency 6293.664 ms oleg210-client.virtnet: 1 46671 0.87 MB/sec execute 218 sec latency 7294.289 ms oleg210-client.virtnet: 1 46671 0.86 MB/sec execute 219 sec latency 8295.312 ms oleg210-client.virtnet: 1 46671 0.86 MB/sec execute 220 sec latency 9295.486 ms oleg210-client.virtnet: 1 46671 0.86 MB/sec execute 221 sec latency 10296.880 ms oleg210-client.virtnet: 1 46671 0.85 MB/sec execute 222 sec latency 11296.978 ms oleg210-client.virtnet: 1 46671 0.85 MB/sec execute 223 sec latency 12297.151 ms oleg210-client.virtnet: 1 46671 0.84 MB/sec execute 224 sec latency 13298.103 ms oleg210-client.virtnet: 1 46671 0.84 MB/sec execute 225 sec latency 14298.689 ms oleg210-client.virtnet: 1 46671 0.84 MB/sec execute 226 sec latency 15299.963 ms oleg210-client.virtnet: 1 46671 0.83 MB/sec execute 227 sec latency 16300.511 ms oleg210-client.virtnet: 1 46671 0.83 MB/sec execute 228 sec latency 17301.904 ms oleg210-client.virtnet: 1 46671 0.83 MB/sec execute 229 sec latency 18302.085 ms oleg210-client.virtnet: 1 46834 0.82 MB/sec execute 230 sec latency 18436.250 ms oleg210-client.virtnet: 1 46940 0.82 MB/sec execute 231 sec latency 87.461 ms oleg210-client.virtnet: 1 47109 0.82 MB/sec execute 232 sec latency 60.477 ms oleg210-client.virtnet: 1 47310 0.82 MB/sec execute 233 sec latency 51.070 ms oleg210-client.virtnet: 1 47564 0.83 MB/sec execute 234 sec latency 51.595 ms oleg210-client.virtnet: 1 47663 0.83 MB/sec execute 235 sec latency 73.683 ms oleg210-client.virtnet: 1 47799 0.82 MB/sec execute 236 sec latency 133.836 ms oleg210-client.virtnet: 1 47974 0.82 MB/sec execute 237 sec latency 49.545 ms oleg210-client.virtnet: 1 48192 0.82 MB/sec execute 238 sec latency 31.986 ms oleg210-client.virtnet: 1 48493 0.82 MB/sec execute 239 sec latency 24.445 ms oleg210-client.virtnet: 1 48621 0.82 MB/sec execute 240 sec latency 63.368 ms oleg210-client.virtnet: 1 48701 0.82 MB/sec execute 241 sec latency 732.6513:20:47 (1709058047) shut down Failover mds1 to oleg210-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 13:21:03 (1709058063) targets are mounted 13:21:03 (1709058063) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 4 ms oleg210-client.virtnet: 1 48701 0.81 MB/sec execute 242 sec latency 1733.483 ms oleg210-client.virtnet: 1 48701 0.81 MB/sec execute 243 sec latency 2733.698 ms oleg210-client.virtnet: 1 48701 0.81 MB/sec execute 244 sec latency 3733.981 ms oleg210-client.virtnet: 1 48701 0.80 MB/sec execute 245 sec latency 4735.487 ms oleg210-client.virtnet: 1 48701 0.80 MB/sec execute 246 sec latency 5735.706 ms oleg210-client.virtnet: 1 48701 0.80 MB/sec execute 247 sec latency 6735.904 ms oleg210-client.virtnet: 1 48701 0.79 MB/sec execute 248 sec latency 7736.423 ms oleg210-client.virtnet: 1 48701 0.79 MB/sec execute 249 sec latency 8736.608 ms oleg210-client.virtnet: 1 48701 0.79 MB/sec execute 250 sec latency 9737.073 ms oleg210-client.virtnet: 1 48701 0.79 MB/sec execute 251 sec latency 10737.938 ms oleg210-client.virtnet: 1 48701 0.78 MB/sec execute 252 sec latency 11738.188 ms oleg210-client.virtnet: 1 48701 0.78 MB/sec execute 253 sec latency 12739.153 ms oleg210-client.virtnet: 1 48701 0.78 MB/sec execute 254 sec latency 13739.345 ms oleg210-client.virtnet: 1 48701 0.77 MB/sec execute 255 sec latency 14739.738 ms oleg210-client.virtnet: 1 48701 0.77 MB/sec execute 256 sec latency 15739.926 ms oleg210-client.virtnet: 1 48701 0.77 MB/sec execute 257 sec latency 16740.061 ms oleg210-client.virtnet: 1 48701 0.76 MB/sec execute 258 sec latency 17740.274 ms oleg210-client.virtnet: 1 48701 0.76 MB/sec execute 259 sec latency 18740.493 ms oleg210-client.virtnet: 1 48701 0.76 MB/sec execute 260 sec latency 19740.716 ms oleg210-client.virtnet: 1 48779 0.76 MB/sec execute 261 sec latency 20257.419 ms oleg210-client.virtnet: 1 49155 0.77 MB/sec execute 262 sec latency 53.371 ms oleg210-client.virtnet: 1 49644 0.79 MB/sec execute 263 sec latency 35.210 ms oleg210-client.virtnet: 1 49891 0.79 MB/sec execute 264 sec latency 36.538 ms oleg210-client.virtnet: 1 50070 0.79 MB/sec execute 265 sec latency 46.951 ms oleg210-client.virtnet: 1 50246 0.78 MB/sec execute 266 sec latency 29.070 ms oleg210-client.virtnet: 1 50503 0.78 MB/sec execute 267 sec latency 27.941 ms oleg210-client.virtnet: 1 50760 0.78 MB/sec execute 268 sec latency 25.808 ms oleg210-client.virtnet: 1 51098 0.79 MB/sec execute 269 sec latency 24.208 ms oleg210-client.virtnet: 1 51292 0.79 MB/sec execute 270 sec latency 28.115 ms oleg210-client.virtnet: 1 51528 0.79 MB/sec execute 271 sec latency 54.334 ms oleg210-client.virtnet: 1 51864 0.79 MB/sec execute 272 sec latency 16.697 ms oleg210-client.virtnet: 1 52141 0.79 MB/sec execute 273 sec latency 31.457 ms oleg210-client.virtnet: 1 52585 0.80 MB/sec execute 274 sec latency 27.444 ms oleg210-client.virtnet: 1 53028 0.81 MB/sec execute 275 sec latency 30.572 ms oleg210-client.virtnet: 1 53356 0.82 MB/sec execute 276 sec latency 23.727 ms oleg210-client.virtnet: 1 53605 0.82 MB/sec execute 277 sec latency 20.894 ms oleg210-client.virtnet: 1 53845 0.82 MB/sec execute 278 sec latency 29.151 ms oleg210-client.virtnet: 1 54117 0.82 MB/sec execute 279 sec latency 23.015 ms oleg210-client.virtnet: 1 54346 0.82 MB/sec execute 280 sec latency 31.760 ms oleg210-client.virtnet: 1 54669 0.83 MB/sec execute 281 sec latency 25.211 ms oleg210-client.virtnet: 1 54928 0.82 MB/sec execute 282 sec latency 22.496 ms oleg210-client.virtnet: 1 55215 0.82 MB/sec execute 283 sec latency 20.758 ms oleg210-client.virtnet: 1 55583 0.82 MB/sec execute 284 sec latency 17.929 ms oleg210-client.virtnet: 1 55834 0.83 MB/sec execute 285 sec latency 25.548 ms oleg210-client.virtnet: 1 56432 0.84 MB/sec execute 286 sec latency 23.998 ms oleg210-client.virtnet: 1 56808 0.85 MB/sec execute 287 sec latency 26.935 ms oleg210-client.virtnet: 1 57054 0.85 MB/sec execute 288 sec latency 32.126 ms oleg210-client.virtnet: 1 57242 0.85 MB/sec execute 289 sec latency 25.616 ms oleg210-client.virtnet: 1 57459 0.85 MB/sec execute 290 sec latency 32.563 ms oleg210-client.virtnet: 1 57651 0.85 MB/sec execute 291 sec latency 52.205 ms oleg210-client.virtnet: 1 57785 0.85 MB/sec execute 292 sec latency 34.077 ms oleg210-client.virtnet: 1 58013 0.85 MB/sec execute 293 sec latency 30.769 ms oleg210-client.virtnet: 1 58350 0.86 MB/sec execute 294 sec latency 22.756 ms oleg210-client.virtnet: 1 58571 0.85 MB/sec execute 295 sec latency 30.559 ms oleg210-client.virtnet: 1 58863 0.85 MB/sec execute 296 sec latency 21.024 ms oleg210-client.virtnet: 1 59167 0.85 MB/sec execute 297 sec latency 28.627 ms oleg210-client.virtnet: 1 59532 0.86 MB/sec execute 298 sec latency 24.908 ms oleg210-client.virtnet: 1 59956 0.87 MB/sec execute 299 sec latency 100.535 ms oleg210-client.virtnet: 1 cleanup 300 sec oleg210-client.virtnet: 0 cleanup 301 sec oleg210-client.virtnet: oleg210-client.virtnet: Operation Count AvgLat MaxLat oleg210-client.virtnet: ---------------------------------------- oleg210-client.virtnet: NTCreateX 8332 19.545 20341.023 oleg210-client.virtnet: Close 6147 5.811 20257.373 oleg210-client.virtnet: Rename 355 26.250 3986.905 oleg210-client.virtnet: Unlink 1654 3.978 24.204 oleg210-client.virtnet: Qpathinfo 7592 5.155 18436.224 oleg210-client.virtnet: Qfileinfo 1341 0.224 6.892 oleg210-client.virtnet: Qfsinfo 1381 0.377 14.964 oleg210-client.virtnet: Sfileinfo 680 38.396 20143.397 oleg210-client.virtnet: Find 2948 0.705 26.998 oleg210-client.virtnet: WriteX 4140 1.763 60.465 oleg210-client.virtnet: ReadX 13306 0.027 8.851 oleg210-client.virtnet: LockX 28 2.556 8.053 oleg210-client.virtnet: UnlockX 28 2.343 10.778 oleg210-client.virtnet: Flush 578 15.692 464.325 oleg210-client.virtnet: oleg210-client.virtnet: Throughput 0.868275 MB/sec 1 clients 1 procs max_latency=20341.049 ms oleg210-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg210-client.virtnet at Tue Feb 27 13:21:47 EST 2024 with return code 0 oleg210-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg210-client.virtnet oleg210-client.virtnet: /mnt/lustre/d70b.replay-single/oleg210-client.virtnet /mnt/lustre/d70b.replay-single/oleg210-client.virtnet oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg210-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg210-client.virtnet: removed directory: 'clients/client0' oleg210-client.virtnet: removed directory: 'clients' oleg210-client.virtnet: removed 'client.txt' oleg210-client.virtnet: /mnt/lustre/d70b.replay-single/oleg210-client.virtnet oleg210-client.virtnet: dbench successfully finished