== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 06:11:31 (1734433891) Starting client oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre Started clients oleg254-client.virtnet: 192.168.202.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg254-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13797 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1285228 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3593664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26148 3580872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7174536 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:11:40 (1734433900) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:11:53 (1734433913) targets are mounted 06:11:53 (1734433913) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2420 1285268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1736 3594108 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26440 3568812 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28176 7162920 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:12:05 (1734433925) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server oleg254-client.virtnet: looking for dbench program oleg254-client.virtnet: /usr/bin/dbench oleg254-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg254-client.virtnet' oleg254-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg254-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg254-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg254-client.virtnet at Tue Dec 17 06:11:32 EST 2024 oleg254-client.virtnet: waiting for dbench pid 13821 oleg254-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg254-client.virtnet: oleg254-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg254-client.virtnet: failed to create barrier semaphore oleg254-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg254-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg254-client.virtnet: releasing clients oleg254-client.virtnet: 1 358 12.78 MB/sec warmup 1 sec latency 19.703 ms oleg254-client.virtnet: 1 765 10.68 MB/sec warmup 2 sec latency 21.238 ms oleg254-client.virtnet: 1 1086 7.64 MB/sec warmup 3 sec latency 18.103 ms oleg254-client.virtnet: 1 1587 6.54 MB/sec warmup 4 sec latency 16.147 ms oleg254-client.virtnet: 1 1995 5.30 MB/sec warmup 5 sec latency 17.696 ms oleg254-client.virtnet: 1 2454 4.69 MB/sec warmup 6 sec latency 15.867 ms oleg254-client.virtnet: 1 3210 4.90 MB/sec warmup 7 sec latency 18.391 ms oleg254-client.virtnet: 1 3814 4.97 MB/sec warmup 8 sec latency 17.519 ms oleg254-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 115.566 ms oleg254-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 1115.810 ms oleg254-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 2115.994 ms oleg254-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 3116.190 ms oleg254-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 4116.726 ms oleg254-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 5116.910 ms oleg254-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 6117.122 ms oleg254-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 7117.335 ms oleg254-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 8117.465 ms oleg254-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 9117.592 ms oleg254-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 10117.723 ms oleg254-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 11117.859 ms oleg254-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 12118.034 ms oleg254-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 13118.151 ms oleg254-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 14118.292 ms oleg254-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 15118.441 ms oleg254-client.virtnet: 1 4302 1.62 MB/sec warmup 25 sec latency 15689.450 ms oleg254-client.virtnet: 1 4787 1.62 MB/sec warmup 26 sec latency 11.221 ms oleg254-client.virtnet: 1 5278 1.68 MB/sec warmup 27 sec latency 16.894 ms oleg254-client.virtnet: 1 5907 1.68 MB/sec warmup 28 sec latency 12.772 ms oleg254-client.virtnet: 1 6741 1.84 MB/sec warmup 29 sec latency 11.845 ms oleg254-client.virtnet: 1 7433 1.96 MB/sec warmup 30 sec latency 10.537 ms oleg254-client.virtnet: 1 8014 1.93 MB/sec warmup 31 sec latency 11.236 ms oleg254-client.virtnet: 1 8722 2.01 MB/sec warmup 32 sec latency 50.273 ms oleg254-client.virtnet: 1 8722 1.95 MB/sec warmup 33 sec latency 1050.452 mFailover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:12:19 (1734433939) targets are mounted 06:12:19 (1734433939) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12164 3593688 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37428 3566696 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49592 7160384 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:12:31 (1734433951) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:12:45 (1734433965) targets are mounted 06:12:45 (1734433965) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec s oleg254-client.virtnet: 1 8722 1.89 MB/sec warmup 34 sec latency 2050.637 ms oleg254-client.virtnet: 1 8722 1.84 MB/sec warmup 35 sec latency 3050.784 ms oleg254-client.virtnet: 1 8722 1.79 MB/sec warmup 36 sec latency 4050.951 ms oleg254-client.virtnet: 1 8722 1.74 MB/sec warmup 37 sec latency 5051.105 ms oleg254-client.virtnet: 1 8722 1.69 MB/sec warmup 38 sec latency 6051.264 ms oleg254-client.virtnet: 1 8722 1.65 MB/sec warmup 39 sec latency 7051.419 ms oleg254-client.virtnet: 1 8722 1.61 MB/sec warmup 40 sec latency 8051.575 ms oleg254-client.virtnet: 1 8722 1.57 MB/sec warmup 41 sec latency 9051.740 ms oleg254-client.virtnet: 1 8722 1.53 MB/sec warmup 42 sec latency 10051.906 ms oleg254-client.virtnet: 1 8722 1.50 MB/sec warmup 43 sec latency 11052.089 ms oleg254-client.virtnet: 1 8722 1.46 MB/sec warmup 44 sec latency 12052.219 ms oleg254-client.virtnet: 1 8722 1.43 MB/sec warmup 45 sec latency 13052.358 ms oleg254-client.virtnet: 1 8722 1.40 MB/sec warmup 46 sec latency 14052.533 ms oleg254-client.virtnet: 1 8722 1.37 MB/sec warmup 47 sec latency 15053.162 ms oleg254-client.virtnet: 1 8722 1.34 MB/sec warmup 48 sec latency 16053.315 ms oleg254-client.virtnet: 1 8722 1.31 MB/sec warmup 49 sec latency 17053.472 ms oleg254-client.virtnet: 1 8722 1.29 MB/sec warmup 50 sec latency 18053.643 ms oleg254-client.virtnet: 1 9018 1.27 MB/sec warmup 51 sec latency 18424.197 ms oleg254-client.virtnet: 1 9493 1.27 MB/sec warmup 52 sec latency 15.717 ms oleg254-client.virtnet: 1 10111 1.35 MB/sec warmup 53 sec latency 16.194 ms oleg254-client.virtnet: 1 10858 1.44 MB/sec warmup 54 sec latency 15.422 ms oleg254-client.virtnet: 1 11202 1.42 MB/sec warmup 55 sec latency 19.342 ms oleg254-client.virtnet: 1 11759 1.43 MB/sec warmup 56 sec latency 11.906 ms oleg254-client.virtnet: 1 12372 1.47 MB/sec warmup 57 sec latency 12.671 ms oleg254-client.virtnet: 1 13034 1.47 MB/sec warmup 58 sec latency 9.653 ms oleg254-client.virtnet: 1 14014 1.57 MB/sec warmup 59 sec latency 10.816 ms oleg254-client.virtnet: 1 15280 1.98 MB/sec execute 1 sec latency 9.147 ms oleg254-client.virtnet: 1 15932 2.67 MB/sec execute 2 sec latency 9.074 ms oleg254-client.virtnet: 1 16602 2.36 MB/sec execute 3 sec latency 8.883 ms oleg254-client.virtnet: 1 17722 4.36 MB/sec execute 4 sec latency 8.671 ms oleg254-client.virtnet: 1 18234 3.82 MB/sec execute 5 sec latency 10.944 ms oleg254-client.virtnet: 1 18776 3.48 MB/sec execute 6 sec latency 11.430 ms oleg254-client.virtnet: 1 19384 3.48 MB/sec execute 7 sec latency 11.803 ms oleg254-client.virtnet: 1 20069 3.19 MB/sec execute 8 sec latency 9.600 ms oleg254-client.virtnet: 1 21139 3.74 MB/sec execute 9 sec latency 10.194 ms oleg254-client.virtnet: 1 21797 3.83 MB/sec execute 10 sec latency 8.201 ms oleg254-client.virtnet: 1 22329 3.57 MB/sec execute 11 sec latency 18.127 ms oleg254-client.virtnet: 1 22890 3.62 MB/sec execute 12 sec latency 12.904 ms oleg254-client.virtnet: 1 23455 3.39 MB/sec execute 13 sec latency 10.192 ms oleg254-client.virtnet: 1 24249 3.63 MB/sec execute 14 sec latency 10.561 ms oleg254-client.virtnet: 1 25131 3.81 MB/sec execute 15 sec latency 10.410 ms oleg254-client.virtnet: 1 25621 3.61 MB/sec execute 16 sec latency 15.642 ms oleg254-client.virtnet: 1 26105 3.50 MB/sec execute 17 sec latency 15.212 ms oleg254-client.virtnet: 1 26612 3.49 MB/sec execute 18 sec latency 13.943 ms oleg254-client.virtnet: 1 27226 3.39 MB/sec execute 19 sec lateUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11920 3594156 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37580 3568760 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162916 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:12:57 (1734433977) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:13:11 (1734433991) targets are mounted 06:13:11 (1734433991) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11920 3594056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37584 3568660 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49504 7162716 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:13:23 (1734434003) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 ncy 11.420 ms oleg254-client.virtnet: 1 28080 3.54 MB/sec execute 20 sec latency 11.838 ms oleg254-client.virtnet: 1 28679 3.63 MB/sec execute 21 sec latency 15.269 ms oleg254-client.virtnet: 1 28984 3.48 MB/sec execute 22 sec latency 14.910 ms oleg254-client.virtnet: 1 29396 3.37 MB/sec execute 23 sec latency 16.870 ms oleg254-client.virtnet: 1 29700 3.28 MB/sec execute 24 sec latency 437.938 ms oleg254-client.virtnet: 1 29700 3.15 MB/sec execute 25 sec latency 1438.087 ms oleg254-client.virtnet: 1 29700 3.03 MB/sec execute 26 sec latency 2438.309 ms oleg254-client.virtnet: 1 29700 2.92 MB/sec execute 27 sec latency 3438.423 ms oleg254-client.virtnet: 1 29700 2.81 MB/sec execute 28 sec latency 4438.565 ms oleg254-client.virtnet: 1 29700 2.71 MB/sec execute 29 sec latency 5438.735 ms oleg254-client.virtnet: 1 29700 2.62 MB/sec execute 30 sec latency 6438.878 ms oleg254-client.virtnet: 1 29700 2.54 MB/sec execute 31 sec latency 7439.019 ms oleg254-client.virtnet: 1 29700 2.46 MB/sec execute 32 sec latency 8439.174 ms oleg254-client.virtnet: 1 29700 2.39 MB/sec execute 33 sec latency 9439.322 ms oleg254-client.virtnet: 1 29700 2.32 MB/sec execute 34 sec latency 10439.859 ms oleg254-client.virtnet: 1 29700 2.25 MB/sec execute 35 sec latency 11440.038 ms oleg254-client.virtnet: 1 29700 2.19 MB/sec execute 36 sec latency 12440.292 ms oleg254-client.virtnet: 1 29700 2.13 MB/sec execute 37 sec latency 13440.513 ms oleg254-client.virtnet: 1 29700 2.07 MB/sec execute 38 sec latency 14440.685 ms oleg254-client.virtnet: 1 29700 2.02 MB/sec execute 39 sec latency 15440.803 ms oleg254-client.virtnet: 1 29700 1.97 MB/sec execute 40 sec latency 16440.960 ms oleg254-client.virtnet: 1 29700 1.92 MB/sec execute 41 sec latency 17441.150 ms oleg254-client.virtnet: 1 29700 1.87 MB/sec execute 42 sec latency 18441.313 ms oleg254-client.virtnet: 1 29936 1.90 MB/sec execute 43 sec latency 19029.364 ms oleg254-client.virtnet: 1 30364 1.86 MB/sec execute 44 sec latency 17.281 ms oleg254-client.virtnet: 1 30915 1.87 MB/sec execute 45 sec latency 12.019 ms oleg254-client.virtnet: 1 31779 1.98 MB/sec execute 46 sec latency 14.606 ms oleg254-client.virtnet: 1 32290 2.03 MB/sec execute 47 sec latency 10.826 ms oleg254-client.virtnet: 1 32686 2.00 MB/sec execute 48 sec latency 15.482 ms oleg254-client.virtnet: 1 33128 1.99 MB/sec execute 49 sec latency 12.903 ms oleg254-client.virtnet: 1 33595 2.02 MB/sec execute 50 sec latency 14.005 ms oleg254-client.virtnet: 1 33974 1.98 MB/sec execute 51 sec latency 17.796 ms oleg254-client.virtnet: 1 34406 1.98 MB/sec execute 52 sec latency 15.149 ms oleg254-client.virtnet: 1 35347 2.08 MB/sec execute 53 sec latency 13.268 ms oleg254-client.virtnet: 1 35933 2.13 MB/sec execute 54 sec latency 9.910 ms oleg254-client.virtnet: 1 36392 2.10 MB/sec execute 55 sec latency 12.519 ms oleg254-client.virtnet: 1 36962 2.14 MB/sec execute 56 sec latency 11.906 ms oleg254-client.virtnet: 1 37464 2.11 MB/sec execute 57 sec latency 12.187 ms oleg254-client.virtnet: 1 38081 2.12 MB/sec execute 58 sec latency 10.074 ms oleg254-client.virtnet: 1 39173 2.25 MB/sec execute 59 sec latency 8.898 ms oleg254-client.virtnet: 1 39670 2.23 MB/sec execute 60 sec latency 9.772 ms oleg254-client.virtnet: 1 40206 2.23 MB/sec execute 61 sec latency 11.574 ms oleg254-client.virtnet: 1 40777 2.24 MB/sec execute 62 sec latency 15.662 ms oleg254-client.virtnet: 1 41401 2.23 MB/sec execute 63 sec latency 9.504 ms oleg254-client.virtnet: 1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:13:38 (1734434018) targets are mounted 06:13:38 (1734434018) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12080 3593004 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3568816 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7161820 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:13:49 (1734434029) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:14:04 (1734434044) targets are mounted 06:14:04 (1734434044) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12172 3591964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37584 3568820 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49756 7160784 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:14:16 (1734434056) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server 42042 2.29 MB/sec execute 64 sec latency 19.143 ms oleg254-client.virtnet: 1 42739 2.33 MB/sec execute 65 sec latency 14.807 ms oleg254-client.virtnet: 1 43063 2.32 MB/sec execute 66 sec latency 14.754 ms oleg254-client.virtnet: 1 43471 2.29 MB/sec execute 67 sec latency 18.160 ms oleg254-client.virtnet: 1 43885 2.28 MB/sec execute 68 sec latency 14.112 ms oleg254-client.virtnet: 1 44365 2.29 MB/sec execute 69 sec latency 13.849 ms oleg254-client.virtnet: 1 44758 2.27 MB/sec execute 70 sec latency 14.574 ms oleg254-client.virtnet: 1 45364 2.31 MB/sec execute 71 sec latency 12.916 ms oleg254-client.virtnet: 1 46175 2.37 MB/sec execute 72 sec latency 12.851 ms oleg254-client.virtnet: 1 46570 2.36 MB/sec execute 73 sec latency 13.995 ms oleg254-client.virtnet: 1 46880 2.33 MB/sec execute 74 sec latency 290.927 ms oleg254-client.virtnet: 1 47283 2.32 MB/sec execute 75 sec latency 401.669 ms oleg254-client.virtnet: 1 47804 2.33 MB/sec execute 76 sec latency 12.186 ms oleg254-client.virtnet: 1 47946 2.31 MB/sec execute 77 sec latency 707.076 ms oleg254-client.virtnet: 1 47946 2.28 MB/sec execute 78 sec latency 1707.212 ms oleg254-client.virtnet: 1 47946 2.25 MB/sec execute 79 sec latency 2707.408 ms oleg254-client.virtnet: 1 47946 2.22 MB/sec execute 80 sec latency 3707.599 ms oleg254-client.virtnet: 1 47946 2.19 MB/sec execute 81 sec latency 4707.764 ms oleg254-client.virtnet: 1 47946 2.16 MB/sec execute 82 sec latency 5708.009 ms oleg254-client.virtnet: 1 47946 2.14 MB/sec execute 83 sec latency 6708.174 ms oleg254-client.virtnet: 1 47946 2.11 MB/sec execute 84 sec latency 7708.417 ms oleg254-client.virtnet: 1 47946 2.09 MB/sec execute 85 sec latency 8708.599 ms oleg254-client.virtnet: 1 47946 2.06 MB/sec execute 86 sec latency 9708.780 ms oleg254-client.virtnet: 1 47946 2.04 MB/sec execute 87 sec latency 10708.997 ms oleg254-client.virtnet: 1 47946 2.02 MB/sec execute 88 sec latency 11709.110 ms oleg254-client.virtnet: 1 47946 1.99 MB/sec execute 89 sec latency 12709.280 ms oleg254-client.virtnet: 1 47946 1.97 MB/sec execute 90 sec latency 13709.418 ms oleg254-client.virtnet: 1 47946 1.95 MB/sec execute 91 sec latency 14709.489 ms oleg254-client.virtnet: 1 47946 1.93 MB/sec execute 92 sec latency 15709.644 ms oleg254-client.virtnet: 1 47946 1.91 MB/sec execute 93 sec latency 16709.803 ms oleg254-client.virtnet: 1 47946 1.89 MB/sec execute 94 sec latency 17709.997 ms oleg254-client.virtnet: 1 48010 1.87 MB/sec execute 95 sec latency 18552.058 ms oleg254-client.virtnet: 1 48597 1.87 MB/sec execute 96 sec latency 11.444 ms oleg254-client.virtnet: 1 49634 1.95 MB/sec execute 97 sec latency 14.417 ms oleg254-client.virtnet: 1 50079 1.95 MB/sec execute 98 sec latency 11.681 ms oleg254-client.virtnet: 1 50440 1.94 MB/sec execute 99 sec latency 25.802 ms oleg254-client.virtnet: 1 50846 1.93 MB/sec execute 100 sec latency 14.223 ms oleg254-client.virtnet: 1 51385 1.95 MB/sec execute 101 sec latency 12.646 ms oleg254-client.virtnet: 1 51860 1.93 MB/sec execute 102 sec latency 11.643 ms oleg254-client.virtnet: 1 52586 1.97 MB/sec execute 103 sec latency 14.949 ms oleg254-client.virtnet: 1 53407 2.01 MB/sec execute 104 sec latency 11.977 ms oleg254-client.virtnet: 1 53954 2.01 MB/sec execute 105 sec latency 12.027 ms oleg254-client.virtnet: 1 54621 2.03 MB/sec execute 106 sec latency 8.796 ms oleg254-client.virtnet: 1 55147 2.02 MB/sec execute 107 sec latency 9.276 ms oleg254-client.virtnet: 1 55814 2.02 MB/sec execute 108 sec latencFailover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:14:30 (1734434070) targets are mounted 06:14:30 (1734434070) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12084 3591884 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37552 3568880 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160764 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:14:42 (1734434082) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:14:56 (1734434096) targets are mounted 06:14:56 (1734434096) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12220 3591364 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37604 3568856 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49824 7160220 1% /mnt/lustre y 8.528 ms oleg254-client.virtnet: 1 56904 2.09 MB/sec execute 109 sec latency 10.731 ms oleg254-client.virtnet: 1 57334 2.09 MB/sec execute 110 sec latency 11.933 ms oleg254-client.virtnet: 1 57827 2.08 MB/sec execute 111 sec latency 14.941 ms oleg254-client.virtnet: 1 58391 2.10 MB/sec execute 112 sec latency 12.361 ms oleg254-client.virtnet: 1 58796 2.08 MB/sec execute 113 sec latency 19.894 ms oleg254-client.virtnet: 1 59333 2.08 MB/sec execute 114 sec latency 14.148 ms oleg254-client.virtnet: 1 60168 2.12 MB/sec execute 115 sec latency 14.803 ms oleg254-client.virtnet: 1 60696 2.14 MB/sec execute 116 sec latency 14.066 ms oleg254-client.virtnet: 1 61138 2.13 MB/sec execute 117 sec latency 11.690 ms oleg254-client.virtnet: 1 61564 2.12 MB/sec execute 118 sec latency 17.856 ms oleg254-client.virtnet: 1 62060 2.13 MB/sec execute 119 sec latency 12.502 ms oleg254-client.virtnet: 1 62440 2.12 MB/sec execute 120 sec latency 16.609 ms oleg254-client.virtnet: 1 62906 2.12 MB/sec execute 121 sec latency 16.328 ms oleg254-client.virtnet: 1 63756 2.16 MB/sec execute 122 sec latency 14.331 ms oleg254-client.virtnet: 1 64149 2.17 MB/sec execute 123 sec latency 13.681 ms oleg254-client.virtnet: 1 64522 2.16 MB/sec execute 124 sec latency 13.473 ms oleg254-client.virtnet: 1 64980 2.16 MB/sec execute 125 sec latency 13.499 ms oleg254-client.virtnet: 1 65505 2.17 MB/sec execute 126 sec latency 14.082 ms oleg254-client.virtnet: 1 65721 2.15 MB/sec execute 127 sec latency 413.008 ms oleg254-client.virtnet: 1 66284 2.15 MB/sec execute 128 sec latency 14.563 ms oleg254-client.virtnet: 1 66837 2.17 MB/sec execute 129 sec latency 421.754 ms oleg254-client.virtnet: 1 66837 2.16 MB/sec execute 130 sec latency 1421.911 ms oleg254-client.virtnet: 1 66837 2.14 MB/sec execute 131 sec latency 2422.091 ms oleg254-client.virtnet: 1 66837 2.12 MB/sec execute 132 sec latency 3422.275 ms oleg254-client.virtnet: 1 66837 2.11 MB/sec execute 133 sec latency 4422.464 ms oleg254-client.virtnet: 1 66837 2.09 MB/sec execute 134 sec latency 5422.613 ms oleg254-client.virtnet: 1 66837 2.08 MB/sec execute 135 sec latency 6422.748 ms oleg254-client.virtnet: 1 66837 2.06 MB/sec execute 136 sec latency 7422.891 ms oleg254-client.virtnet: 1 66837 2.05 MB/sec execute 137 sec latency 8423.051 ms oleg254-client.virtnet: 1 66837 2.03 MB/sec execute 138 sec latency 9423.211 ms oleg254-client.virtnet: 1 66837 2.02 MB/sec execute 139 sec latency 10423.394 ms oleg254-client.virtnet: 1 66837 2.00 MB/sec execute 140 sec latency 11423.580 ms oleg254-client.virtnet: 1 66837 1.99 MB/sec execute 141 sec latency 12423.733 ms oleg254-client.virtnet: 1 66837 1.97 MB/sec execute 142 sec latency 13423.871 ms oleg254-client.virtnet: 1 66837 1.96 MB/sec execute 143 sec latency 14423.993 ms oleg254-client.virtnet: 1 66837 1.95 MB/sec execute 144 sec latency 15424.096 ms oleg254-client.virtnet: 1 66837 1.93 MB/sec execute 145 sec latency 16424.240 ms oleg254-client.virtnet: 1 66837 1.92 MB/sec execute 146 sec latency 17424.470 ms oleg254-client.virtnet: 1 66842 1.91 MB/sec execute 147 sec latency 18405.709 ms oleg254-client.virtnet: 1 67435 1.93 MB/sec execute 148 sec latency 16.232 ms oleg254-client.virtnet: 1 67863 1.93 MB/sec execute 149 sec latency 10.300 ms oleg254-client.virtnet: 1 68305 1.92 MB/sec execute 150 sec latency 15.208 ms oleg254-client.virtnet: 1 68886 1.93 MB/sec execute 151 sec latency 11.154 ms oleg254-client.virtnet: 1 69412 1.92 MB/sec execute 152 sec latency 11.415 ms oleg254-client.virtnet: 1 6test_70b fail mds1 9 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:15:08 (1734434108) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:15:22 (1734434122) targets are mounted 06:15:22 (1734434122) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12476 3593528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3569016 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49892 7162544 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:15:34 (1734434134) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:15:48 (1734434148) targets are mounted 06:15:48 (1734434148) facet_failover done 9905 1.93 MB/sec execute 153 sec latency 13.052 ms oleg254-client.virtnet: 1 70785 1.96 MB/sec execute 154 sec latency 13.427 ms oleg254-client.virtnet: 1 71291 1.97 MB/sec execute 155 sec latency 19.521 ms oleg254-client.virtnet: 1 71752 1.96 MB/sec execute 156 sec latency 11.206 ms oleg254-client.virtnet: 1 72243 1.96 MB/sec execute 157 sec latency 10.348 ms oleg254-client.virtnet: 1 72802 1.97 MB/sec execute 158 sec latency 11.970 ms oleg254-client.virtnet: 1 73325 1.97 MB/sec execute 159 sec latency 12.241 ms oleg254-client.virtnet: 1 73989 1.99 MB/sec execute 160 sec latency 14.068 ms oleg254-client.virtnet: 1 74713 2.01 MB/sec execute 161 sec latency 11.763 ms oleg254-client.virtnet: 1 75054 2.01 MB/sec execute 162 sec latency 13.419 ms oleg254-client.virtnet: 1 75416 2.00 MB/sec execute 163 sec latency 20.421 ms oleg254-client.virtnet: 1 75775 2.00 MB/sec execute 164 sec latency 14.273 ms oleg254-client.virtnet: 1 76318 2.00 MB/sec execute 165 sec latency 14.105 ms oleg254-client.virtnet: 1 76872 2.00 MB/sec execute 166 sec latency 11.113 ms oleg254-client.virtnet: 1 77554 2.02 MB/sec execute 167 sec latency 11.662 ms oleg254-client.virtnet: 1 78331 2.05 MB/sec execute 168 sec latency 11.567 ms oleg254-client.virtnet: 1 78787 2.04 MB/sec execute 169 sec latency 13.102 ms oleg254-client.virtnet: 1 79194 2.04 MB/sec execute 170 sec latency 14.544 ms oleg254-client.virtnet: 1 79711 2.04 MB/sec execute 171 sec latency 13.069 ms oleg254-client.virtnet: 1 80280 2.04 MB/sec execute 172 sec latency 12.119 ms oleg254-client.virtnet: 1 81290 2.07 MB/sec execute 173 sec latency 9.204 ms oleg254-client.virtnet: 1 81981 2.09 MB/sec execute 174 sec latency 11.546 ms oleg254-client.virtnet: 1 82467 2.08 MB/sec execute 175 sec latency 15.144 ms oleg254-client.virtnet: 1 83223 2.09 MB/sec execute 176 sec latency 7.742 ms oleg254-client.virtnet: 1 83969 2.09 MB/sec execute 177 sec latency 8.276 ms oleg254-client.virtnet: 1 84632 2.11 MB/sec execute 178 sec latency 290.925 ms oleg254-client.virtnet: 1 85469 2.14 MB/sec execute 179 sec latency 421.735 ms oleg254-client.virtnet: 1 85900 2.13 MB/sec execute 180 sec latency 13.520 ms oleg254-client.virtnet: 1 86056 2.12 MB/sec execute 181 sec latency 667.476 ms oleg254-client.virtnet: 1 86056 2.11 MB/sec execute 182 sec latency 1667.658 ms oleg254-client.virtnet: 1 86056 2.09 MB/sec execute 183 sec latency 2667.869 ms oleg254-client.virtnet: 1 86056 2.08 MB/sec execute 184 sec latency 3668.056 ms oleg254-client.virtnet: 1 86056 2.07 MB/sec execute 185 sec latency 4668.252 ms oleg254-client.virtnet: 1 86056 2.06 MB/sec execute 186 sec latency 5668.444 ms oleg254-client.virtnet: 1 86056 2.05 MB/sec execute 187 sec latency 6668.809 ms oleg254-client.virtnet: 1 86056 2.04 MB/sec execute 188 sec latency 7669.026 ms oleg254-client.virtnet: 1 86056 2.03 MB/sec execute 189 sec latency 8669.219 ms oleg254-client.virtnet: 1 86056 2.02 MB/sec execute 190 sec latency 9669.396 ms oleg254-client.virtnet: 1 86056 2.01 MB/sec execute 191 sec latency 10669.546 ms oleg254-client.virtnet: 1 86056 2.00 MB/sec execute 192 sec latency 11670.484 ms oleg254-client.virtnet: 1 86056 1.99 MB/sec execute 193 sec latency 12671.646 ms oleg254-client.virtnet: 1 86056 1.98 MB/sec execute 194 sec latency 13672.695 ms oleg254-client.virtnet: 1 86056 1.96 MB/sec execute 195 sec latency 14674.495 ms oleg254-client.virtnet: 1 86056 1.95 MB/sec execute 196 sec latency 15674.698 ms oleg254-client.virtnet: 1 86056 1.95 MB/sec execute 197 sec latency 16674.847oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12132 3591728 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37424 3568756 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49556 7160484 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg254-server Stopping /mnt/lustre-mds1 (opts:) on oleg254-server 06:16:00 (1734434160) shut down facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server Failover mds1 to oleg254-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 06:16:14 (1734434174) targets are mounted 06:16:14 (1734434174) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2380 1285308 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12476 3593528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3569016 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49892 7162544 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg254-server Stopping /mnt/lustre-mds2 (opts:) on oleg254-server 06:16:26 (1734434186) shut down facet: mds2 facet_host: oleg254-server facet_failover_host: oleg254-server ms oleg254-client.virtnet: 1 86056 1.94 MB/sec execute 198 sec latency 17674.981 ms oleg254-client.virtnet: 1 86056 1.93 MB/sec execute 199 sec latency 18675.103 ms oleg254-client.virtnet: 1 86344 1.92 MB/sec execute 200 sec latency 18865.169 ms oleg254-client.virtnet: 1 86835 1.93 MB/sec execute 201 sec latency 13.353 ms oleg254-client.virtnet: 1 87378 1.92 MB/sec execute 202 sec latency 9.708 ms oleg254-client.virtnet: 1 88344 1.95 MB/sec execute 203 sec latency 10.672 ms oleg254-client.virtnet: 1 89050 1.97 MB/sec execute 204 sec latency 11.303 ms oleg254-client.virtnet: 1 89502 1.96 MB/sec execute 205 sec latency 12.488 ms oleg254-client.virtnet: 1 90029 1.96 MB/sec execute 206 sec latency 12.396 ms oleg254-client.virtnet: 1 90562 1.97 MB/sec execute 207 sec latency 11.391 ms oleg254-client.virtnet: 1 91292 1.97 MB/sec execute 208 sec latency 8.681 ms oleg254-client.virtnet: 1 92415 2.01 MB/sec execute 209 sec latency 8.431 ms oleg254-client.virtnet: 1 92932 2.00 MB/sec execute 210 sec latency 11.464 ms oleg254-client.virtnet: 1 93415 2.00 MB/sec execute 211 sec latency 13.928 ms oleg254-client.virtnet: 1 93986 2.01 MB/sec execute 212 sec latency 13.011 ms oleg254-client.virtnet: 1 94478 2.00 MB/sec execute 213 sec latency 11.867 ms oleg254-client.virtnet: 1 95457 2.03 MB/sec execute 214 sec latency 10.274 ms oleg254-client.virtnet: 1 96202 2.05 MB/sec execute 215 sec latency 8.436 ms oleg254-client.virtnet: 1 96700 2.04 MB/sec execute 216 sec latency 11.749 ms oleg254-client.virtnet: 1 97327 2.05 MB/sec execute 217 sec latency 9.106 ms oleg254-client.virtnet: 1 97907 2.04 MB/sec execute 218 sec latency 9.586 ms oleg254-client.virtnet: 1 98737 2.07 MB/sec execute 219 sec latency 9.369 ms oleg254-client.virtnet: 1 99578 2.08 MB/sec execute 220 sec latency 10.025 ms oleg254-client.virtnet: 1 100083 2.08 MB/sec execute 221 sec latency 12.513 ms oleg254-client.virtnet: 1 100592 2.08 MB/sec execute 222 sec latency 10.118 ms oleg254-client.virtnet: 1 101179 2.08 MB/sec execute 223 sec latency 11.516 ms oleg254-client.virtnet: 1 101829 2.08 MB/sec execute 224 sec latency 8.415 ms oleg254-client.virtnet: 1 102935 2.12 MB/sec execute 225 sec latency 10.245 ms oleg254-client.virtnet: 1 103347 2.12 MB/sec execute 226 sec latency 12.049 ms oleg254-client.virtnet: 1 103784 2.11 MB/sec execute 227 sec latency 14.315 ms oleg254-client.virtnet: 1 104179 2.11 MB/sec execute 228 sec latency 15.907 ms oleg254-client.virtnet: 1 104691 2.11 MB/sec execute 229 sec latency 11.650 ms oleg254-client.virtnet: 1 105159 2.10 MB/sec execute 230 sec latency 12.076 ms oleg254-client.virtnet: 1 105781 2.12 MB/sec execute 231 sec latency 195.745 ms oleg254-client.virtnet: 1 106566 2.14 MB/sec execute 232 sec latency 11.142 ms oleg254-client.virtnet: 1 106759 2.13 MB/sec execute 233 sec latency 466.879 ms oleg254-client.virtnet: 1 106759 2.12 MB/sec execute 234 sec latency 1467.076 ms oleg254-client.virtnet: 1 106759 2.11 MB/sec execute 235 sec latency 2467.260 ms oleg254-client.virtnet: 1 106759 2.11 MB/sec execute 236 sec latency 3467.439 ms oleg254-client.virtnet: 1 106759 2.10 MB/sec execute 237 sec latency 4467.591 ms oleg254-client.virtnet: 1 106759 2.09 MB/sec execute 238 sec latency 5467.792 ms oleg254-client.virtnet: 1 106759 2.08 MB/sec execute 239 sec latency 6468.518 ms oleg254-client.virtnet: 1 106759 2.07 MB/sec execute 240 sec latency 7468.718 ms oleg254-client.virtnet: 1 106759 2.06 MB/sec execute 241 sec latency 8468.894 ms oleg254-client.virtnet: 1 106759 2.05 MB/sec execute 242 sFailover mds2 to oleg254-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 06:16:40 (1734434200) targets are mounted 06:16:40 (1734434200) facet_failover done oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec ec latency 9469.069 ms oleg254-client.virtnet: 1 106759 2.04 MB/sec execute 243 sec latency 10469.225 ms oleg254-client.virtnet: 1 106759 2.04 MB/sec execute 244 sec latency 11469.321 ms oleg254-client.virtnet: 1 106759 2.03 MB/sec execute 245 sec latency 12469.469 ms oleg254-client.virtnet: 1 106759 2.02 MB/sec execute 246 sec latency 13469.654 ms oleg254-client.virtnet: 1 106759 2.01 MB/sec execute 247 sec latency 14469.844 ms oleg254-client.virtnet: 1 106759 2.00 MB/sec execute 248 sec latency 15470.011 ms oleg254-client.virtnet: 1 106759 2.00 MB/sec execute 249 sec latency 16470.255 ms oleg254-client.virtnet: 1 106759 1.99 MB/sec execute 250 sec latency 17470.411 ms oleg254-client.virtnet: 1 106759 1.98 MB/sec execute 251 sec latency 18470.895 ms oleg254-client.virtnet: 1 106939 1.97 MB/sec execute 252 sec latency 18820.202 ms oleg254-client.virtnet: 1 107221 1.97 MB/sec execute 253 sec latency 28.356 ms oleg254-client.virtnet: 1 107510 1.97 MB/sec execute 254 sec latency 19.959 ms oleg254-client.virtnet: 1 107923 1.97 MB/sec execute 255 sec latency 16.131 ms oleg254-client.virtnet: 1 108277 1.96 MB/sec execute 256 sec latency 13.798 ms oleg254-client.virtnet: 1 108802 1.96 MB/sec execute 257 sec latency 12.447 ms oleg254-client.virtnet: 1 109682 1.98 MB/sec execute 258 sec latency 10.201 ms oleg254-client.virtnet: 1 110321 1.99 MB/sec execute 259 sec latency 11.033 ms oleg254-client.virtnet: 1 110777 1.99 MB/sec execute 260 sec latency 12.848 ms oleg254-client.virtnet: 1 111275 1.99 MB/sec execute 261 sec latency 10.663 ms oleg254-client.virtnet: 1 111822 1.99 MB/sec execute 262 sec latency 11.082 ms oleg254-client.virtnet: 1 112468 1.99 MB/sec execute 263 sec latency 8.779 ms oleg254-client.virtnet: 1 113539 2.02 MB/sec execute 264 sec latency 10.041 ms oleg254-client.virtnet: 1 114030 2.02 MB/sec execute 265 sec latency 9.282 ms oleg254-client.virtnet: 1 114548 2.02 MB/sec execute 266 sec latency 15.853 ms oleg254-client.virtnet: 1 115134 2.02 MB/sec execute 267 sec latency 10.414 ms oleg254-client.virtnet: 1 115604 2.02 MB/sec execute 268 sec latency 15.326 ms oleg254-client.virtnet: 1 116420 2.03 MB/sec execute 269 sec latency 11.763 ms oleg254-client.virtnet: 1 117239 2.05 MB/sec execute 270 sec latency 10.956 ms oleg254-client.virtnet: 1 117714 2.05 MB/sec execute 271 sec latency 9.679 ms oleg254-client.virtnet: 1 118259 2.05 MB/sec execute 272 sec latency 13.698 ms oleg254-client.virtnet: 1 118807 2.05 MB/sec execute 273 sec latency 15.573 ms oleg254-client.virtnet: 1 119241 2.04 MB/sec execute 274 sec latency 12.230 ms oleg254-client.virtnet: 1 119979 2.06 MB/sec execute 275 sec latency 13.331 ms oleg254-client.virtnet: 1 120696 2.07 MB/sec execute 276 sec latency 13.780 ms oleg254-client.virtnet: 1 121029 2.07 MB/sec execute 277 sec latency 17.223 ms oleg254-client.virtnet: 1 121392 2.06 MB/sec execute 278 sec latency 15.156 ms oleg254-client.virtnet: 1 121796 2.06 MB/sec execute 279 sec latency 12.222 ms oleg254-client.virtnet: 1 122284 2.07 MB/sec execute 280 sec latency 12.279 ms oleg254-client.virtnet: 1 122765 2.06 MB/sec execute 281 sec latency 12.502 ms oleg254-client.virtnet: 1 123381 2.07 MB/sec execute 282 sec latency 14.025 ms oleg254-client.virtnet: 1 124190 2.09 MB/sec execute 283 sec latency 15.883 ms oleg254-client.virtnet: 1 124597 2.09 MB/sec execute 284 sec latency 11.952 ms oleg254-client.virtnet: 1 125041 2.08 MB/sec execute 285 sec latency 13.469 ms oleg254-client.virtnet: 1 125454 2.08 MB/sec execute 286 sec latency 12.440 ms oleg254-client.virtnet: 1 125921 2.08 MB/sec execute 287 sec latency 13.668 ms oleg254-client.virtnet: 1 126399 2.08 MB/sec execute 288 sec latency 10.334 ms oleg254-client.virtnet: 1 127163 2.09 MB/sec execute 289 sec latency 15.072 ms oleg254-client.virtnet: 1 127972 2.11 MB/sec execute 290 sec latency 10.964 ms oleg254-client.virtnet: 1 128472 2.10 MB/sec execute 291 sec latency 10.802 ms oleg254-client.virtnet: 1 129132 2.11 MB/sec execute 292 sec latency 10.468 ms oleg254-client.virtnet: 1 129619 2.11 MB/sec execute 293 sec latency 10.787 ms oleg254-client.virtnet: 1 130226 2.11 MB/sec execute 294 sec latency 11.304 ms oleg254-client.virtnet: 1 131257 2.13 MB/sec execute 295 sec latency 11.229 ms oleg254-client.virtnet: 1 131716 2.13 MB/sec execute 296 sec latency 10.972 ms oleg254-client.virtnet: 1 132235 2.13 MB/sec execute 297 sec latency 11.390 ms oleg254-client.virtnet: 1 132795 2.13 MB/sec execute 298 sec latency 12.979 ms oleg254-client.virtnet: 1 133322 2.13 MB/sec execute 299 sec latency 10.303 ms oleg254-client.virtnet: 1 cleanup 300 sec oleg254-client.virtnet: 0 cleanup 300 sec oleg254-client.virtnet: oleg254-client.virtnet: Operation Count AvgLat MaxLat oleg254-client.virtnet: ---------------------------------------- oleg254-client.virtnet: NTCreateX 20637 8.289 18820.193 oleg254-client.virtnet: Close 15184 2.270 18865.149 oleg254-client.virtnet: Rename 873 8.254 20.386 oleg254-client.virtnet: Unlink 4162 6.552 19029.347 oleg254-client.virtnet: Qpathinfo 18742 1.464 16.314 oleg254-client.virtnet: Qfileinfo 3264 0.153 2.827 oleg254-client.virtnet: Qfsinfo 3412 0.223 11.408 oleg254-client.virtnet: Sfileinfo 1698 4.938 18.114 oleg254-client.virtnet: Find 7227 0.410 12.952 oleg254-client.virtnet: WriteX 10201 0.856 9.462 oleg254-client.virtnet: ReadX 32206 0.021 3.932 oleg254-client.virtnet: LockX 66 1.084 2.538 oleg254-client.virtnet: UnlockX 66 0.978 1.838 oleg254-client.virtnet: Flush 1453 6.441 421.722 oleg254-client.virtnet: oleg254-client.virtnet: Throughput 2.1294 MB/sec 1 clients 1 procs max_latency=19029.364 ms oleg254-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg254-client.virtnet at Tue Dec 17 06:17:33 EST 2024 with return code 0 oleg254-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: /mnt/lustre/d70b.replay-single/oleg254-client.virtnet /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg254-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg254-client.virtnet: removed directory: 'clients/client0' oleg254-client.virtnet: removed directory: 'clients' oleg254-client.virtnet: removed 'client.txt' oleg254-client.virtnet: /mnt/lustre/d70b.replay-single/oleg254-client.virtnet oleg254-client.virtnet: dbench successfully finished