== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 16:25:49 (1744143949) Starting client oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre Started clients oleg431-client.virtnet: 192.168.204.131@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg431-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14867 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3592856 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26140 3580880 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7173736 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg431-server Stopping /mnt/lustre-mds1 (opts:) on oleg431-server 16:25:57 (1744143957) shut down facet: mds1 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds1 to oleg431-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 16:26:12 (1744143972) targets are mounted 16:26:12 (1744143972) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1748 3593020 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26404 3569868 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28152 7162888 1% /mnt/lustre oleg431-client.virtnet: looking for dbench program oleg431-client.virtnet: /usr/bin/dbench oleg431-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg431-client.virtnet oleg431-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg431-client.virtnet' oleg431-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg431-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg431-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg431-client.virtnet at Tue Apr 8 16:25:50 EDT 2025 oleg431-client.virtnet: waiting for dbench pid 14892 oleg431-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg431-client.virtnet: oleg431-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg431-client.virtnet: failed to create barrier semaphore oleg431-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg431-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg431-client.virtnet: releasing clients oleg431-client.virtnet: 1 313 11.48 MB/sec warmup 1 sec latency 28.203 ms oleg431-client.virtnet: 1 689 10.59 MB/sec warmup 2 sec latency 19.656 ms oleg431-client.virtnet: 1 1202 7.68 MB/sec warmup 3 sec latency 12.458 ms oleg431-client.virtnet: 1 1785 6.57 MB/sec warmup 4 sec latency 13.532 ms oleg431-client.virtnet: 1 2434 5.62 MB/sec warmup 5 sec latency 10.365 ms oleg431-client.virtnet: 1 3541 6.40 MB/sec warmup 6 sec latency 14.165 ms oleg431-client.virtnet: 1 3926 5.72 MB/sec warmup 7 sec latency 15.228 ms oleg431-client.virtnet: 1 4083 5.01 MB/sec warmup 8 sec latency 612.022 ms oleg431-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 1612.320 ms oleg431-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 2612.555 ms oleg431-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 3612.850 ms oleg431-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 4613.151 ms oleg431-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 5613.426 ms oleg431-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 6613.611 ms oleg431-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 7613.845 ms oleg431-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 8614.258 ms oleg431-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 9614.522 ms oleg431-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 10614.766 ms oleg431-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 11615.060 ms oleg431-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 12615.223 ms oleg431-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 13615.474 ms oleg431-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 14615.761 ms oleg431-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 15616.034 ms oleg431-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 16616.298 ms oleg431-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 17616.603 ms oleg431-client.virtnet: 1 4419 1.56 MB/sec warmup 26 sec latency 17658.576 ms oleg431-client.virtnet: 1 4751 1.56 MB/sec warmup 27 sec latency 18.877 ms oleg431-client.virtnet: 1 5119 1.61 MB/sec warmup 28 sec latency 18.357 ms oleg431-client.virtnet: 1 5370 1.57 MB/sec warmup 29 sec latency 21.370 ms oleg431-client.virtnet: 1 5720 1.53 MB/sec warmup 30 sec latency 13.961 ms oleg431-client.virtnet: 1 6050 1.54 MB/sec warmup 31 sec latency 18.147 ms oleg431-client.virtnet: 1 6486 1.64 MB/sec warmup 32 sec latency 358.598 ms oleg431-client.virtnet: 1 7114 1.74 MB/sec warmup 33 sec latency 18.test_70b fail mds2 2 times Failing mds2 on oleg431-server Stopping /mnt/lustre-mds2 (opts:) on oleg431-server 16:26:25 (1744143985) shut down facet: mds2 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds2 to oleg431-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 16:26:41 (1744144001) targets are mounted 16:26:41 (1744144001) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12948 3593300 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36544 3569940 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7163240 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg431-server Stopping /mnt/lustre-mds1 (opts:) on oleg431-server 16:26:54 (1744144014) shut down facet: mds1 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds1 to oleg431-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 682 ms oleg431-client.virtnet: 1 7408 1.73 MB/sec warmup 34 sec latency 16.811 ms oleg431-client.virtnet: 1 7510 1.69 MB/sec warmup 35 sec latency 630.605 ms oleg431-client.virtnet: 1 7510 1.64 MB/sec warmup 36 sec latency 1630.939 ms oleg431-client.virtnet: 1 7510 1.60 MB/sec warmup 37 sec latency 2631.266 ms oleg431-client.virtnet: 1 7510 1.56 MB/sec warmup 38 sec latency 3631.466 ms oleg431-client.virtnet: 1 7510 1.52 MB/sec warmup 39 sec latency 4631.758 ms oleg431-client.virtnet: 1 7510 1.48 MB/sec warmup 40 sec latency 5632.035 ms oleg431-client.virtnet: 1 7510 1.44 MB/sec warmup 41 sec latency 6632.423 ms oleg431-client.virtnet: 1 7510 1.41 MB/sec warmup 42 sec latency 7632.759 ms oleg431-client.virtnet: 1 7510 1.38 MB/sec warmup 43 sec latency 8632.991 ms oleg431-client.virtnet: 1 7510 1.34 MB/sec warmup 44 sec latency 9633.287 ms oleg431-client.virtnet: 1 7510 1.31 MB/sec warmup 45 sec latency 10633.568 ms oleg431-client.virtnet: 1 7510 1.29 MB/sec warmup 46 sec latency 11633.770 ms oleg431-client.virtnet: 1 7510 1.26 MB/sec warmup 47 sec latency 12633.969 ms oleg431-client.virtnet: 1 7510 1.23 MB/sec warmup 48 sec latency 13634.185 ms oleg431-client.virtnet: 1 7510 1.21 MB/sec warmup 49 sec latency 14634.345 ms oleg431-client.virtnet: 1 7510 1.18 MB/sec warmup 50 sec latency 15634.603 ms oleg431-client.virtnet: 1 7510 1.16 MB/sec warmup 51 sec latency 16634.977 ms oleg431-client.virtnet: 1 7510 1.14 MB/sec warmup 52 sec latency 17635.284 ms oleg431-client.virtnet: 1 7510 1.12 MB/sec warmup 53 sec latency 18635.509 ms oleg431-client.virtnet: 1 7510 1.10 MB/sec warmup 54 sec latency 19635.728 ms oleg431-client.virtnet: 1 7783 1.08 MB/sec warmup 55 sec latency 19759.744 ms oleg431-client.virtnet: 1 8066 1.07 MB/sec warmup 56 sec latency 18.863 ms oleg431-client.virtnet: 1 8500 1.13 MB/sec warmup 57 sec latency 18.180 ms oleg431-client.virtnet: 1 8931 1.11 MB/sec warmup 58 sec latency 15.274 ms oleg431-client.virtnet: 1 9296 1.10 MB/sec warmup 59 sec latency 15.390 ms oleg431-client.virtnet: 1 10592 9.27 MB/sec execute 1 sec latency 17.917 ms oleg431-client.virtnet: 1 10919 5.34 MB/sec execute 2 sec latency 16.063 ms oleg431-client.virtnet: 1 11200 3.67 MB/sec execute 3 sec latency 14.887 ms oleg431-client.virtnet: 1 11594 2.95 MB/sec execute 4 sec latency 17.155 ms oleg431-client.virtnet: 1 12208 3.23 MB/sec execute 5 sec latency 11.183 ms oleg431-client.virtnet: 1 12469 2.73 MB/sec execute 6 sec latency 19.176 ms oleg431-client.virtnet: 1 12817 2.39 MB/sec execute 7 sec latency 13.126 ms oleg431-client.virtnet: 1 13150 2.35 MB/sec execute 8 sec latency 18.615 ms oleg431-client.virtnet: 1 13847 2.70 MB/sec execute 9 sec latency 18.336 ms oleg431-client.virtnet: 1 14399 2.98 MB/sec execute 10 sec latency 16.804 ms oleg431-client.virtnet: 1 14646 2.74 MB/sec execute 11 sec latency 16.862 ms oleg431-client.virtnet: 1 14972 2.54 MB/sec execute 12 sec latency 18.615 ms oleg431-client.virtnet: 1 15255 2.47 MB/sec execute 13 sec latency 20.276 ms oleg431-client.virtnet: 1 15633 2.51 MB/sec execute 14 sec latency 20.258 ms oleg431-client.virtnet: 1 15916 2.36 MB/sec execute 15 sec latency 20.770 ms oleg431-client.virtnet: 1 16221 2.23 MB/sec execute 16 sec latency 16.179 ms oleg431-client.virtnet: 1 16596 2.18 MB/sec execute 17 sec latency 16.645 ms oleg431-client.virtnet: 1 17160 2.37 MB/sec execute 18 sec latency 17.509 ms oleg431-client.virtnet: 1 17857 2.52 MB/sec exeStarted lustre-MDT0000 16:27:09 (1744144029) targets are mounted 16:27:09 (1744144029) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12952 3593124 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36548 3569820 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162944 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg431-server Stopping /mnt/lustre-mds2 (opts:) on oleg431-server 16:27:23 (1744144043) shut down facet: mds2 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds2 to oleg431-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 16:27:38 (1744144058) targets are mounted 16:27:38 (1744144058) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12948 3593244 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36544 3569912 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7163156 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg431-server Stopping /mnt/lustre-mds1 (opts:) on oleg431-server 16:27:52 (1744144072) shut down facet: mds1 facet_host: oleg431-server facet_failover_host: oleg431-server cute 19 sec latency 18.612 ms oleg431-client.virtnet: 1 18115 2.46 MB/sec execute 20 sec latency 16.249 ms oleg431-client.virtnet: 1 18400 2.35 MB/sec execute 21 sec latency 16.475 ms oleg431-client.virtnet: 1 18675 2.28 MB/sec execute 22 sec latency 19.317 ms oleg431-client.virtnet: 1 19004 2.23 MB/sec execute 23 sec latency 18.821 ms oleg431-client.virtnet: 1 19370 2.27 MB/sec execute 24 sec latency 15.573 ms oleg431-client.virtnet: 1 19662 2.19 MB/sec execute 25 sec latency 19.364 ms oleg431-client.virtnet: 1 19987 2.12 MB/sec execute 26 sec latency 13.288 ms oleg431-client.virtnet: 1 20359 2.12 MB/sec execute 27 sec latency 18.314 ms oleg431-client.virtnet: 1 21012 2.24 MB/sec execute 28 sec latency 17.992 ms oleg431-client.virtnet: 1 21471 2.31 MB/sec execute 29 sec latency 13.902 ms oleg431-client.virtnet: 1 21746 2.28 MB/sec execute 30 sec latency 16.123 ms oleg431-client.virtnet: 1 22047 2.21 MB/sec execute 31 sec latency 19.387 ms oleg431-client.virtnet: 1 22289 2.16 MB/sec execute 32 sec latency 124.994 ms oleg431-client.virtnet: 1 22289 2.10 MB/sec execute 33 sec latency 1125.296 ms oleg431-client.virtnet: 1 22289 2.04 MB/sec execute 34 sec latency 2125.598 ms oleg431-client.virtnet: 1 22289 1.98 MB/sec execute 35 sec latency 3125.889 ms oleg431-client.virtnet: 1 22289 1.92 MB/sec execute 36 sec latency 4126.155 ms oleg431-client.virtnet: 1 22289 1.87 MB/sec execute 37 sec latency 5126.415 ms oleg431-client.virtnet: 1 22289 1.82 MB/sec execute 38 sec latency 6126.659 ms oleg431-client.virtnet: 1 22289 1.78 MB/sec execute 39 sec latency 7126.906 ms oleg431-client.virtnet: 1 22289 1.73 MB/sec execute 40 sec latency 8127.172 ms oleg431-client.virtnet: 1 22289 1.69 MB/sec execute 41 sec latency 9127.385 ms oleg431-client.virtnet: 1 22289 1.65 MB/sec execute 42 sec latency 10127.654 ms oleg431-client.virtnet: 1 22289 1.61 MB/sec execute 43 sec latency 11127.941 ms oleg431-client.virtnet: 1 22289 1.57 MB/sec execute 44 sec latency 12128.124 ms oleg431-client.virtnet: 1 22289 1.54 MB/sec execute 45 sec latency 13128.321 ms oleg431-client.virtnet: 1 22289 1.51 MB/sec execute 46 sec latency 14128.485 ms oleg431-client.virtnet: 1 22289 1.47 MB/sec execute 47 sec latency 15128.723 ms oleg431-client.virtnet: 1 22289 1.44 MB/sec execute 48 sec latency 16128.962 ms oleg431-client.virtnet: 1 22289 1.41 MB/sec execute 49 sec latency 17129.241 ms oleg431-client.virtnet: 1 22289 1.39 MB/sec execute 50 sec latency 18129.474 ms oleg431-client.virtnet: 1 22289 1.36 MB/sec execute 51 sec latency 19129.798 ms oleg431-client.virtnet: 1 22296 1.33 MB/sec execute 52 sec latency 20088.052 ms oleg431-client.virtnet: 1 22720 1.39 MB/sec execute 53 sec latency 17.637 ms oleg431-client.virtnet: 1 22995 1.36 MB/sec execute 54 sec latency 15.920 ms oleg431-client.virtnet: 1 23308 1.34 MB/sec execute 55 sec latency 19.052 ms oleg431-client.virtnet: 1 23726 1.35 MB/sec execute 56 sec latency 12.749 ms oleg431-client.virtnet: 1 24532 1.43 MB/sec execute 57 sec latency 17.937 ms oleg431-client.virtnet: 1 25004 1.49 MB/sec execute 58 sec latency 17.972 ms oleg431-client.virtnet: 1 25268 1.48 MB/sec execute 59 sec latency 16.479 ms oleg431-client.virtnet: 1 25565 1.46 MB/sec execute 60 sec latency 18.139 ms oleg431-client.virtnet: 1 25851 1.45 MB/sec execute 61 sec latency 17.275 ms oleg431-client.virtnet: 1 26268 1.49 MB/sec execute 62 sec latency 18.922 ms oleg431-client.virtnet: 1 26568 1.47 MB/sec execute 63 sec latency 20.993 ms oleg431-Failover mds1 to oleg431-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 16:28:07 (1744144087) targets are mounted 16:28:07 (1744144087) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13084 3592824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36668 3567220 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49752 7160044 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg431-server Stopping /mnt/lustre-mds2 (opts:) on oleg431-server 16:28:21 (1744144101) shut down facet: mds2 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds2 to oleg431-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 16:28:36 (1744144116) targets are mounted 16:28:36 (1744144116) facet_failover done client.virtnet: 1 26889 1.45 MB/sec execute 64 sec latency 15.028 ms oleg431-client.virtnet: 1 27302 1.46 MB/sec execute 65 sec latency 18.599 ms oleg431-client.virtnet: 1 27805 1.52 MB/sec execute 66 sec latency 18.700 ms oleg431-client.virtnet: 1 28435 1.57 MB/sec execute 67 sec latency 17.498 ms oleg431-client.virtnet: 1 28766 1.57 MB/sec execute 68 sec latency 13.429 ms oleg431-client.virtnet: 1 29037 1.55 MB/sec execute 69 sec latency 16.744 ms oleg431-client.virtnet: 1 29393 1.54 MB/sec execute 70 sec latency 17.986 ms oleg431-client.virtnet: 1 29697 1.53 MB/sec execute 71 sec latency 20.936 ms oleg431-client.virtnet: 1 30057 1.55 MB/sec execute 72 sec latency 15.147 ms oleg431-client.virtnet: 1 30347 1.54 MB/sec execute 73 sec latency 18.719 ms oleg431-client.virtnet: 1 30767 1.53 MB/sec execute 74 sec latency 12.452 ms oleg431-client.virtnet: 1 31346 1.59 MB/sec execute 75 sec latency 18.076 ms oleg431-client.virtnet: 1 31976 1.63 MB/sec execute 76 sec latency 16.755 ms oleg431-client.virtnet: 1 32266 1.63 MB/sec execute 77 sec latency 15.576 ms oleg431-client.virtnet: 1 32518 1.61 MB/sec execute 78 sec latency 16.036 ms oleg431-client.virtnet: 1 32875 1.60 MB/sec execute 79 sec latency 18.330 ms oleg431-client.virtnet: 1 33374 1.64 MB/sec execute 80 sec latency 13.587 ms oleg431-client.virtnet: 1 33656 1.62 MB/sec execute 81 sec latency 15.905 ms oleg431-client.virtnet: 1 33980 1.60 MB/sec execute 82 sec latency 19.787 ms oleg431-client.virtnet: 1 34354 1.60 MB/sec execute 83 sec latency 13.438 ms oleg431-client.virtnet: 1 34878 1.65 MB/sec execute 84 sec latency 18.790 ms oleg431-client.virtnet: 1 35527 1.69 MB/sec execute 85 sec latency 17.802 ms oleg431-client.virtnet: 1 35812 1.68 MB/sec execute 86 sec latency 16.812 ms oleg431-client.virtnet: 1 36035 1.67 MB/sec execute 87 sec latency 17.950 ms oleg431-client.virtnet: 1 36378 1.65 MB/sec execute 88 sec latency 20.207 ms oleg431-client.virtnet: 1 36678 1.65 MB/sec execute 89 sec latency 19.751 ms oleg431-client.virtnet: 1 36962 1.67 MB/sec execute 90 sec latency 346.787 ms oleg431-client.virtnet: 1 36962 1.65 MB/sec execute 91 sec latency 1347.092 ms oleg431-client.virtnet: 1 36962 1.63 MB/sec execute 92 sec latency 2347.378 ms oleg431-client.virtnet: 1 36962 1.61 MB/sec execute 93 sec latency 3347.651 ms oleg431-client.virtnet: 1 36962 1.60 MB/sec execute 94 sec latency 4347.928 ms oleg431-client.virtnet: 1 36962 1.58 MB/sec execute 95 sec latency 5348.162 ms oleg431-client.virtnet: 1 36962 1.56 MB/sec execute 96 sec latency 6348.384 ms oleg431-client.virtnet: 1 36962 1.55 MB/sec execute 97 sec latency 7348.600 ms oleg431-client.virtnet: 1 36962 1.53 MB/sec execute 98 sec latency 8348.811 ms oleg431-client.virtnet: 1 36962 1.51 MB/sec execute 99 sec latency 9349.087 ms oleg431-client.virtnet: 1 36962 1.50 MB/sec execute 100 sec latency 10349.263 ms oleg431-client.virtnet: 1 36962 1.48 MB/sec execute 101 sec latency 11349.455 ms oleg431-client.virtnet: 1 36962 1.47 MB/sec execute 102 sec latency 12349.644 ms oleg431-client.virtnet: 1 36962 1.46 MB/sec execute 103 sec latency 13349.841 ms oleg431-client.virtnet: 1 36962 1.44 MB/sec execute 104 sec latency 14350.077 ms oleg431-client.virtnet: 1 36962 1.43 MB/sec execute 105 sec latency 15350.249 ms oleg431-client.virtnet: 1 36962 1.41 MB/sec execute 106 sec latency 16350.410 ms oleg431-client.virtnet: 1 36962 1.40 MB/sec execute 107 sec latency 17350.585 ms oleg431-client.virtnet: 1 36962 1.39 MB/sec eoleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13068 3593016 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36548 3567624 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160640 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg431-server Stopping /mnt/lustre-mds1 (opts:) on oleg431-server 16:28:49 (1744144129) shut down facet: mds1 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds1 to oleg431-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 16:29:04 (1744144144) targets are mounted 16:29:04 (1744144144) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3593188 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36544 3569964 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163152 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg431-server Stopping /mnt/lustre-mds2 (opts:) on oleg431-server 16:29:17 (1744144157) shut down facet: mds2 facet_host: oleg431-server facet_failover_host: oleg431-server xecute 108 sec latency 18350.776 ms oleg431-client.virtnet: 1 36962 1.38 MB/sec execute 109 sec latency 19351.114 ms oleg431-client.virtnet: 1 37067 1.36 MB/sec execute 110 sec latency 20062.566 ms oleg431-client.virtnet: 1 37652 1.36 MB/sec execute 111 sec latency 10.037 ms oleg431-client.virtnet: 1 38488 1.41 MB/sec execute 112 sec latency 10.032 ms oleg431-client.virtnet: 1 39197 1.44 MB/sec execute 113 sec latency 11.173 ms oleg431-client.virtnet: 1 39501 1.44 MB/sec execute 114 sec latency 17.031 ms oleg431-client.virtnet: 1 39911 1.43 MB/sec execute 115 sec latency 14.588 ms oleg431-client.virtnet: 1 40252 1.43 MB/sec execute 116 sec latency 18.077 ms oleg431-client.virtnet: 1 40634 1.45 MB/sec execute 117 sec latency 15.772 ms oleg431-client.virtnet: 1 40951 1.44 MB/sec execute 118 sec latency 16.341 ms oleg431-client.virtnet: 1 41465 1.44 MB/sec execute 119 sec latency 10.835 ms oleg431-client.virtnet: 1 42240 1.48 MB/sec execute 120 sec latency 14.782 ms oleg431-client.virtnet: 1 42802 1.51 MB/sec execute 121 sec latency 14.421 ms oleg431-client.virtnet: 1 43112 1.50 MB/sec execute 122 sec latency 12.752 ms oleg431-client.virtnet: 1 43478 1.49 MB/sec execute 123 sec latency 17.973 ms oleg431-client.virtnet: 1 43827 1.49 MB/sec execute 124 sec latency 15.293 ms oleg431-client.virtnet: 1 44218 1.51 MB/sec execute 125 sec latency 13.612 ms oleg431-client.virtnet: 1 44540 1.50 MB/sec execute 126 sec latency 18.035 ms oleg431-client.virtnet: 1 44952 1.50 MB/sec execute 127 sec latency 15.310 ms oleg431-client.virtnet: 1 45475 1.52 MB/sec execute 128 sec latency 16.369 ms oleg431-client.virtnet: 1 46127 1.55 MB/sec execute 129 sec latency 18.214 ms oleg431-client.virtnet: 1 46520 1.56 MB/sec execute 130 sec latency 13.690 ms oleg431-client.virtnet: 1 46870 1.55 MB/sec execute 131 sec latency 14.868 ms oleg431-client.virtnet: 1 47186 1.55 MB/sec execute 132 sec latency 16.295 ms oleg431-client.virtnet: 1 47679 1.56 MB/sec execute 133 sec latency 17.399 ms oleg431-client.virtnet: 1 48127 1.55 MB/sec execute 134 sec latency 11.268 ms oleg431-client.virtnet: 1 48537 1.55 MB/sec execute 135 sec latency 13.626 ms oleg431-client.virtnet: 1 49153 1.58 MB/sec execute 136 sec latency 14.690 ms oleg431-client.virtnet: 1 49786 1.61 MB/sec execute 137 sec latency 15.026 ms oleg431-client.virtnet: 1 50063 1.60 MB/sec execute 138 sec latency 16.109 ms oleg431-client.virtnet: 1 50379 1.59 MB/sec execute 139 sec latency 23.743 ms oleg431-client.virtnet: 1 50633 1.59 MB/sec execute 140 sec latency 18.910 ms oleg431-client.virtnet: 1 50941 1.59 MB/sec execute 141 sec latency 18.289 ms oleg431-client.virtnet: 1 51292 1.60 MB/sec execute 142 sec latency 18.088 ms oleg431-client.virtnet: 1 51568 1.59 MB/sec execute 143 sec latency 19.931 ms oleg431-client.virtnet: 1 51938 1.58 MB/sec execute 144 sec latency 13.390 ms oleg431-client.virtnet: 1 52284 1.58 MB/sec execute 145 sec latency 19.649 ms oleg431-client.virtnet: 1 52933 1.61 MB/sec execute 146 sec latency 19.080 ms oleg431-client.virtnet: 1 53217 1.62 MB/sec execute 147 sec latency 563.205 ms oleg431-client.virtnet: 1 53217 1.61 MB/sec execute 148 sec latency 1563.481 ms oleg431-client.virtnet: 1 53217 1.60 MB/sec execute 149 sec latency 2563.690 ms oleg431-client.virtnet: 1 53217 1.59 MB/sec execute 150 sec latency 3563.923 ms oleg431-client.virtnet: 1 53217 1.58 MB/sec execute 151 sec latency 4564.218 ms oleg431-client.virtnet: 1 53217 1.57 MB/sec execute 152 sec latency 5564.471 ms oleg431-client.virtnet: 1 53217Failover mds2 to oleg431-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 16:29:33 (1744144173) targets are mounted 16:29:33 (1744144173) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13288 3592624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36384 3567768 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49672 7160392 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg431-server Stopping /mnt/lustre-mds1 (opts:) on oleg431-server 16:29:47 (1744144187) shut down facet: mds1 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds1 to oleg431-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 16:30:02 (1744144202) targets are mounted 16:30:02 (1744144202) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1.56 MB/sec execute 153 sec latency 6564.697 ms oleg431-client.virtnet: 1 53217 1.55 MB/sec execute 154 sec latency 7564.922 ms oleg431-client.virtnet: 1 53217 1.54 MB/sec execute 155 sec latency 8565.219 ms oleg431-client.virtnet: 1 53217 1.53 MB/sec execute 156 sec latency 9565.503 ms oleg431-client.virtnet: 1 53217 1.52 MB/sec execute 157 sec latency 10565.757 ms oleg431-client.virtnet: 1 53217 1.51 MB/sec execute 158 sec latency 11565.948 ms oleg431-client.virtnet: 1 53217 1.50 MB/sec execute 159 sec latency 12566.184 ms oleg431-client.virtnet: 1 53217 1.49 MB/sec execute 160 sec latency 13566.415 ms oleg431-client.virtnet: 1 53217 1.48 MB/sec execute 161 sec latency 14566.596 ms oleg431-client.virtnet: 1 53217 1.47 MB/sec execute 162 sec latency 15566.787 ms oleg431-client.virtnet: 1 53217 1.46 MB/sec execute 163 sec latency 16566.985 ms oleg431-client.virtnet: 1 53217 1.46 MB/sec execute 164 sec latency 17567.204 ms oleg431-client.virtnet: 1 53217 1.45 MB/sec execute 165 sec latency 18567.409 ms oleg431-client.virtnet: 1 53217 1.44 MB/sec execute 166 sec latency 19567.591 ms oleg431-client.virtnet: 1 53416 1.43 MB/sec execute 167 sec latency 19888.784 ms oleg431-client.virtnet: 1 53669 1.43 MB/sec execute 168 sec latency 15.850 ms oleg431-client.virtnet: 1 53940 1.42 MB/sec execute 169 sec latency 20.643 ms oleg431-client.virtnet: 1 54203 1.42 MB/sec execute 170 sec latency 19.430 ms oleg431-client.virtnet: 1 54511 1.42 MB/sec execute 171 sec latency 18.278 ms oleg431-client.virtnet: 1 54869 1.43 MB/sec execute 172 sec latency 15.130 ms oleg431-client.virtnet: 1 55177 1.42 MB/sec execute 173 sec latency 19.547 ms oleg431-client.virtnet: 1 55568 1.42 MB/sec execute 174 sec latency 13.511 ms oleg431-client.virtnet: 1 55955 1.44 MB/sec execute 175 sec latency 19.245 ms oleg431-client.virtnet: 1 56595 1.45 MB/sec execute 176 sec latency 17.926 ms oleg431-client.virtnet: 1 56997 1.46 MB/sec execute 177 sec latency 16.224 ms oleg431-client.virtnet: 1 57231 1.46 MB/sec execute 178 sec latency 16.420 ms oleg431-client.virtnet: 1 57529 1.45 MB/sec execute 179 sec latency 17.869 ms oleg431-client.virtnet: 1 57785 1.45 MB/sec execute 180 sec latency 18.315 ms oleg431-client.virtnet: 1 58186 1.46 MB/sec execute 181 sec latency 18.677 ms oleg431-client.virtnet: 1 58459 1.46 MB/sec execute 182 sec latency 16.348 ms oleg431-client.virtnet: 1 58760 1.45 MB/sec execute 183 sec latency 19.904 ms oleg431-client.virtnet: 1 59126 1.45 MB/sec execute 184 sec latency 12.138 ms oleg431-client.virtnet: 1 59538 1.46 MB/sec execute 185 sec latency 19.727 ms oleg431-client.virtnet: 1 60141 1.47 MB/sec execute 186 sec latency 17.494 ms oleg431-client.virtnet: 1 60543 1.49 MB/sec execute 187 sec latency 15.235 ms oleg431-client.virtnet: 1 60778 1.48 MB/sec execute 188 sec latency 16.448 ms oleg431-client.virtnet: 1 61087 1.48 MB/sec execute 189 sec latency 20.533 ms oleg431-client.virtnet: 1 61421 1.48 MB/sec execute 190 sec latency 18.453 ms oleg431-client.virtnet: 1 61848 1.49 MB/sec execute 191 sec latency 17.314 ms oleg431-client.virtnet: 1 62132 1.48 MB/sec execute 192 sec latency 18.051 ms oleg431-client.virtnet: 1 62501 1.47 MB/sec execute 193 sec latency 12.572 ms oleg431-client.virtnet: 1 62883 1.48 MB/sec execute 194 sec latency 18.328 ms oleg431-client.virtnet: 1 63506 1.50 MB/sec execute 195 sec latency 17.178 ms oleg431-client.virtnet: 1 63993 1.51 MB/sec execute 196 sec latency 17.725 ms oleg431-client.virtnet: 1 64258 1.51 MB/sec execute 197 sec latency 17.012 mUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13180 3593668 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36384 3570208 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7163876 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg431-server Stopping /mnt/lustre-mds2 (opts:) on oleg431-server 16:30:16 (1744144216) shut down facet: mds2 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds2 to oleg431-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 16:30:31 (1744144231) targets are mounted 16:30:31 (1744144231) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13148 3590988 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36500 3569756 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7160744 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg431-server Stopping /mnt/lustre-mds1 (opts:) on oleg431-server 16:30:45 (1744144245) shut down facet: mds1 facet_host: oleg431-server facet_failover_host: oleg431-server s oleg431-client.virtnet: 1 64549 1.51 MB/sec execute 198 sec latency 24.786 ms oleg431-client.virtnet: 1 64819 1.50 MB/sec execute 199 sec latency 19.326 ms oleg431-client.virtnet: 1 65121 1.50 MB/sec execute 200 sec latency 20.086 ms oleg431-client.virtnet: 1 65484 1.51 MB/sec execute 201 sec latency 14.499 ms oleg431-client.virtnet: 1 65747 1.50 MB/sec execute 202 sec latency 20.819 ms oleg431-client.virtnet: 1 66117 1.50 MB/sec execute 203 sec latency 14.626 ms oleg431-client.virtnet: 1 66466 1.50 MB/sec execute 204 sec latency 19.700 ms oleg431-client.virtnet: 1 66870 1.51 MB/sec execute 205 sec latency 352.215 ms oleg431-client.virtnet: 1 66870 1.51 MB/sec execute 206 sec latency 1352.409 ms oleg431-client.virtnet: 1 66870 1.50 MB/sec execute 207 sec latency 2352.610 ms oleg431-client.virtnet: 1 66870 1.49 MB/sec execute 208 sec latency 3352.855 ms oleg431-client.virtnet: 1 66870 1.49 MB/sec execute 209 sec latency 4353.129 ms oleg431-client.virtnet: 1 66870 1.48 MB/sec execute 210 sec latency 5353.375 ms oleg431-client.virtnet: 1 66870 1.47 MB/sec execute 211 sec latency 6353.560 ms oleg431-client.virtnet: 1 66870 1.46 MB/sec execute 212 sec latency 7353.764 ms oleg431-client.virtnet: 1 66870 1.46 MB/sec execute 213 sec latency 8354.096 ms oleg431-client.virtnet: 1 66870 1.45 MB/sec execute 214 sec latency 9354.381 ms oleg431-client.virtnet: 1 66870 1.44 MB/sec execute 215 sec latency 10354.632 ms oleg431-client.virtnet: 1 66870 1.44 MB/sec execute 216 sec latency 11354.863 ms oleg431-client.virtnet: 1 66870 1.43 MB/sec execute 217 sec latency 12355.041 ms oleg431-client.virtnet: 1 66870 1.42 MB/sec execute 218 sec latency 13355.270 ms oleg431-client.virtnet: 1 66870 1.42 MB/sec execute 219 sec latency 14355.498 ms oleg431-client.virtnet: 1 66870 1.41 MB/sec execute 220 sec latency 15355.726 ms oleg431-client.virtnet: 1 66870 1.40 MB/sec execute 221 sec latency 16356.058 ms oleg431-client.virtnet: 1 66870 1.40 MB/sec execute 222 sec latency 17356.297 ms oleg431-client.virtnet: 1 66870 1.39 MB/sec execute 223 sec latency 18356.594 ms oleg431-client.virtnet: 1 66870 1.39 MB/sec execute 224 sec latency 19356.936 ms oleg431-client.virtnet: 1 67240 1.39 MB/sec execute 225 sec latency 19866.927 ms oleg431-client.virtnet: 1 67653 1.40 MB/sec execute 226 sec latency 15.406 ms oleg431-client.virtnet: 1 67883 1.40 MB/sec execute 227 sec latency 15.771 ms oleg431-client.virtnet: 1 68160 1.39 MB/sec execute 228 sec latency 23.509 ms oleg431-client.virtnet: 1 68413 1.39 MB/sec execute 229 sec latency 18.838 ms oleg431-client.virtnet: 1 68809 1.40 MB/sec execute 230 sec latency 18.609 ms oleg431-client.virtnet: 1 69079 1.39 MB/sec execute 231 sec latency 14.800 ms oleg431-client.virtnet: 1 69387 1.39 MB/sec execute 232 sec latency 20.960 ms oleg431-client.virtnet: 1 69755 1.39 MB/sec execute 233 sec latency 11.842 ms oleg431-client.virtnet: 1 70135 1.40 MB/sec execute 234 sec latency 18.827 ms oleg431-client.virtnet: 1 70768 1.41 MB/sec execute 235 sec latency 17.892 ms oleg431-client.virtnet: 1 71135 1.42 MB/sec execute 236 sec latency 14.039 ms oleg431-client.virtnet: 1 71388 1.42 MB/sec execute 237 sec latency 16.767 ms oleg431-client.virtnet: 1 71684 1.41 MB/sec execute 238 sec latency 19.359 ms oleg431-client.virtnet: 1 71942 1.41 MB/sec execute 239 sec latency 21.176 ms oleg431-client.virtnet: 1 72349 1.42 MB/sec execute 240 sec latency 19.700 ms oleg431-client.virtnet: 1 72620 1.42 MB/sec execute 241 sec latency 16.888 ms oleg431-client.virtnet: 1 72926Failover mds1 to oleg431-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 16:31:01 (1744144261) targets are mounted 16:31:01 (1744144261) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1.41 MB/sec execute 242 sec latency 19.726 ms oleg431-client.virtnet: 1 73250 1.41 MB/sec execute 243 sec latency 12.004 ms oleg431-client.virtnet: 1 73592 1.41 MB/sec execute 244 sec latency 18.797 ms oleg431-client.virtnet: 1 74298 1.43 MB/sec execute 245 sec latency 18.290 ms oleg431-client.virtnet: 1 74722 1.44 MB/sec execute 246 sec latency 12.929 ms oleg431-client.virtnet: 1 74976 1.44 MB/sec execute 247 sec latency 15.555 ms oleg431-client.virtnet: 1 75288 1.43 MB/sec execute 248 sec latency 19.072 ms oleg431-client.virtnet: 1 75562 1.43 MB/sec execute 249 sec latency 19.590 ms oleg431-client.virtnet: 1 75985 1.44 MB/sec execute 250 sec latency 17.992 ms oleg431-client.virtnet: 1 76328 1.44 MB/sec execute 251 sec latency 19.256 ms oleg431-client.virtnet: 1 76648 1.43 MB/sec execute 252 sec latency 13.477 ms oleg431-client.virtnet: 1 76984 1.43 MB/sec execute 253 sec latency 17.826 ms oleg431-client.virtnet: 1 77521 1.45 MB/sec execute 254 sec latency 18.571 ms oleg431-client.virtnet: 1 78160 1.46 MB/sec execute 255 sec latency 16.358 ms oleg431-client.virtnet: 1 78437 1.46 MB/sec execute 256 sec latency 15.536 ms oleg431-client.virtnet: 1 78711 1.46 MB/sec execute 257 sec latency 16.303 ms oleg431-client.virtnet: 1 78994 1.45 MB/sec execute 258 sec latency 21.903 ms oleg431-client.virtnet: 1 79308 1.45 MB/sec execute 259 sec latency 18.156 ms oleg431-client.virtnet: 1 79664 1.46 MB/sec execute 260 sec latency 18.338 ms oleg431-client.virtnet: 1 79930 1.45 MB/sec execute 261 sec latency 20.490 ms oleg431-client.virtnet: 1 80271 1.45 MB/sec execute 262 sec latency 12.025 ms oleg431-client.virtnet: 1 80594 1.45 MB/sec execute 263 sec latency 19.974 ms oleg431-client.virtnet: 1 81260 1.47 MB/sec execute 264 sec latency 19.297 ms oleg431-client.virtnet: 1 81745 1.48 MB/sec execute 265 sec latency 17.669 ms oleg431-client.virtnet: 1 82006 1.48 MB/sec execute 266 sec latency 16.509 ms oleg431-client.virtnet: 1 82303 1.47 MB/sec execute 267 sec latency 16.028 ms oleg431-client.virtnet: 1 82560 1.47 MB/sec execute 268 sec latency 19.493 ms oleg431-client.virtnet: 1 82871 1.47 MB/sec execute 269 sec latency 19.269 ms oleg431-client.virtnet: 1 83232 1.48 MB/sec execute 270 sec latency 21.044 ms oleg431-client.virtnet: 1 83509 1.47 MB/sec execute 271 sec latency 21.151 ms oleg431-client.virtnet: 1 83840 1.47 MB/sec execute 272 sec latency 12.693 ms oleg431-client.virtnet: 1 84201 1.47 MB/sec execute 273 sec latency 19.446 ms oleg431-client.virtnet: 1 84813 1.48 MB/sec execute 274 sec latency 19.268 ms oleg431-client.virtnet: 1 85276 1.50 MB/sec execute 275 sec latency 18.874 ms oleg431-client.virtnet: 1 85528 1.49 MB/sec execute 276 sec latency 17.658 ms oleg431-client.virtnet: 1 85796 1.49 MB/sec execute 277 sec latency 16.483 ms oleg431-client.virtnet: 1 86072 1.49 MB/sec execute 278 sec latency 21.978 ms oleg431-client.virtnet: 1 86363 1.49 MB/sec execute 279 sec latency 19.052 ms oleg431-client.virtnet: 1 86729 1.49 MB/sec execute 280 sec latency 18.950 ms oleg431-client.virtnet: 1 86981 1.49 MB/sec execute 281 sec latency 20.445 ms oleg431-client.virtnet: 1 87324 1.48 MB/sec execute 282 sec latency 12.707 ms oleg431-client.virtnet: 1 87631 1.48 MB/sec execute 283 sec latency 18.086 ms oleg431-client.virtnet: 1 88164 1.50 MB/sec execute 284 sec latency 18.181 ms oleg431-client.virtnet: 1 88753 1.51 MB/sec execute 285 sec latency 17.702 ms oleg431-client.virtnet: 1 89050 1.51 MB/sec execute 286 sec latency 16.678 ms oleg431-client.virtnet: 1 89319 1.50 MB/sec execute 287 sec latency 16.121 ms oleg431-client.virtnet: 1 89618 1.50 MB/sec execute 288 sec latency 19.110 ms oleg431-client.virtnet: 1 89914 1.50 MB/sec execute 289 sec latency 19.918 ms oleg431-client.virtnet: 1 90288 1.51 MB/sec execute 290 sec latency 19.488 ms oleg431-client.virtnet: 1 90539 1.50 MB/sec execute 291 sec latency 21.853 ms oleg431-client.virtnet: 1 90885 1.50 MB/sec execute 292 sec latency 12.363 ms oleg431-client.virtnet: 1 91212 1.50 MB/sec execute 293 sec latency 19.013 ms oleg431-client.virtnet: 1 91736 1.51 MB/sec execute 294 sec latency 19.702 ms oleg431-client.virtnet: 1 92347 1.52 MB/sec execute 295 sec latency 17.414 ms oleg431-client.virtnet: 1 92616 1.52 MB/sec execute 296 sec latency 15.392 ms oleg431-client.virtnet: 1 92894 1.52 MB/sec execute 297 sec latency 16.197 ms oleg431-client.virtnet: 1 93180 1.52 MB/sec execute 298 sec latency 21.517 ms oleg431-client.virtnet: 1 93490 1.51 MB/sec execute 299 sec latency 17.448 ms oleg431-client.virtnet: 1 cleanup 300 sec oleg431-client.virtnet: 0 cleanup 300 sec oleg431-client.virtnet: oleg431-client.virtnet: Operation Count AvgLat MaxLat oleg431-client.virtnet: ---------------------------------------- oleg431-client.virtnet: NTCreateX 14543 11.157 20088.031 oleg431-client.virtnet: Close 10655 1.694 16.094 oleg431-client.virtnet: Rename 617 13.039 21.835 oleg431-client.virtnet: Unlink 2957 2.900 6.557 oleg431-client.virtnet: Qpathinfo 13141 5.349 19888.759 oleg431-client.virtnet: Qfileinfo 2287 0.223 3.394 oleg431-client.virtnet: Qfsinfo 2434 0.370 17.423 oleg431-client.virtnet: Sfileinfo 1184 7.714 19.995 oleg431-client.virtnet: Find 5092 0.532 12.287 oleg431-client.virtnet: WriteX 7170 1.341 9.371 oleg431-client.virtnet: ReadX 22929 0.027 2.969 oleg431-client.virtnet: LockX 48 1.641 2.507 oleg431-client.virtnet: UnlockX 48 1.765 2.205 oleg431-client.virtnet: Flush 1023 8.148 24.774 oleg431-client.virtnet: oleg431-client.virtnet: Throughput 1.51478 MB/sec 1 clients 1 procs max_latency=20088.052 ms oleg431-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg431-client.virtnet at Tue Apr 8 16:31:51 EDT 2025 with return code 0 oleg431-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg431-client.virtnet oleg431-client.virtnet: /mnt/lustre/d70b.replay-single/oleg431-client.virtnet /mnt/lustre/d70b.replay-single/oleg431-client.virtnet oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg431-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg431-client.virtnet: removed directory: 'clients/client0' oleg431-client.virtnet: removed directory: 'clients' oleg431-client.virtnet: removed 'client.txt' oleg431-client.virtnet: /mnt/lustre/d70b.replay-single/oleg431-client.virtnet oleg431-client.virtnet: dbench successfully finished