== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 17:06:06 (1746133566) Starting client oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre Started clients oleg414-client.virtnet: 192.168.204.114@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg414-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13974 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26160 3556140 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7161596 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 17:06:17 (1746133577) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 17:06:34 (1746133594) targets are mounted 17:06:34 (1746133594) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg414-client.virtnet: looking for dbench program oleg414-client.virtnet: /usr/bin/dbench oleg414-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg414-client.virtnet oleg414-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg414-client.virtnet' oleg414-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg414-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg414-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg414-client.virtnet at Thu May 1 17:06:08 EDT 2025 oleg414-client.virtnet: waiting for dbench pid 13999 oleg414-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg414-client.virtnet: oleg414-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg414-client.virtnet: failed to create barrier semaphore oleg414-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg414-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg414-client.virtnet: releasing clients oleg414-client.virtnet: 1 155 6.51 MB/sec warmup 1 sec latency 54.902 ms oleg414-client.virtnet: 1 331 5.99 MB/sec warmup 2 sec latency 50.948 ms oleg414-client.virtnet: 1 501 5.81 MB/sec warmup 3 sec latency 55.807 ms oleg414-client.virtnet: 1 689 5.29 MB/sec warmup 4 sec latency 90.092 ms oleg414-client.virtnet: 1 789 4.29 MB/sec warmup 5 sec latency 308.716 ms oleg414-client.virtnet: 1 921 3.62 MB/sec warmup 6 sec latency 552.472 ms oleg414-client.virtnet: 1 1147 3.28 MB/sec warmup 7 sec latency 27.634 ms oleg414-client.virtnet: 1 1389 3.25 MB/sec warmup 8 sec latency 70.914 ms oleg414-client.virtnet: 1 1405 2.89 MB/sec warmup 9 sec latency 873.821 ms oleg414-client.virtnet: 1 1405 2.60 MB/sec warmup 10 sec latency 1874.451 ms oleg414-client.virtnet: 1 1405 2.36 MB/sec warmup 11 sec latency 2874.632 ms oleg414-client.virtnet: 1 1405 2.17 MB/sec warmup 12 sec latency 3875.195 ms oleg414-client.virtnet: 1 1405 2.00 MB/sec warmup 13 sec latency 4875.345 ms oleg414-client.virtnet: 1 1405 1.86 MB/sec warmup 14 sec latency 5877.703 ms oleg414-client.virtnet: 1 1405 1.73 MB/sec warmup 15 sec latency 6877.871 ms oleg414-client.virtnet: 1 1405 1.62 MB/sec warmup 16 sec latency 7878.090 ms oleg414-client.virtnet: 1 1405 1.53 MB/sec warmup 17 sec latency 8878.402 ms oleg414-client.virtnet: 1 1405 1.44 MB/sec warmup 18 sec latency 9879.483 ms oleg414-client.virtnet: 1 1405 1.37 MB/sec warmup 19 sec latency 10879.648 ms oleg414-client.virtnet: 1 1405 1.30 MB/sec warmup 20 sec latency 11880.477 ms oleg414-client.virtnet: 1 1405 1.24 MB/sec warmup 21 sec latency 12880.700 ms oleg414-client.virtnet: 1 1405 1.18 MB/sec warmup 22 sec latency 13880.872 ms oleg414-client.virtnet: 1 1405 1.13 MB/sec warmup 23 sec latency 14881.437 ms oleg414-client.virtnet: 1 1405 1.08 MB/sec warmup 24 sec latency 15882.085 ms oleg414-client.virtnet: 1 1405 1.04 MB/sec warmup 25 sec latency 16882.435 ms oleg414-client.virtnet: 1 1405 1.00 MB/sec warmup 26 sec latency 17882.602 ms oleg414-client.virtnet: 1 1405 0.96 MB/sec warmup 27 sec latency 18882.747 ms oleg414-client.virtnet: 1 1405 0.93 MB/sec warmup 28 sec latency 19883.891 ms oleg414-client.virtnet: 1 1422 0.90 MB/sec warmup 29 sec latency 20740.387 ms oleg414-client.virtnet: 1 1609 0.87 MB/sec warmup 30 sec latency 27.569 ms oleg414-client.virtnet: 1 1877 0.85 MB/sec warmup 31 sec latency 27.611 ms oleg414-client.virtnet: 1 2185 0.84 MB/sec warmup 32 sec latency 30.652 ms oleg414-client.virtnet: 1 2485 0.87 MB/sec warmup 33 sec lateUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 27192 3568804 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5572 3594300 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 32764 7163104 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg414-server Stopping /mnt/lustre-mds2 (opts:) on oleg414-server 17:06:47 (1746133607) shut down facet: mds2 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds2 to oleg414-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0001 17:07:03 (1746133623) targets are mounted 17:07:03 (1746133623) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37644 3567868 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11900 3593900 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49544 7161768 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 17:07:16 (1746133636) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds1 to oleg414-server mount facets: mds1 ncy 26.011 ms oleg414-client.virtnet: 1 3007 0.99 MB/sec warmup 34 sec latency 25.490 ms oleg414-client.virtnet: 1 3541 1.10 MB/sec warmup 35 sec latency 38.860 ms oleg414-client.virtnet: 1 3786 1.10 MB/sec warmup 36 sec latency 28.834 ms oleg414-client.virtnet: 1 4014 1.08 MB/sec warmup 37 sec latency 19.019 ms oleg414-client.virtnet: 1 4268 1.06 MB/sec warmup 38 sec latency 23.833 ms oleg414-client.virtnet: 1 4460 1.05 MB/sec warmup 39 sec latency 36.828 ms oleg414-client.virtnet: 1 4652 1.05 MB/sec warmup 40 sec latency 36.468 ms oleg414-client.virtnet: 1 4862 1.04 MB/sec warmup 41 sec latency 25.000 ms oleg414-client.virtnet: 1 5178 1.08 MB/sec warmup 42 sec latency 25.192 ms oleg414-client.virtnet: 1 5405 1.06 MB/sec warmup 43 sec latency 22.011 ms oleg414-client.virtnet: 1 5743 1.04 MB/sec warmup 44 sec latency 20.182 ms oleg414-client.virtnet: 1 6049 1.06 MB/sec warmup 45 sec latency 20.970 ms oleg414-client.virtnet: 1 6551 1.14 MB/sec warmup 46 sec latency 21.667 ms oleg414-client.virtnet: 1 7171 1.23 MB/sec warmup 47 sec latency 28.599 ms oleg414-client.virtnet: 1 7407 1.23 MB/sec warmup 48 sec latency 20.780 ms oleg414-client.virtnet: 1 7617 1.21 MB/sec warmup 49 sec latency 19.289 ms oleg414-client.virtnet: 1 7848 1.19 MB/sec warmup 50 sec latency 35.803 ms oleg414-client.virtnet: 1 7990 1.17 MB/sec warmup 51 sec latency 44.672 ms oleg414-client.virtnet: 1 8111 1.16 MB/sec warmup 52 sec latency 41.197 ms oleg414-client.virtnet: 1 8332 1.16 MB/sec warmup 53 sec latency 33.300 ms oleg414-client.virtnet: 1 8564 1.19 MB/sec warmup 54 sec latency 237.638 ms oleg414-client.virtnet: 1 8564 1.17 MB/sec warmup 55 sec latency 1238.116 ms oleg414-client.virtnet: 1 8564 1.15 MB/sec warmup 56 sec latency 2240.066 ms oleg414-client.virtnet: 1 8564 1.13 MB/sec warmup 57 sec latency 3240.259 ms oleg414-client.virtnet: 1 8564 1.11 MB/sec warmup 58 sec latency 4241.605 ms oleg414-client.virtnet: 1 8682 1.09 MB/sec warmup 59 sec latency 4698.837 ms oleg414-client.virtnet: 1 9189 0.35 MB/sec execute 1 sec latency 27.528 ms oleg414-client.virtnet: 1 9530 0.91 MB/sec execute 2 sec latency 24.100 ms oleg414-client.virtnet: 1 10042 2.40 MB/sec execute 3 sec latency 26.341 ms oleg414-client.virtnet: 1 10631 3.02 MB/sec execute 4 sec latency 28.604 ms oleg414-client.virtnet: 1 10871 2.69 MB/sec execute 5 sec latency 26.779 ms oleg414-client.virtnet: 1 11061 2.30 MB/sec execute 6 sec latency 26.433 ms oleg414-client.virtnet: 1 11352 2.00 MB/sec execute 7 sec latency 18.978 ms oleg414-client.virtnet: 1 11463 1.79 MB/sec execute 8 sec latency 576.095 ms oleg414-client.virtnet: 1 11463 1.59 MB/sec execute 9 sec latency 1576.254 ms oleg414-client.virtnet: 1 11463 1.43 MB/sec execute 10 sec latency 2576.615 ms oleg414-client.virtnet: 1 11463 1.30 MB/sec execute 11 sec latency 3577.116 ms oleg414-client.virtnet: 1 11463 1.19 MB/sec execute 12 sec latency 4577.715 ms oleg414-client.virtnet: 1 11463 1.10 MB/sec execute 13 sec latency 5578.389 ms oleg414-client.virtnet: 1 11463 1.02 MB/sec execute 14 sec latency 6579.364 ms oleg414-client.virtnet: 1 11463 0.95 MB/sec execute 15 sec latency 7579.956 ms oleg414-client.virtnet: 1 11463 0.89 MB/sec execute 16 sec latency 8580.956 ms oleg414-client.virtnet: 1 11463 0.84 MB/sec execute 17 sec latency 9581.297 ms oleg414-client.virtnet: 1 11463 0.79 MB/sec execute 18 sec latency 10581.548 ms oleg414-client.virtnet: 1 11463 0.75 MB/sec execute 19 secStarting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 17:07:31 (1746133651) targets are mounted 17:07:31 (1746133651) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37592 3568864 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11908 3594368 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7163232 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg414-server Stopping /mnt/lustre-mds2 (opts:) on oleg414-server 17:07:46 (1746133666) shut down facet: mds2 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds2 to oleg414-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0001 17:08:02 (1746133682) targets are mounted 17:08:02 (1746133682) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37596 3568628 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11900 3594156 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7162784 1% /mnt/lustre latency 11581.727 ms oleg414-client.virtnet: 1 11463 0.72 MB/sec execute 20 sec latency 12581.950 ms oleg414-client.virtnet: 1 11463 0.68 MB/sec execute 21 sec latency 13582.385 ms oleg414-client.virtnet: 1 11463 0.65 MB/sec execute 22 sec latency 14582.561 ms oleg414-client.virtnet: 1 11463 0.62 MB/sec execute 23 sec latency 15583.144 ms oleg414-client.virtnet: 1 11463 0.60 MB/sec execute 24 sec latency 16583.605 ms oleg414-client.virtnet: 1 11463 0.57 MB/sec execute 25 sec latency 17584.206 ms oleg414-client.virtnet: 1 11463 0.55 MB/sec execute 26 sec latency 18585.658 ms oleg414-client.virtnet: 1 11463 0.53 MB/sec execute 27 sec latency 19587.472 ms oleg414-client.virtnet: 1 11580 0.52 MB/sec execute 28 sec latency 19681.589 ms oleg414-client.virtnet: 1 11759 0.54 MB/sec execute 29 sec latency 44.777 ms oleg414-client.virtnet: 1 11930 0.53 MB/sec execute 30 sec latency 75.940 ms oleg414-client.virtnet: 1 12221 0.61 MB/sec execute 31 sec latency 29.196 ms oleg414-client.virtnet: 1 12384 0.60 MB/sec execute 32 sec latency 32.390 ms oleg414-client.virtnet: 1 12528 0.58 MB/sec execute 33 sec latency 37.929 ms oleg414-client.virtnet: 1 12749 0.57 MB/sec execute 34 sec latency 38.769 ms oleg414-client.virtnet: 1 13032 0.59 MB/sec execute 35 sec latency 23.999 ms oleg414-client.virtnet: 1 13254 0.61 MB/sec execute 36 sec latency 31.485 ms oleg414-client.virtnet: 1 13593 0.71 MB/sec execute 37 sec latency 44.655 ms oleg414-client.virtnet: 1 14014 0.74 MB/sec execute 38 sec latency 34.888 ms oleg414-client.virtnet: 1 14348 0.81 MB/sec execute 39 sec latency 21.827 ms oleg414-client.virtnet: 1 14546 0.82 MB/sec execute 40 sec latency 24.957 ms oleg414-client.virtnet: 1 14707 0.80 MB/sec execute 41 sec latency 26.680 ms oleg414-client.virtnet: 1 14906 0.79 MB/sec execute 42 sec latency 43.710 ms oleg414-client.virtnet: 1 15078 0.78 MB/sec execute 43 sec latency 39.046 ms oleg414-client.virtnet: 1 15266 0.79 MB/sec execute 44 sec latency 30.334 ms oleg414-client.virtnet: 1 15498 0.79 MB/sec execute 45 sec latency 23.560 ms oleg414-client.virtnet: 1 15763 0.83 MB/sec execute 46 sec latency 32.146 ms oleg414-client.virtnet: 1 15945 0.81 MB/sec execute 47 sec latency 32.437 ms oleg414-client.virtnet: 1 16187 0.80 MB/sec execute 48 sec latency 29.255 ms oleg414-client.virtnet: 1 16470 0.80 MB/sec execute 49 sec latency 24.330 ms oleg414-client.virtnet: 1 16696 0.81 MB/sec execute 50 sec latency 37.404 ms oleg414-client.virtnet: 1 17096 0.88 MB/sec execute 51 sec latency 40.694 ms oleg414-client.virtnet: 1 17469 0.91 MB/sec execute 52 sec latency 30.698 ms oleg414-client.virtnet: 1 17777 0.95 MB/sec execute 53 sec latency 30.648 ms oleg414-client.virtnet: 1 17921 0.95 MB/sec execute 54 sec latency 180.105 ms oleg414-client.virtnet: 1 17921 0.93 MB/sec execute 55 sec latency 1180.283 ms oleg414-client.virtnet: 1 17921 0.91 MB/sec execute 56 sec latency 2180.451 ms oleg414-client.virtnet: 1 17921 0.90 MB/sec execute 57 sec latency 3180.951 ms oleg414-client.virtnet: 1 18088 0.90 MB/sec execute 58 sec latency 3510.694 ms oleg414-client.virtnet: 1 18368 0.88 MB/sec execute 59 sec latency 18.733 ms oleg414-client.virtnet: 1 18642 0.88 MB/sec execute 60 sec latency 33.139 ms oleg414-client.virtnet: 1 18935 0.89 MB/sec execute 61 sec latency 18.006 ms oleg414-client.virtnet: 1 19255 0.92 MB/sec execute 62 sec latency 33.267 ms oleg414-client.virtnet: 1 19481 0.91 MB/sec execute 63 sec latency 25.417 ms oleg414-client.virtnet: 1 test_70b fail mds1 5 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 17:08:15 (1746133695) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 17:08:31 (1746133711) targets are mounted 17:08:31 (1746133711) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37588 3566676 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12024 3594172 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7160848 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg414-server Stopping /mnt/lustre-mds2 (opts:) on oleg414-server 17:08:44 (1746133724) shut down facet: mds2 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds2 to oleg414-server mount facets: mds2 19771 0.90 MB/sec execute 64 sec latency 20.013 ms oleg414-client.virtnet: 1 20028 0.89 MB/sec execute 65 sec latency 24.440 ms oleg414-client.virtnet: 1 20238 0.90 MB/sec execute 66 sec latency 39.567 ms oleg414-client.virtnet: 1 20274 0.89 MB/sec execute 67 sec latency 854.093 ms oleg414-client.virtnet: 1 20274 0.88 MB/sec execute 68 sec latency 1854.960 ms oleg414-client.virtnet: 1 20274 0.87 MB/sec execute 69 sec latency 2856.074 ms oleg414-client.virtnet: 1 20274 0.85 MB/sec execute 70 sec latency 3856.720 ms oleg414-client.virtnet: 1 20274 0.84 MB/sec execute 71 sec latency 4857.235 ms oleg414-client.virtnet: 1 20274 0.83 MB/sec execute 72 sec latency 5857.511 ms oleg414-client.virtnet: 1 20274 0.82 MB/sec execute 73 sec latency 6857.674 ms oleg414-client.virtnet: 1 20274 0.81 MB/sec execute 74 sec latency 7858.685 ms oleg414-client.virtnet: 1 20274 0.80 MB/sec execute 75 sec latency 8858.846 ms oleg414-client.virtnet: 1 20274 0.79 MB/sec execute 76 sec latency 9859.069 ms oleg414-client.virtnet: 1 20274 0.78 MB/sec execute 77 sec latency 10859.240 ms oleg414-client.virtnet: 1 20274 0.77 MB/sec execute 78 sec latency 11859.435 ms oleg414-client.virtnet: 1 20274 0.76 MB/sec execute 79 sec latency 12859.616 ms oleg414-client.virtnet: 1 20274 0.75 MB/sec execute 80 sec latency 13860.308 ms oleg414-client.virtnet: 1 20274 0.74 MB/sec execute 81 sec latency 14860.632 ms oleg414-client.virtnet: 1 20274 0.73 MB/sec execute 82 sec latency 15861.127 ms oleg414-client.virtnet: 1 20274 0.72 MB/sec execute 83 sec latency 16863.174 ms oleg414-client.virtnet: 1 20274 0.71 MB/sec execute 84 sec latency 17863.376 ms oleg414-client.virtnet: 1 20274 0.70 MB/sec execute 85 sec latency 18864.037 ms oleg414-client.virtnet: 1 20274 0.70 MB/sec execute 86 sec latency 19864.202 ms oleg414-client.virtnet: 1 20499 0.72 MB/sec execute 87 sec latency 20371.343 ms oleg414-client.virtnet: 1 20960 0.74 MB/sec execute 88 sec latency 30.048 ms oleg414-client.virtnet: 1 21398 0.78 MB/sec execute 89 sec latency 22.577 ms oleg414-client.virtnet: 1 21621 0.79 MB/sec execute 90 sec latency 26.174 ms oleg414-client.virtnet: 1 21824 0.78 MB/sec execute 91 sec latency 29.256 ms oleg414-client.virtnet: 1 22076 0.78 MB/sec execute 92 sec latency 25.232 ms oleg414-client.virtnet: 1 22329 0.78 MB/sec execute 93 sec latency 22.095 ms oleg414-client.virtnet: 1 22616 0.78 MB/sec execute 94 sec latency 36.377 ms oleg414-client.virtnet: 1 22908 0.80 MB/sec execute 95 sec latency 28.451 ms oleg414-client.virtnet: 1 23140 0.80 MB/sec execute 96 sec latency 24.576 ms oleg414-client.virtnet: 1 23425 0.79 MB/sec execute 97 sec latency 24.697 ms oleg414-client.virtnet: 1 23752 0.80 MB/sec execute 98 sec latency 22.743 ms oleg414-client.virtnet: 1 24308 0.85 MB/sec execute 99 sec latency 15.842 ms oleg414-client.virtnet: 1 24909 0.89 MB/sec execute 100 sec latency 18.300 ms oleg414-client.virtnet: 1 25254 0.89 MB/sec execute 101 sec latency 16.350 ms oleg414-client.virtnet: 1 25621 0.89 MB/sec execute 102 sec latency 22.283 ms oleg414-client.virtnet: 1 25982 0.90 MB/sec execute 103 sec latency 14.339 ms oleg414-client.virtnet: 1 26394 0.92 MB/sec execute 104 sec latency 17.983 ms oleg414-client.virtnet: 1 26736 0.91 MB/sec execute 105 sec latency 15.811 ms oleg414-client.virtnet: 1 27252 0.92 MB/sec execute 106 sec latency 10.492 ms oleg414-client.virtnet: 1 27868 0.96 MB/sec execute 107 sec latency 13.803 ms oleg414-client.virtnet: 1 28487 1.00 MB/sec execute 108 sec Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0001 17:08:59 (1746133739) targets are mounted 17:08:59 (1746133739) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37592 3568748 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11908 3594168 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162916 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 17:09:12 (1746133752) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 17:09:27 (1746133767) targets are mounted 17:09:27 (1746133767) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37588 3566628 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12120 3593520 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49708 7160148 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg414-server Stopping /mnt/lustre-mds2 (opts:) on oleg414-server latency 17.453 ms oleg414-client.virtnet: 1 28769 1.00 MB/sec execute 109 sec latency 28.612 ms oleg414-client.virtnet: 1 29036 0.99 MB/sec execute 110 sec latency 17.166 ms oleg414-client.virtnet: 1 29275 0.99 MB/sec execute 111 sec latency 54.700 ms oleg414-client.virtnet: 1 29275 0.98 MB/sec execute 112 sec latency 1055.198 ms oleg414-client.virtnet: 1 29275 0.97 MB/sec execute 113 sec latency 2055.434 ms oleg414-client.virtnet: 1 29275 0.97 MB/sec execute 114 sec latency 3056.242 ms oleg414-client.virtnet: 1 29375 0.96 MB/sec execute 115 sec latency 3622.864 ms oleg414-client.virtnet: 1 29608 0.96 MB/sec execute 116 sec latency 25.267 ms oleg414-client.virtnet: 1 29897 0.98 MB/sec execute 117 sec latency 34.993 ms oleg414-client.virtnet: 1 30114 0.97 MB/sec execute 118 sec latency 24.539 ms oleg414-client.virtnet: 1 30408 0.97 MB/sec execute 119 sec latency 22.219 ms oleg414-client.virtnet: 1 30720 0.97 MB/sec execute 120 sec latency 20.806 ms oleg414-client.virtnet: 1 30855 0.96 MB/sec execute 121 sec latency 539.106 ms oleg414-client.virtnet: 1 31408 1.00 MB/sec execute 122 sec latency 24.238 ms oleg414-client.virtnet: 1 31987 1.03 MB/sec execute 123 sec latency 23.568 ms oleg414-client.virtnet: 1 32094 1.03 MB/sec execute 124 sec latency 656.216 ms oleg414-client.virtnet: 1 32094 1.02 MB/sec execute 125 sec latency 1656.458 ms oleg414-client.virtnet: 1 32094 1.01 MB/sec execute 126 sec latency 2656.629 ms oleg414-client.virtnet: 1 32094 1.00 MB/sec execute 127 sec latency 3656.915 ms oleg414-client.virtnet: 1 32094 0.99 MB/sec execute 128 sec latency 4657.093 ms oleg414-client.virtnet: 1 32094 0.99 MB/sec execute 129 sec latency 5657.247 ms oleg414-client.virtnet: 1 32094 0.98 MB/sec execute 130 sec latency 6657.427 ms oleg414-client.virtnet: 1 32094 0.97 MB/sec execute 131 sec latency 7657.597 ms oleg414-client.virtnet: 1 32094 0.96 MB/sec execute 132 sec latency 8657.807 ms oleg414-client.virtnet: 1 32094 0.96 MB/sec execute 133 sec latency 9658.347 ms oleg414-client.virtnet: 1 32094 0.95 MB/sec execute 134 sec latency 10659.052 ms oleg414-client.virtnet: 1 32094 0.94 MB/sec execute 135 sec latency 11659.364 ms oleg414-client.virtnet: 1 32094 0.94 MB/sec execute 136 sec latency 12659.506 ms oleg414-client.virtnet: 1 32094 0.93 MB/sec execute 137 sec latency 13659.677 ms oleg414-client.virtnet: 1 32094 0.92 MB/sec execute 138 sec latency 14660.062 ms oleg414-client.virtnet: 1 32094 0.92 MB/sec execute 139 sec latency 15660.184 ms oleg414-client.virtnet: 1 32094 0.91 MB/sec execute 140 sec latency 16660.419 ms oleg414-client.virtnet: 1 32094 0.90 MB/sec execute 141 sec latency 17660.578 ms oleg414-client.virtnet: 1 32094 0.90 MB/sec execute 142 sec latency 18660.745 ms oleg414-client.virtnet: 1 32094 0.89 MB/sec execute 143 sec latency 19660.876 ms oleg414-client.virtnet: 1 32254 0.89 MB/sec execute 144 sec latency 19915.873 ms oleg414-client.virtnet: 1 32452 0.89 MB/sec execute 145 sec latency 24.117 ms oleg414-client.virtnet: 1 32668 0.88 MB/sec execute 146 sec latency 28.684 ms oleg414-client.virtnet: 1 32909 0.88 MB/sec execute 147 sec latency 24.984 ms oleg414-client.virtnet: 1 33192 0.88 MB/sec execute 148 sec latency 25.523 ms oleg414-client.virtnet: 1 33493 0.90 MB/sec execute 149 sec latency 21.915 ms oleg414-client.virtnet: 1 33666 0.89 MB/sec execute 150 sec latency 29.683 ms oleg414-client.virtnet: 1 33954 0.89 MB/sec execute 151 sec latency 25.000 ms oleg414-client.virtnet: 1 34255 0.89 MB/sec execute 152 sec latency 22.209 ms oleg414-clien17:09:41 (1746133781) shut down facet: mds2 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds2 to oleg414-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0001 17:09:56 (1746133796) targets are mounted 17:09:56 (1746133796) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3568808 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12064 3592992 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7161800 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 17:10:08 (1746133808) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 17:10:23 (1746133823) targets are mounted 17:10:23 (1746133823) facet_failover done t.virtnet: 1 34586 0.89 MB/sec execute 153 sec latency 19.526 ms oleg414-client.virtnet: 1 35326 0.93 MB/sec execute 154 sec latency 16.479 ms oleg414-client.virtnet: 1 35681 0.95 MB/sec execute 155 sec latency 20.528 ms oleg414-client.virtnet: 1 36013 0.95 MB/sec execute 156 sec latency 14.425 ms oleg414-client.virtnet: 1 36420 0.94 MB/sec execute 157 sec latency 16.944 ms oleg414-client.virtnet: 1 36775 0.95 MB/sec execute 158 sec latency 18.659 ms oleg414-client.virtnet: 1 37179 0.96 MB/sec execute 159 sec latency 17.448 ms oleg414-client.virtnet: 1 37553 0.96 MB/sec execute 160 sec latency 14.792 ms oleg414-client.virtnet: 1 37959 0.96 MB/sec execute 161 sec latency 16.588 ms oleg414-client.virtnet: 1 38673 0.99 MB/sec execute 162 sec latency 17.877 ms oleg414-client.virtnet: 1 39174 1.01 MB/sec execute 163 sec latency 17.239 ms oleg414-client.virtnet: 1 39405 1.02 MB/sec execute 164 sec latency 26.889 ms oleg414-client.virtnet: 1 39641 1.01 MB/sec execute 165 sec latency 23.160 ms oleg414-client.virtnet: 1 39956 1.01 MB/sec execute 166 sec latency 20.434 ms oleg414-client.virtnet: 1 40270 1.01 MB/sec execute 167 sec latency 20.245 ms oleg414-client.virtnet: 1 40715 1.02 MB/sec execute 168 sec latency 18.585 ms oleg414-client.virtnet: 1 40749 1.02 MB/sec execute 169 sec latency 912.623 ms oleg414-client.virtnet: 1 40749 1.01 MB/sec execute 170 sec latency 1912.813 ms oleg414-client.virtnet: 1 40749 1.01 MB/sec execute 171 sec latency 2912.986 ms oleg414-client.virtnet: 1 40861 1.00 MB/sec execute 172 sec latency 3423.923 ms oleg414-client.virtnet: 1 41172 1.00 MB/sec execute 173 sec latency 22.643 ms oleg414-client.virtnet: 1 41538 1.00 MB/sec execute 174 sec latency 28.345 ms oleg414-client.virtnet: 1 42405 1.04 MB/sec execute 175 sec latency 13.667 ms oleg414-client.virtnet: 1 42856 1.05 MB/sec execute 176 sec latency 16.354 ms oleg414-client.virtnet: 1 43056 1.05 MB/sec execute 177 sec latency 40.193 ms oleg414-client.virtnet: 1 43349 1.05 MB/sec execute 178 sec latency 22.716 ms oleg414-client.virtnet: 1 43741 1.05 MB/sec execute 179 sec latency 16.948 ms oleg414-client.virtnet: 1 44156 1.06 MB/sec execute 180 sec latency 104.341 ms oleg414-client.virtnet: 1 44156 1.06 MB/sec execute 181 sec latency 1104.558 ms oleg414-client.virtnet: 1 44156 1.05 MB/sec execute 182 sec latency 2104.719 ms oleg414-client.virtnet: 1 44156 1.04 MB/sec execute 183 sec latency 3104.883 ms oleg414-client.virtnet: 1 44156 1.04 MB/sec execute 184 sec latency 4105.052 ms oleg414-client.virtnet: 1 44156 1.03 MB/sec execute 185 sec latency 5105.254 ms oleg414-client.virtnet: 1 44156 1.03 MB/sec execute 186 sec latency 6105.498 ms oleg414-client.virtnet: 1 44156 1.02 MB/sec execute 187 sec latency 7105.671 ms oleg414-client.virtnet: 1 44156 1.02 MB/sec execute 188 sec latency 8105.947 ms oleg414-client.virtnet: 1 44156 1.01 MB/sec execute 189 sec latency 9106.128 ms oleg414-client.virtnet: 1 44156 1.01 MB/sec execute 190 sec latency 10106.634 ms oleg414-client.virtnet: 1 44156 1.00 MB/sec execute 191 sec latency 11107.012 ms oleg414-client.virtnet: 1 44156 0.99 MB/sec execute 192 sec latency 12107.201 ms oleg414-client.virtnet: 1 44156 0.99 MB/sec execute 193 sec latency 13107.344 ms oleg414-client.virtnet: 1 44156 0.98 MB/sec execute 194 sec latency 14107.551 ms oleg414-client.virtnet: 1 44156 0.98 MB/sec execute 195 sec latency 15108.075 ms oleg414-client.virtnet: 1 44156 0.97 MB/sec execute 196 sec latency 16108.373 ms oleg414-client.virtnet: 1 44156 0.97 MB/sec exoleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3568996 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12068 3593860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49504 7162856 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg414-server Stopping /mnt/lustre-mds2 (opts:) on oleg414-server 17:10:37 (1746133837) shut down facet: mds2 facet_host: oleg414-server facet_failover_host: oleg414-server Failover mds2 to oleg414-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0001 17:10:51 (1746133851) targets are mounted 17:10:51 (1746133851) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37428 3569004 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12068 3593680 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7162684 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 17:11:04 (1746133864) shut down facet: mds1 facet_host: oleg414-server facet_failover_host: oleg414-server ecute 197 sec latency 17108.744 ms oleg414-client.virtnet: 1 44156 0.96 MB/sec execute 198 sec latency 18109.579 ms oleg414-client.virtnet: 1 44156 0.96 MB/sec execute 199 sec latency 19109.745 ms oleg414-client.virtnet: 1 44324 0.96 MB/sec execute 200 sec latency 19414.738 ms oleg414-client.virtnet: 1 44611 0.95 MB/sec execute 201 sec latency 22.365 ms oleg414-client.virtnet: 1 44972 0.95 MB/sec execute 202 sec latency 19.437 ms oleg414-client.virtnet: 1 45493 0.97 MB/sec execute 203 sec latency 21.359 ms oleg414-client.virtnet: 1 45975 0.98 MB/sec execute 204 sec latency 28.711 ms oleg414-client.virtnet: 1 46283 0.99 MB/sec execute 205 sec latency 30.670 ms oleg414-client.virtnet: 1 46467 0.99 MB/sec execute 206 sec latency 41.280 ms oleg414-client.virtnet: 1 46667 0.99 MB/sec execute 207 sec latency 24.488 ms oleg414-client.virtnet: 1 46935 0.99 MB/sec execute 208 sec latency 40.825 ms oleg414-client.virtnet: 1 47206 0.99 MB/sec execute 209 sec latency 69.392 ms oleg414-client.virtnet: 1 47560 1.00 MB/sec execute 210 sec latency 25.284 ms oleg414-client.virtnet: 1 47862 1.00 MB/sec execute 211 sec latency 17.973 ms oleg414-client.virtnet: 1 48173 0.99 MB/sec execute 212 sec latency 20.870 ms oleg414-client.virtnet: 1 48596 1.00 MB/sec execute 213 sec latency 15.390 ms oleg414-client.virtnet: 1 49340 1.02 MB/sec execute 214 sec latency 17.062 ms oleg414-client.virtnet: 1 49891 1.04 MB/sec execute 215 sec latency 16.974 ms oleg414-client.virtnet: 1 50224 1.04 MB/sec execute 216 sec latency 13.612 ms oleg414-client.virtnet: 1 50686 1.04 MB/sec execute 217 sec latency 12.168 ms oleg414-client.virtnet: 1 51199 1.05 MB/sec execute 218 sec latency 13.205 ms oleg414-client.virtnet: 1 51558 1.05 MB/sec execute 219 sec latency 14.746 ms oleg414-client.virtnet: 1 52026 1.05 MB/sec execute 220 sec latency 18.071 ms oleg414-client.virtnet: 1 52583 1.07 MB/sec execute 221 sec latency 22.729 ms oleg414-client.virtnet: 1 53265 1.09 MB/sec execute 222 sec latency 16.276 ms oleg414-client.virtnet: 1 53606 1.09 MB/sec execute 223 sec latency 13.761 ms oleg414-client.virtnet: 1 53622 1.09 MB/sec execute 224 sec latency 959.411 ms oleg414-client.virtnet: 1 53622 1.08 MB/sec execute 225 sec latency 1959.618 ms oleg414-client.virtnet: 1 53622 1.08 MB/sec execute 226 sec latency 2959.859 ms oleg414-client.virtnet: 1 53622 1.07 MB/sec execute 227 sec latency 3960.277 ms oleg414-client.virtnet: 1 53885 1.07 MB/sec execute 228 sec latency 4084.619 ms oleg414-client.virtnet: 1 54194 1.07 MB/sec execute 229 sec latency 17.409 ms oleg414-client.virtnet: 1 54623 1.08 MB/sec execute 230 sec latency 31.380 ms oleg414-client.virtnet: 1 54953 1.08 MB/sec execute 231 sec latency 16.779 ms oleg414-client.virtnet: 1 55252 1.07 MB/sec execute 232 sec latency 20.918 ms oleg414-client.virtnet: 1 55583 1.07 MB/sec execute 233 sec latency 188.415 ms oleg414-client.virtnet: 1 56004 1.09 MB/sec execute 234 sec latency 228.098 ms oleg414-client.virtnet: 1 56467 1.09 MB/sec execute 235 sec latency 34.556 ms oleg414-client.virtnet: 1 56802 1.10 MB/sec execute 236 sec latency 338.671 ms oleg414-client.virtnet: 1 56802 1.10 MB/sec execute 237 sec latency 1339.328 ms oleg414-client.virtnet: 1 56802 1.10 MB/sec execute 238 sec latency 2339.498 ms oleg414-client.virtnet: 1 56802 1.09 MB/sec execute 239 sec latency 3339.750 ms oleg414-client.virtnet: 1 56802 1.09 MB/sec execute 240 sec latency 4339.931 ms oleg414-client.virtnet: 1 56802 1.08 MB/sec execute 241 sec latency 5340.124 ms oleg414-client.virtnet: Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 17:11:21 (1746133881) targets are mounted 17:11:21 (1746133881) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 56802 1.08 MB/sec execute 242 sec latency 6341.741 ms oleg414-client.virtnet: 1 56802 1.07 MB/sec execute 243 sec latency 7343.506 ms oleg414-client.virtnet: 1 56802 1.07 MB/sec execute 244 sec latency 8343.674 ms oleg414-client.virtnet: 1 56802 1.06 MB/sec execute 245 sec latency 9343.887 ms oleg414-client.virtnet: 1 56802 1.06 MB/sec execute 246 sec latency 10344.096 ms oleg414-client.virtnet: 1 56802 1.06 MB/sec execute 247 sec latency 11344.250 ms oleg414-client.virtnet: 1 56802 1.05 MB/sec execute 248 sec latency 12344.427 ms oleg414-client.virtnet: 1 56802 1.05 MB/sec execute 249 sec latency 13345.922 ms oleg414-client.virtnet: 1 56802 1.04 MB/sec execute 250 sec latency 14346.284 ms oleg414-client.virtnet: 1 56802 1.04 MB/sec execute 251 sec latency 15346.521 ms oleg414-client.virtnet: 1 56802 1.03 MB/sec execute 252 sec latency 16347.075 ms oleg414-client.virtnet: 1 56802 1.03 MB/sec execute 253 sec latency 17349.164 ms oleg414-client.virtnet: 1 56802 1.03 MB/sec execute 254 sec latency 18349.450 ms oleg414-client.virtnet: 1 56802 1.02 MB/sec execute 255 sec latency 19349.785 ms oleg414-client.virtnet: 1 56806 1.02 MB/sec execute 256 sec latency 20301.112 ms oleg414-client.virtnet: 1 57025 1.02 MB/sec execute 257 sec latency 35.674 ms oleg414-client.virtnet: 1 57144 1.02 MB/sec execute 258 sec latency 67.093 ms oleg414-client.virtnet: 1 57303 1.01 MB/sec execute 259 sec latency 31.870 ms oleg414-client.virtnet: 1 57484 1.01 MB/sec execute 260 sec latency 44.131 ms oleg414-client.virtnet: 1 57669 1.01 MB/sec execute 261 sec latency 29.539 ms oleg414-client.virtnet: 1 57888 1.01 MB/sec execute 262 sec latency 34.878 ms oleg414-client.virtnet: 1 58213 1.02 MB/sec execute 263 sec latency 22.977 ms oleg414-client.virtnet: 1 58511 1.01 MB/sec execute 264 sec latency 16.870 ms oleg414-client.virtnet: 1 58846 1.01 MB/sec execute 265 sec latency 16.354 ms oleg414-client.virtnet: 1 59215 1.01 MB/sec execute 266 sec latency 21.044 ms oleg414-client.virtnet: 1 59758 1.03 MB/sec execute 267 sec latency 24.151 ms oleg414-client.virtnet: 1 60354 1.04 MB/sec execute 268 sec latency 23.775 ms oleg414-client.virtnet: 1 60619 1.05 MB/sec execute 269 sec latency 18.193 ms oleg414-client.virtnet: 1 60853 1.04 MB/sec execute 270 sec latency 18.407 ms oleg414-client.virtnet: 1 61146 1.04 MB/sec execute 271 sec latency 28.206 ms oleg414-client.virtnet: 1 61404 1.04 MB/sec execute 272 sec latency 27.941 ms oleg414-client.virtnet: 1 61754 1.05 MB/sec execute 273 sec latency 18.031 ms oleg414-client.virtnet: 1 62019 1.05 MB/sec execute 274 sec latency 31.208 ms oleg414-client.virtnet: 1 62300 1.04 MB/sec execute 275 sec latency 21.215 ms oleg414-client.virtnet: 1 62678 1.04 MB/sec execute 276 sec latency 18.995 ms oleg414-client.virtnet: 1 62925 1.04 MB/sec execute 277 sec latency 30.125 ms oleg414-client.virtnet: 1 63637 1.06 MB/sec execute 278 sec latency 20.242 ms oleg414-client.virtnet: 1 64083 1.08 MB/sec execute 279 sec latency 20.583 ms oleg414-client.virtnet: 1 64301 1.07 MB/sec execute 280 sec latency 27.000 ms oleg414-client.virtnet: 1 64560 1.07 MB/sec execute 281 sec latency 26.848 ms oleg414-client.virtnet: 1 64767 1.07 MB/sec execute 282 sec latency 27.328 ms oleg414-client.virtnet: 1 65004 1.07 MB/sec execute 283 sec latency 24.578 ms oleg414-client.virtnet: 1 65306 1.08 MB/sec execute 284 sec latency 23.323 ms oleg414-client.virtnet: 1 65563 1.07 MB/sec execute 285 sec latency 20.772 ms oleg414-client.virtnet: 1 65860 1.07 MB/sec execute 286 sec latency 19.129 ms oleg414-client.virtnet: 1 66217 1.07 MB/sec execute 287 sec latency 25.889 ms oleg414-client.virtnet: 1 66639 1.08 MB/sec execute 288 sec latency 22.866 ms oleg414-client.virtnet: 1 67278 1.09 MB/sec execute 289 sec latency 19.360 ms oleg414-client.virtnet: 1 67590 1.10 MB/sec execute 290 sec latency 23.180 ms oleg414-client.virtnet: 1 67768 1.10 MB/sec execute 291 sec latency 35.247 ms oleg414-client.virtnet: 1 67960 1.09 MB/sec execute 292 sec latency 25.230 ms oleg414-client.virtnet: 1 68197 1.09 MB/sec execute 293 sec latency 31.219 ms oleg414-client.virtnet: 1 68379 1.09 MB/sec execute 294 sec latency 30.813 ms oleg414-client.virtnet: 1 68608 1.09 MB/sec execute 295 sec latency 21.218 ms oleg414-client.virtnet: 1 68915 1.10 MB/sec execute 296 sec latency 29.396 ms oleg414-client.virtnet: 1 69118 1.09 MB/sec execute 297 sec latency 25.611 ms oleg414-client.virtnet: 1 69297 1.09 MB/sec execute 298 sec latency 28.469 ms oleg414-client.virtnet: 1 69591 1.09 MB/sec execute 299 sec latency 22.250 ms oleg414-client.virtnet: 1 cleanup 300 sec oleg414-client.virtnet: 0 cleanup 301 sec oleg414-client.virtnet: oleg414-client.virtnet: Operation Count AvgLat MaxLat oleg414-client.virtnet: ---------------------------------------- oleg414-client.virtnet: NTCreateX 10562 17.008 20301.095 oleg414-client.virtnet: Close 7770 1.947 17.701 oleg414-client.virtnet: Rename 445 60.087 20371.326 oleg414-client.virtnet: Unlink 2119 3.354 21.536 oleg414-client.virtnet: Qpathinfo 9606 4.487 19414.721 oleg414-client.virtnet: Qfileinfo 1675 0.203 3.865 oleg414-client.virtnet: Qfsinfo 1742 0.313 7.566 oleg414-client.virtnet: Sfileinfo 860 9.109 38.548 oleg414-client.virtnet: Find 3700 0.551 31.818 oleg414-client.virtnet: WriteX 5221 1.457 22.730 oleg414-client.virtnet: ReadX 16477 0.030 21.070 oleg414-client.virtnet: LockX 34 1.776 3.598 oleg414-client.virtnet: UnlockX 34 1.711 5.875 oleg414-client.virtnet: Flush 737 11.567 539.092 oleg414-client.virtnet: oleg414-client.virtnet: Throughput 1.08838 MB/sec 1 clients 1 procs max_latency=20371.343 ms oleg414-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg414-client.virtnet at Thu May 1 17:12:09 EDT 2025 with return code 0 oleg414-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg414-client.virtnet oleg414-client.virtnet: /mnt/lustre/d70b.replay-single/oleg414-client.virtnet /mnt/lustre/d70b.replay-single/oleg414-client.virtnet oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg414-client.virtnet: removed directory: 'clients/client0' oleg414-client.virtnet: removed directory: 'clients' oleg414-client.virtnet: removed 'client.txt' oleg414-client.virtnet: /mnt/lustre/d70b.replay-single/oleg414-client.virtnet oleg414-client.virtnet: dbench successfully finished