== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 20:04:28 (1746144268) Starting client oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre Started clients oleg443-client.virtnet: 192.168.204.143@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg443-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found dbench: no process found Started rundbench load pid=14106 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2508 1285180 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26148 3580872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7186316 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg443-server Stopping /mnt/lustre-mds1 (opts:) on oleg443-server 20:04:41 (1746144281) shut down facet: mds1 facet_host: oleg443-server facet_failover_host: oleg443-server Failover mds1 to oleg443-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 20:04:59 (1746144299) targets are mounted 20:04:59 (1746144299) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg443-client.virtnet: looking for dbench program oleg443-client.virtnet: /usr/bin/dbench oleg443-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg443-client.virtnet oleg443-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg443-client.virtnet' oleg443-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg443-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg443-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg443-client.virtnet at Thu May 1 20:04:31 EDT 2025 oleg443-client.virtnet: waiting for dbench pid 14137 oleg443-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg443-client.virtnet: oleg443-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg443-client.virtnet: failed to create barrier semaphore oleg443-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg443-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg443-client.virtnet: releasing clients oleg443-client.virtnet: 1 157 6.54 MB/sec warmup 1 sec latency 48.458 ms oleg443-client.virtnet: 1 247 4.70 MB/sec warmup 2 sec latency 94.837 ms oleg443-client.virtnet: 1 405 4.78 MB/sec warmup 3 sec latency 50.597 ms oleg443-client.virtnet: 1 611 5.24 MB/sec warmup 4 sec latency 67.244 ms oleg443-client.virtnet: 1 740 4.26 MB/sec warmup 5 sec latency 57.924 ms oleg443-client.virtnet: 1 863 3.59 MB/sec warmup 6 sec latency 56.509 ms oleg443-client.virtnet: 1 930 3.11 MB/sec warmup 7 sec latency 338.953 ms oleg443-client.virtnet: 1 1145 2.87 MB/sec warmup 8 sec latency 29.735 ms oleg443-client.virtnet: 1 1252 2.56 MB/sec warmup 9 sec latency 47.838 ms oleg443-client.virtnet: 1 1428 2.60 MB/sec warmup 10 sec latency 360.181 ms oleg443-client.virtnet: 1 1428 2.36 MB/sec warmup 11 sec latency 1367.322 ms oleg443-client.virtnet: 1 1428 2.17 MB/sec warmup 12 sec latency 2367.478 ms oleg443-client.virtnet: 1 1428 2.00 MB/sec warmup 13 sec latency 3368.699 ms oleg443-client.virtnet: 1 1428 1.86 MB/sec warmup 14 sec latency 4368.868 ms oleg443-client.virtnet: 1 1428 1.73 MB/sec warmup 15 sec latency 5369.082 ms oleg443-client.virtnet: 1 1428 1.62 MB/sec warmup 16 sec latency 6371.007 ms oleg443-client.virtnet: 1 1428 1.53 MB/sec warmup 17 sec latency 7371.165 ms oleg443-client.virtnet: 1 1428 1.44 MB/sec warmup 18 sec latency 8371.319 ms oleg443-client.virtnet: 1 1428 1.37 MB/sec warmup 19 sec latency 9371.933 ms oleg443-client.virtnet: 1 1428 1.30 MB/sec warmup 20 sec latency 10372.620 ms oleg443-client.virtnet: 1 1428 1.24 MB/sec warmup 21 sec latency 11372.814 ms oleg443-client.virtnet: 1 1428 1.18 MB/sec warmup 22 sec latency 12373.516 ms oleg443-client.virtnet: 1 1428 1.13 MB/sec warmup 23 sec latency 13373.685 ms oleg443-client.virtnet: 1 1428 1.08 MB/sec warmup 24 sec latency 14376.331 ms oleg443-client.virtnet: 1 1428 1.04 MB/sec warmup 25 sec latency 15377.778 ms oleg443-client.virtnet: 1 1428 1.00 MB/sec warmup 26 sec latency 16378.512 ms oleg443-client.virtnet: 1 1428 0.96 MB/sec warmup 27 sec latency 17381.712 ms oleg443-client.virtnet: 1 1428 0.93 MB/sec warmup 28 sec latency 18381.862 ms oleg443-client.virtnet: 1 1428 0.90 MB/sec warmup 29 sec latency 19382.065 ms oleg443-client.virtnet: 1 1428 0.87 MB/sec warmup 30 sec latency 20382.278 ms oleg443-client.virtnet: 1 1428 0.84 MB/sec warmup 31 sec latency 21384.133 ms oleg443-client.virtnet: 1 1502 0.81 MB/sec warmup 32 sec latency 21708.243 ms oleg443-client.virtnet: 1 1656 0.79 MB/sec warmup 33 sec mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1748 3594096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 27184 3568840 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28932 7162936 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg443-server Stopping /mnt/lustre-mds2 (opts:) on oleg443-server 20:05:15 (1746144315) shut down facet: mds2 facet_host: oleg443-server facet_failover_host: oleg443-server Failover mds2 to oleg443-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0001 20:05:31 (1746144331) targets are mounted 20:05:31 (1746144331) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11912 3593532 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37576 3569360 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7162892 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg443-server Stopping /mnt/lustre-mds1 (opts:) on oleg443-server 20:05:45 (1746144345) shut down facet: mds1 facet_host: oleg443-server facet_failover_host: oleg443-server latency 54.632 ms oleg443-client.virtnet: 1 1828 0.77 MB/sec warmup 34 sec latency 47.030 ms oleg443-client.virtnet: 1 2020 0.76 MB/sec warmup 35 sec latency 54.398 ms oleg443-client.virtnet: 1 2209 0.74 MB/sec warmup 36 sec latency 38.837 ms oleg443-client.virtnet: 1 2406 0.76 MB/sec warmup 37 sec latency 72.214 ms oleg443-client.virtnet: 1 2568 0.76 MB/sec warmup 38 sec latency 52.341 ms oleg443-client.virtnet: 1 2876 0.82 MB/sec warmup 39 sec latency 64.651 ms oleg443-client.virtnet: 1 3019 0.84 MB/sec warmup 40 sec latency 46.503 ms oleg443-client.virtnet: 1 3432 0.89 MB/sec warmup 41 sec latency 33.715 ms oleg443-client.virtnet: 1 3663 0.92 MB/sec warmup 42 sec latency 39.305 ms oleg443-client.virtnet: 1 3819 0.92 MB/sec warmup 43 sec latency 31.824 ms oleg443-client.virtnet: 1 3948 0.91 MB/sec warmup 44 sec latency 50.294 ms oleg443-client.virtnet: 1 4083 0.89 MB/sec warmup 45 sec latency 193.140 ms oleg443-client.virtnet: 1 4083 0.87 MB/sec warmup 46 sec latency 1193.363 ms oleg443-client.virtnet: 1 4083 0.85 MB/sec warmup 47 sec latency 2193.717 ms oleg443-client.virtnet: 1 4083 0.83 MB/sec warmup 48 sec latency 3195.758 ms oleg443-client.virtnet: 1 4083 0.82 MB/sec warmup 49 sec latency 4196.233 ms oleg443-client.virtnet: 1 4083 0.80 MB/sec warmup 50 sec latency 5196.476 ms oleg443-client.virtnet: 1 4083 0.79 MB/sec warmup 51 sec latency 6198.300 ms oleg443-client.virtnet: 1 4083 0.77 MB/sec warmup 52 sec latency 7198.513 ms oleg443-client.virtnet: 1 4083 0.76 MB/sec warmup 53 sec latency 8198.746 ms oleg443-client.virtnet: 1 4083 0.74 MB/sec warmup 54 sec latency 9199.443 ms oleg443-client.virtnet: 1 4083 0.73 MB/sec warmup 55 sec latency 10199.592 ms oleg443-client.virtnet: 1 4083 0.72 MB/sec warmup 56 sec latency 11199.730 ms oleg443-client.virtnet: 1 4083 0.70 MB/sec warmup 57 sec latency 12200.815 ms oleg443-client.virtnet: 1 4083 0.69 MB/sec warmup 58 sec latency 13201.447 ms oleg443-client.virtnet: 1 4083 0.68 MB/sec warmup 59 sec latency 14201.569 ms oleg443-client.virtnet: 1 4083 0.00 MB/sec execute 1 sec latency 16202.174 ms oleg443-client.virtnet: 1 4083 0.00 MB/sec execute 2 sec latency 17203.349 ms oleg443-client.virtnet: 1 4083 0.00 MB/sec execute 3 sec latency 18203.530 ms oleg443-client.virtnet: 1 4226 0.02 MB/sec execute 4 sec latency 18677.048 ms oleg443-client.virtnet: 1 4386 0.10 MB/sec execute 5 sec latency 37.838 ms oleg443-client.virtnet: 1 4628 0.32 MB/sec execute 6 sec latency 24.822 ms oleg443-client.virtnet: 1 4840 0.31 MB/sec execute 7 sec latency 27.277 ms oleg443-client.virtnet: 1 5114 0.64 MB/sec execute 8 sec latency 30.427 ms oleg443-client.virtnet: 1 5252 0.58 MB/sec execute 9 sec latency 50.021 ms oleg443-client.virtnet: 1 5397 0.54 MB/sec execute 10 sec latency 35.388 ms oleg443-client.virtnet: 1 5549 0.50 MB/sec execute 11 sec latency 265.094 ms oleg443-client.virtnet: 1 5762 0.48 MB/sec execute 12 sec latency 27.970 ms oleg443-client.virtnet: 1 5976 0.55 MB/sec execute 13 sec latency 50.673 ms oleg443-client.virtnet: 1 6113 0.57 MB/sec execute 14 sec latency 407.860 ms oleg443-client.virtnet: 1 6113 0.53 MB/sec execute 15 sec latency 1408.368 ms oleg443-client.virtnet: 1 6113 0.50 MB/sec execute 16 sec latency 2408.638 ms oleg443-client.virtnet: 1 6113 0.47 MB/sec execute 17 sec latency 3409.346 ms oleg443-client.virtnet: 1 6113 0.44 MB/sec execute 18 sec latency 4409.728 ms oleg443-client.virtnet: 1 6113 Failover mds1 to oleg443-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 20:06:02 (1746144362) targets are mounted 20:06:02 (1746144362) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12036 3594044 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37576 3565924 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7159968 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg443-server Stopping /mnt/lustre-mds2 (opts:) on oleg443-server 20:06:16 (1746144376) shut down facet: mds2 facet_host: oleg443-server facet_failover_host: oleg443-server Failover mds2 to oleg443-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0001 20:06:32 (1746144392) targets are mounted 20:06:32 (1746144392) facet_failover done 0.42 MB/sec execute 19 sec latency 5410.021 ms oleg443-client.virtnet: 1 6113 0.40 MB/sec execute 20 sec latency 6410.319 ms oleg443-client.virtnet: 1 6113 0.38 MB/sec execute 21 sec latency 7412.191 ms oleg443-client.virtnet: 1 6113 0.36 MB/sec execute 22 sec latency 8412.353 ms oleg443-client.virtnet: 1 6113 0.34 MB/sec execute 23 sec latency 9415.212 ms oleg443-client.virtnet: 1 6113 0.33 MB/sec execute 24 sec latency 10415.414 ms oleg443-client.virtnet: 1 6113 0.32 MB/sec execute 25 sec latency 11415.534 ms oleg443-client.virtnet: 1 6113 0.31 MB/sec execute 26 sec latency 12416.912 ms oleg443-client.virtnet: 1 6113 0.29 MB/sec execute 27 sec latency 13417.042 ms oleg443-client.virtnet: 1 6113 0.28 MB/sec execute 28 sec latency 14417.323 ms oleg443-client.virtnet: 1 6113 0.27 MB/sec execute 29 sec latency 15418.227 ms oleg443-client.virtnet: 1 6113 0.26 MB/sec execute 30 sec latency 16418.435 ms oleg443-client.virtnet: 1 6113 0.26 MB/sec execute 31 sec latency 17418.593 ms oleg443-client.virtnet: 1 6113 0.25 MB/sec execute 32 sec latency 18419.855 ms oleg443-client.virtnet: 1 6113 0.24 MB/sec execute 33 sec latency 19420.297 ms oleg443-client.virtnet: 1 6113 0.23 MB/sec execute 34 sec latency 20420.818 ms oleg443-client.virtnet: 1 6213 0.26 MB/sec execute 35 sec latency 20711.124 ms oleg443-client.virtnet: 1 6552 0.35 MB/sec execute 36 sec latency 32.996 ms oleg443-client.virtnet: 1 6933 0.39 MB/sec execute 37 sec latency 47.716 ms oleg443-client.virtnet: 1 7211 0.47 MB/sec execute 38 sec latency 37.017 ms oleg443-client.virtnet: 1 7404 0.48 MB/sec execute 39 sec latency 24.075 ms oleg443-client.virtnet: 1 7539 0.48 MB/sec execute 40 sec latency 41.170 ms oleg443-client.virtnet: 1 7710 0.47 MB/sec execute 41 sec latency 35.143 ms oleg443-client.virtnet: 1 7902 0.47 MB/sec execute 42 sec latency 26.856 ms oleg443-client.virtnet: 1 8085 0.46 MB/sec execute 43 sec latency 27.773 ms oleg443-client.virtnet: 1 8298 0.48 MB/sec execute 44 sec latency 35.637 ms oleg443-client.virtnet: 1 8531 0.54 MB/sec execute 45 sec latency 39.163 ms oleg443-client.virtnet: 1 8690 0.53 MB/sec execute 46 sec latency 33.296 ms oleg443-client.virtnet: 1 8889 0.52 MB/sec execute 47 sec latency 28.936 ms oleg443-client.virtnet: 1 9131 0.51 MB/sec execute 48 sec latency 23.614 ms oleg443-client.virtnet: 1 9481 0.53 MB/sec execute 49 sec latency 20.203 ms oleg443-client.virtnet: 1 9861 0.60 MB/sec execute 50 sec latency 23.849 ms oleg443-client.virtnet: 1 10311 0.64 MB/sec execute 51 sec latency 27.534 ms oleg443-client.virtnet: 1 10677 0.70 MB/sec execute 52 sec latency 28.286 ms oleg443-client.virtnet: 1 10903 0.72 MB/sec execute 53 sec latency 24.164 ms oleg443-client.virtnet: 1 11075 0.71 MB/sec execute 54 sec latency 26.284 ms oleg443-client.virtnet: 1 11325 0.70 MB/sec execute 55 sec latency 24.710 ms oleg443-client.virtnet: 1 11541 0.69 MB/sec execute 56 sec latency 27.908 ms oleg443-client.virtnet: 1 11743 0.70 MB/sec execute 57 sec latency 26.164 ms oleg443-client.virtnet: 1 11918 0.70 MB/sec execute 58 sec latency 57.901 ms oleg443-client.virtnet: 1 12152 0.73 MB/sec execute 59 sec latency 38.456 ms oleg443-client.virtnet: 1 12344 0.72 MB/sec execute 60 sec latency 26.839 ms oleg443-client.virtnet: 1 12542 0.72 MB/sec execute 61 sec latency 25.386 ms oleg443-client.virtnet: 1 12644 0.71 MB/sec execute 62 sec latency 739.937 ms oleg443-client.virtnet: 1 12644 0.69 MB/sec execute 63 sec latency 1740.31oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12036 3593992 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37576 3566516 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7160508 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg443-server Stopping /mnt/lustre-mds1 (opts:) on oleg443-server 20:06:45 (1746144405) shut down facet: mds1 facet_host: oleg443-server facet_failover_host: oleg443-server Failover mds1 to oleg443-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 20:07:03 (1746144423) targets are mounted 20:07:03 (1746144423) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11912 3594200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37596 3569292 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49508 7163492 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg443-server Stopping /mnt/lustre-mds2 (opts:) on oleg443-server 20:07:19 (1746144439) shut down facet: mds2 facet_host: oleg443-server facet_failover_host: oleg443-server 9 ms oleg443-client.virtnet: 1 12644 0.68 MB/sec execute 64 sec latency 2740.767 ms oleg443-client.virtnet: 1 12689 0.67 MB/sec execute 65 sec latency 3510.757 ms oleg443-client.virtnet: 1 13010 0.68 MB/sec execute 66 sec latency 20.812 ms oleg443-client.virtnet: 1 13419 0.73 MB/sec execute 67 sec latency 23.408 ms oleg443-client.virtnet: 1 13998 0.77 MB/sec execute 68 sec latency 20.536 ms oleg443-client.virtnet: 1 14317 0.81 MB/sec execute 69 sec latency 21.168 ms oleg443-client.virtnet: 1 14492 0.81 MB/sec execute 70 sec latency 30.848 ms oleg443-client.virtnet: 1 14630 0.81 MB/sec execute 71 sec latency 40.961 ms oleg443-client.virtnet: 1 14872 0.80 MB/sec execute 72 sec latency 37.304 ms oleg443-client.virtnet: 1 15122 0.80 MB/sec execute 73 sec latency 19.826 ms oleg443-client.virtnet: 1 15375 0.80 MB/sec execute 74 sec latency 50.979 ms oleg443-client.virtnet: 1 15375 0.79 MB/sec execute 75 sec latency 1051.333 ms oleg443-client.virtnet: 1 15375 0.78 MB/sec execute 76 sec latency 2051.596 ms oleg443-client.virtnet: 1 15375 0.77 MB/sec execute 77 sec latency 3051.782 ms oleg443-client.virtnet: 1 15375 0.76 MB/sec execute 78 sec latency 4051.974 ms oleg443-client.virtnet: 1 15375 0.75 MB/sec execute 79 sec latency 5052.124 ms oleg443-client.virtnet: 1 15375 0.74 MB/sec execute 80 sec latency 6052.766 ms oleg443-client.virtnet: 1 15375 0.73 MB/sec execute 81 sec latency 7052.928 ms oleg443-client.virtnet: 1 15375 0.72 MB/sec execute 82 sec latency 8054.722 ms oleg443-client.virtnet: 1 15375 0.72 MB/sec execute 83 sec latency 9054.914 ms oleg443-client.virtnet: 1 15375 0.71 MB/sec execute 84 sec latency 10055.072 ms oleg443-client.virtnet: 1 15375 0.70 MB/sec execute 85 sec latency 11055.740 ms oleg443-client.virtnet: 1 15375 0.69 MB/sec execute 86 sec latency 12056.462 ms oleg443-client.virtnet: 1 15375 0.68 MB/sec execute 87 sec latency 13059.069 ms oleg443-client.virtnet: 1 15375 0.67 MB/sec execute 88 sec latency 14059.310 ms oleg443-client.virtnet: 1 15375 0.67 MB/sec execute 89 sec latency 15059.646 ms oleg443-client.virtnet: 1 15375 0.66 MB/sec execute 90 sec latency 16061.348 ms oleg443-client.virtnet: 1 15375 0.65 MB/sec execute 91 sec latency 17061.474 ms oleg443-client.virtnet: 1 15375 0.65 MB/sec execute 92 sec latency 18061.778 ms oleg443-client.virtnet: 1 15375 0.64 MB/sec execute 93 sec latency 19062.119 ms oleg443-client.virtnet: 1 15375 0.63 MB/sec execute 94 sec latency 20062.990 ms oleg443-client.virtnet: 1 15375 0.62 MB/sec execute 95 sec latency 21073.784 ms oleg443-client.virtnet: 1 15460 0.62 MB/sec execute 96 sec latency 21427.795 ms oleg443-client.virtnet: 1 15664 0.64 MB/sec execute 97 sec latency 66.286 ms oleg443-client.virtnet: 1 15804 0.64 MB/sec execute 98 sec latency 64.497 ms oleg443-client.virtnet: 1 15978 0.63 MB/sec execute 99 sec latency 31.741 ms oleg443-client.virtnet: 1 16181 0.63 MB/sec execute 100 sec latency 29.526 ms oleg443-client.virtnet: 1 16310 0.62 MB/sec execute 101 sec latency 41.052 ms oleg443-client.virtnet: 1 16423 0.62 MB/sec execute 102 sec latency 73.730 ms oleg443-client.virtnet: 1 16619 0.63 MB/sec execute 103 sec latency 32.865 ms oleg443-client.virtnet: 1 16771 0.63 MB/sec execute 104 sec latency 56.822 ms oleg443-client.virtnet: 1 17135 0.67 MB/sec execute 105 sec latency 57.317 ms oleg443-client.virtnet: 1 17402 0.67 MB/sec execute 106 sec latency 60.466 ms oleg443-client.virtnet: 1 17700 0.70 MB/sec execute 107 sec latency 47.859 ms oleg443-client.virtnet: Failover mds2 to oleg443-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0001 20:07:36 (1746144456) targets are mounted 20:07:36 (1746144456) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12072 3594864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3569204 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7164068 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg443-server Stopping /mnt/lustre-mds1 (opts:) on oleg443-server 20:07:52 (1746144472) shut down facet: mds1 facet_host: oleg443-server facet_failover_host: oleg443-server Failover mds1 to oleg443-server mount facets: mds1 1 17861 0.70 MB/sec execute 108 sec latency 65.012 ms oleg443-client.virtnet: 1 18026 0.70 MB/sec execute 109 sec latency 31.840 ms oleg443-client.virtnet: 1 18189 0.69 MB/sec execute 110 sec latency 37.129 ms oleg443-client.virtnet: 1 18266 0.69 MB/sec execute 111 sec latency 372.825 ms oleg443-client.virtnet: 1 18266 0.68 MB/sec execute 112 sec latency 1372.985 ms oleg443-client.virtnet: 1 18266 0.68 MB/sec execute 113 sec latency 2373.415 ms oleg443-client.virtnet: 1 18266 0.67 MB/sec execute 114 sec latency 3373.560 ms oleg443-client.virtnet: 1 18266 0.66 MB/sec execute 115 sec latency 4373.779 ms oleg443-client.virtnet: 1 18266 0.66 MB/sec execute 116 sec latency 5375.308 ms oleg443-client.virtnet: 1 18266 0.65 MB/sec execute 117 sec latency 6375.661 ms oleg443-client.virtnet: 1 18266 0.65 MB/sec execute 118 sec latency 7375.821 ms oleg443-client.virtnet: 1 18266 0.64 MB/sec execute 119 sec latency 8376.107 ms oleg443-client.virtnet: 1 18266 0.64 MB/sec execute 120 sec latency 9376.278 ms oleg443-client.virtnet: 1 18266 0.63 MB/sec execute 121 sec latency 10376.454 ms oleg443-client.virtnet: 1 18266 0.63 MB/sec execute 122 sec latency 11378.996 ms oleg443-client.virtnet: 1 18266 0.62 MB/sec execute 123 sec latency 12379.879 ms oleg443-client.virtnet: 1 18266 0.62 MB/sec execute 124 sec latency 13380.071 ms oleg443-client.virtnet: 1 18266 0.61 MB/sec execute 125 sec latency 14380.763 ms oleg443-client.virtnet: 1 18266 0.61 MB/sec execute 126 sec latency 15381.575 ms oleg443-client.virtnet: 1 18266 0.60 MB/sec execute 127 sec latency 16381.775 ms oleg443-client.virtnet: 1 18338 0.60 MB/sec execute 128 sec latency 17011.102 ms oleg443-client.virtnet: 1 18467 0.59 MB/sec execute 129 sec latency 47.541 ms oleg443-client.virtnet: 1 18560 0.59 MB/sec execute 130 sec latency 62.314 ms oleg443-client.virtnet: 1 18689 0.59 MB/sec execute 131 sec latency 62.846 ms oleg443-client.virtnet: 1 18875 0.59 MB/sec execute 132 sec latency 42.536 ms oleg443-client.virtnet: 1 19021 0.59 MB/sec execute 133 sec latency 36.982 ms oleg443-client.virtnet: 1 19219 0.61 MB/sec execute 134 sec latency 64.516 ms oleg443-client.virtnet: 1 19355 0.60 MB/sec execute 135 sec latency 39.265 ms oleg443-client.virtnet: 1 19465 0.60 MB/sec execute 136 sec latency 82.721 ms oleg443-client.virtnet: 1 19557 0.60 MB/sec execute 137 sec latency 50.847 ms oleg443-client.virtnet: 1 19729 0.59 MB/sec execute 138 sec latency 36.996 ms oleg443-client.virtnet: 1 19855 0.59 MB/sec execute 139 sec latency 41.292 ms oleg443-client.virtnet: 1 19945 0.59 MB/sec execute 140 sec latency 344.563 ms oleg443-client.virtnet: 1 19945 0.58 MB/sec execute 141 sec latency 1344.709 ms oleg443-client.virtnet: 1 19945 0.58 MB/sec execute 142 sec latency 2344.818 ms oleg443-client.virtnet: 1 19945 0.58 MB/sec execute 143 sec latency 3345.130 ms oleg443-client.virtnet: 1 19945 0.57 MB/sec execute 144 sec latency 4347.003 ms oleg443-client.virtnet: 1 19945 0.57 MB/sec execute 145 sec latency 5347.183 ms oleg443-client.virtnet: 1 19945 0.56 MB/sec execute 146 sec latency 6348.761 ms oleg443-client.virtnet: 1 19945 0.56 MB/sec execute 147 sec latency 7349.022 ms oleg443-client.virtnet: 1 19945 0.56 MB/sec execute 148 sec latency 8349.441 ms oleg443-client.virtnet: 1 19945 0.55 MB/sec execute 149 sec latency 9350.078 ms oleg443-client.virtnet: 1 19945 0.55 MB/sec execute 150 sec latency 10350.279 ms oleg443-client.virtnet: 1 19945 0.54 MB/sec execute 151 sec latency 11352.353 ms oleg443-client.virtnet: 1 19945 0.54 MBStarting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 20:08:09 (1746144489) targets are mounted 20:08:09 (1746144489) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12232 3593708 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37428 3566832 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49660 7160540 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg443-server Stopping /mnt/lustre-mds2 (opts:) on oleg443-server 20:08:24 (1746144504) shut down facet: mds2 facet_host: oleg443-server facet_failover_host: oleg443-server Failover mds2 to oleg443-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0001 20:08:40 (1746144520) targets are mounted 20:08:40 (1746144520) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11920 3594356 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37576 3568848 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7163204 1% /mnt/lustre /sec execute 152 sec latency 12352.540 ms oleg443-client.virtnet: 1 19945 0.54 MB/sec execute 153 sec latency 13354.020 ms oleg443-client.virtnet: 1 19945 0.53 MB/sec execute 154 sec latency 14354.889 ms oleg443-client.virtnet: 1 19945 0.53 MB/sec execute 155 sec latency 15357.013 ms oleg443-client.virtnet: 1 19945 0.53 MB/sec execute 156 sec latency 16360.800 ms oleg443-client.virtnet: 1 19945 0.52 MB/sec execute 157 sec latency 17360.907 ms oleg443-client.virtnet: 1 19945 0.52 MB/sec execute 158 sec latency 18361.087 ms oleg443-client.virtnet: 1 19945 0.52 MB/sec execute 159 sec latency 19361.501 ms oleg443-client.virtnet: 1 19945 0.51 MB/sec execute 160 sec latency 20361.653 ms oleg443-client.virtnet: 1 19974 0.51 MB/sec execute 161 sec latency 21019.007 ms oleg443-client.virtnet: 1 20173 0.52 MB/sec execute 162 sec latency 56.235 ms oleg443-client.virtnet: 1 20287 0.52 MB/sec execute 163 sec latency 59.845 ms oleg443-client.virtnet: 1 20712 0.54 MB/sec execute 164 sec latency 41.964 ms oleg443-client.virtnet: 1 21130 0.55 MB/sec execute 165 sec latency 28.975 ms oleg443-client.virtnet: 1 21395 0.57 MB/sec execute 166 sec latency 42.357 ms oleg443-client.virtnet: 1 21566 0.57 MB/sec execute 167 sec latency 38.085 ms oleg443-client.virtnet: 1 21709 0.57 MB/sec execute 168 sec latency 30.522 ms oleg443-client.virtnet: 1 21813 0.57 MB/sec execute 169 sec latency 53.000 ms oleg443-client.virtnet: 1 22003 0.56 MB/sec execute 170 sec latency 77.753 ms oleg443-client.virtnet: 1 22104 0.56 MB/sec execute 171 sec latency 62.015 ms oleg443-client.virtnet: 1 22239 0.56 MB/sec execute 172 sec latency 47.050 ms oleg443-client.virtnet: 1 22384 0.56 MB/sec execute 173 sec latency 36.885 ms oleg443-client.virtnet: 1 22590 0.56 MB/sec execute 174 sec latency 28.529 ms oleg443-client.virtnet: 1 22854 0.58 MB/sec execute 175 sec latency 28.473 ms oleg443-client.virtnet: 1 23005 0.57 MB/sec execute 176 sec latency 37.707 ms oleg443-client.virtnet: 1 23170 0.57 MB/sec execute 177 sec latency 26.503 ms oleg443-client.virtnet: 1 23382 0.57 MB/sec execute 178 sec latency 32.098 ms oleg443-client.virtnet: 1 23627 0.57 MB/sec execute 179 sec latency 28.216 ms oleg443-client.virtnet: 1 23815 0.57 MB/sec execute 180 sec latency 40.442 ms oleg443-client.virtnet: 1 24247 0.60 MB/sec execute 181 sec latency 27.005 ms oleg443-client.virtnet: 1 24628 0.60 MB/sec execute 182 sec latency 38.483 ms oleg443-client.virtnet: 1 24941 0.62 MB/sec execute 183 sec latency 43.810 ms oleg443-client.virtnet: 1 25129 0.62 MB/sec execute 184 sec latency 28.807 ms oleg443-client.virtnet: 1 25293 0.62 MB/sec execute 185 sec latency 35.999 ms oleg443-client.virtnet: 1 25474 0.62 MB/sec execute 186 sec latency 27.413 ms oleg443-client.virtnet: 1 25664 0.62 MB/sec execute 187 sec latency 36.312 ms oleg443-client.virtnet: 1 25798 0.61 MB/sec execute 188 sec latency 38.411 ms oleg443-client.virtnet: 1 25924 0.62 MB/sec execute 189 sec latency 224.009 ms oleg443-client.virtnet: 1 25924 0.61 MB/sec execute 190 sec latency 1224.284 ms oleg443-client.virtnet: 1 25924 0.61 MB/sec execute 191 sec latency 2224.464 ms oleg443-client.virtnet: 1 25924 0.61 MB/sec execute 192 sec latency 3224.649 ms oleg443-client.virtnet: 1 26102 0.61 MB/sec execute 193 sec latency 3233.180 ms oleg443-client.virtnet: 1 26352 0.62 MB/sec execute 194 sec latency 38.185 ms oleg443-client.virtnet: 1 26537 0.62 MB/sec execute 195 sec latency 29.736 ms oleg443-client.virtnet: 1 26678 0.61 MB/sec execute 196 sec latency 47.250 ms oleg443-clietest_70b fail mds1 9 times Failing mds1 on oleg443-server Stopping /mnt/lustre-mds1 (opts:) on oleg443-server 20:08:54 (1746144534) shut down facet: mds1 facet_host: oleg443-server facet_failover_host: oleg443-server Failover mds1 to oleg443-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 20:09:11 (1746144551) targets are mounted 20:09:11 (1746144551) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12036 3594132 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37632 3566276 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49668 7160408 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg443-server Stopping /mnt/lustre-mds2 (opts:) on oleg443-server 20:09:27 (1746144567) shut down facet: mds2 facet_host: oleg443-server facet_failover_host: oleg443-server nt.virtnet: 1 26883 0.61 MB/sec execute 197 sec latency 44.368 ms oleg443-client.virtnet: 1 27046 0.61 MB/sec execute 198 sec latency 39.240 ms oleg443-client.virtnet: 1 27221 0.61 MB/sec execute 199 sec latency 325.545 ms oleg443-client.virtnet: 1 27281 0.61 MB/sec execute 200 sec latency 807.513 ms oleg443-client.virtnet: 1 27482 0.61 MB/sec execute 201 sec latency 35.409 ms oleg443-client.virtnet: 1 27839 0.63 MB/sec execute 202 sec latency 53.273 ms oleg443-client.virtnet: 1 28048 0.63 MB/sec execute 203 sec latency 442.530 ms oleg443-client.virtnet: 1 28048 0.63 MB/sec execute 204 sec latency 1442.784 ms oleg443-client.virtnet: 1 28048 0.62 MB/sec execute 205 sec latency 2443.115 ms oleg443-client.virtnet: 1 28048 0.62 MB/sec execute 206 sec latency 3443.345 ms oleg443-client.virtnet: 1 28048 0.62 MB/sec execute 207 sec latency 4444.303 ms oleg443-client.virtnet: 1 28048 0.61 MB/sec execute 208 sec latency 5444.466 ms oleg443-client.virtnet: 1 28048 0.61 MB/sec execute 209 sec latency 6444.628 ms oleg443-client.virtnet: 1 28048 0.61 MB/sec execute 210 sec latency 7444.799 ms oleg443-client.virtnet: 1 28048 0.61 MB/sec execute 211 sec latency 8444.962 ms oleg443-client.virtnet: 1 28048 0.60 MB/sec execute 212 sec latency 9448.161 ms oleg443-client.virtnet: 1 28048 0.60 MB/sec execute 213 sec latency 10448.369 ms oleg443-client.virtnet: 1 28048 0.60 MB/sec execute 214 sec latency 11448.738 ms oleg443-client.virtnet: 1 28048 0.59 MB/sec execute 215 sec latency 12449.377 ms oleg443-client.virtnet: 1 28048 0.59 MB/sec execute 216 sec latency 13449.578 ms oleg443-client.virtnet: 1 28048 0.59 MB/sec execute 217 sec latency 14452.174 ms oleg443-client.virtnet: 1 28048 0.59 MB/sec execute 218 sec latency 15452.530 ms oleg443-client.virtnet: 1 28048 0.58 MB/sec execute 219 sec latency 16454.451 ms oleg443-client.virtnet: 1 28048 0.58 MB/sec execute 220 sec latency 17454.858 ms oleg443-client.virtnet: 1 28048 0.58 MB/sec execute 221 sec latency 18455.511 ms oleg443-client.virtnet: 1 28048 0.58 MB/sec execute 222 sec latency 19455.695 ms oleg443-client.virtnet: 1 28048 0.57 MB/sec execute 223 sec latency 20456.758 ms oleg443-client.virtnet: 1 28213 0.58 MB/sec execute 224 sec latency 20797.072 ms oleg443-client.virtnet: 1 28435 0.59 MB/sec execute 225 sec latency 55.852 ms oleg443-client.virtnet: 1 28557 0.59 MB/sec execute 226 sec latency 56.916 ms oleg443-client.virtnet: 1 28730 0.59 MB/sec execute 227 sec latency 33.605 ms oleg443-client.virtnet: 1 28849 0.59 MB/sec execute 228 sec latency 31.244 ms oleg443-client.virtnet: 1 28986 0.58 MB/sec execute 229 sec latency 44.296 ms oleg443-client.virtnet: 1 29139 0.58 MB/sec execute 230 sec latency 46.690 ms oleg443-client.virtnet: 1 29248 0.58 MB/sec execute 231 sec latency 37.293 ms oleg443-client.virtnet: 1 29360 0.58 MB/sec execute 232 sec latency 50.665 ms oleg443-client.virtnet: 1 29555 0.58 MB/sec execute 233 sec latency 40.593 ms oleg443-client.virtnet: 1 29674 0.58 MB/sec execute 234 sec latency 39.397 ms oleg443-client.virtnet: 1 29879 0.59 MB/sec execute 235 sec latency 48.582 ms oleg443-client.virtnet: 1 30011 0.59 MB/sec execute 236 sec latency 41.756 ms oleg443-client.virtnet: 1 30171 0.59 MB/sec execute 237 sec latency 46.113 ms oleg443-client.virtnet: 1 30333 0.59 MB/sec execute 238 sec latency 29.570 ms oleg443-client.virtnet: 1 30484 0.58 MB/sec execute 239 sec latency 48.798 ms oleg443-client.virtnet: 1 30720 0.59 MB/sec execute 240 sec latency 35.539 ms oleg443-client.virtnet: 1 30897 0.59Failover mds2 to oleg443-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0001 20:09:44 (1746144584) targets are mounted 20:09:44 (1746144584) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec MB/sec execute 241 sec latency 36.123 ms oleg443-client.virtnet: 1 31330 0.60 MB/sec execute 242 sec latency 35.059 ms oleg443-client.virtnet: 1 31767 0.61 MB/sec execute 243 sec latency 24.983 ms oleg443-client.virtnet: 1 32070 0.62 MB/sec execute 244 sec latency 33.301 ms oleg443-client.virtnet: 1 32258 0.62 MB/sec execute 245 sec latency 29.296 ms oleg443-client.virtnet: 1 32417 0.62 MB/sec execute 246 sec latency 32.500 ms oleg443-client.virtnet: 1 32489 0.62 MB/sec execute 247 sec latency 299.423 ms oleg443-client.virtnet: 1 32489 0.62 MB/sec execute 248 sec latency 1299.585 ms oleg443-client.virtnet: 1 32489 0.61 MB/sec execute 249 sec latency 2300.574 ms oleg443-client.virtnet: 1 32489 0.61 MB/sec execute 250 sec latency 3303.347 ms oleg443-client.virtnet: 1 32489 0.61 MB/sec execute 251 sec latency 4304.251 ms oleg443-client.virtnet: 1 32489 0.61 MB/sec execute 252 sec latency 5304.413 ms oleg443-client.virtnet: 1 32489 0.60 MB/sec execute 253 sec latency 6304.589 ms oleg443-client.virtnet: 1 32489 0.60 MB/sec execute 254 sec latency 7305.921 ms oleg443-client.virtnet: 1 32489 0.60 MB/sec execute 255 sec latency 8306.263 ms oleg443-client.virtnet: 1 32489 0.60 MB/sec execute 256 sec latency 9307.463 ms oleg443-client.virtnet: 1 32641 0.60 MB/sec execute 257 sec latency 9390.841 ms oleg443-client.virtnet: 1 32741 0.59 MB/sec execute 258 sec latency 63.818 ms oleg443-client.virtnet: 1 32867 0.59 MB/sec execute 259 sec latency 68.275 ms oleg443-client.virtnet: 1 33033 0.60 MB/sec execute 260 sec latency 46.277 ms oleg443-client.virtnet: 1 33194 0.59 MB/sec execute 261 sec latency 45.360 ms oleg443-client.virtnet: 1 33430 0.60 MB/sec execute 262 sec latency 30.517 ms oleg443-client.virtnet: 1 33597 0.60 MB/sec execute 263 sec latency 28.906 ms oleg443-client.virtnet: 1 33755 0.60 MB/sec execute 264 sec latency 36.008 ms oleg443-client.virtnet: 1 33952 0.60 MB/sec execute 265 sec latency 33.084 ms oleg443-client.virtnet: 1 34129 0.60 MB/sec execute 266 sec latency 36.507 ms oleg443-client.virtnet: 1 34360 0.60 MB/sec execute 267 sec latency 24.233 ms oleg443-client.virtnet: 1 34571 0.60 MB/sec execute 268 sec latency 33.296 ms oleg443-client.virtnet: 1 34977 0.62 MB/sec execute 269 sec latency 49.622 ms oleg443-client.virtnet: 1 35469 0.63 MB/sec execute 270 sec latency 34.277 ms oleg443-client.virtnet: 1 35659 0.63 MB/sec execute 271 sec latency 29.116 ms oleg443-client.virtnet: 1 35874 0.63 MB/sec execute 272 sec latency 22.745 ms oleg443-client.virtnet: 1 36136 0.63 MB/sec execute 273 sec latency 21.289 ms oleg443-client.virtnet: 1 36388 0.63 MB/sec execute 274 sec latency 40.103 ms oleg443-client.virtnet: 1 36584 0.63 MB/sec execute 275 sec latency 29.617 ms oleg443-client.virtnet: 1 36800 0.63 MB/sec execute 276 sec latency 23.620 ms oleg443-client.virtnet: 1 37082 0.64 MB/sec execute 277 sec latency 23.306 ms oleg443-client.virtnet: 1 37243 0.64 MB/sec execute 278 sec latency 27.808 ms oleg443-client.virtnet: 1 37468 0.64 MB/sec execute 279 sec latency 31.911 ms oleg443-client.virtnet: 1 37667 0.63 MB/sec execute 280 sec latency 34.626 ms oleg443-client.virtnet: 1 37893 0.64 MB/sec execute 281 sec latency 35.853 ms oleg443-client.virtnet: 1 38101 0.64 MB/sec execute 282 sec latency 33.679 ms oleg443-client.virtnet: 1 38516 0.65 MB/sec execute 283 sec latency 25.538 ms oleg443-client.virtnet: 1 39042 0.67 MB/sec execute 284 sec latency 31.946 ms oleg443-client.virtnet: 1 39224 0.67 MB/sec execute 285 sec latency 36.534 ms oleg443-client.virtnet: 1 39424 0.67 MB/sec execute 286 sec latency 28.862 ms oleg443-client.virtnet: 1 39580 0.67 MB/sec execute 287 sec latency 29.445 ms oleg443-client.virtnet: 1 39802 0.66 MB/sec execute 288 sec latency 42.458 ms oleg443-client.virtnet: 1 40001 0.66 MB/sec execute 289 sec latency 25.854 ms oleg443-client.virtnet: 1 40241 0.67 MB/sec execute 290 sec latency 32.949 ms oleg443-client.virtnet: 1 40522 0.67 MB/sec execute 291 sec latency 26.397 ms oleg443-client.virtnet: 1 40715 0.67 MB/sec execute 292 sec latency 26.786 ms oleg443-client.virtnet: 1 40912 0.67 MB/sec execute 293 sec latency 29.422 ms oleg443-client.virtnet: 1 41211 0.67 MB/sec execute 294 sec latency 23.843 ms oleg443-client.virtnet: 1 41465 0.67 MB/sec execute 295 sec latency 27.198 ms oleg443-client.virtnet: 1 41831 0.68 MB/sec execute 296 sec latency 21.130 ms oleg443-client.virtnet: 1 42397 0.69 MB/sec execute 297 sec latency 23.413 ms oleg443-client.virtnet: 1 42709 0.70 MB/sec execute 298 sec latency 27.754 ms oleg443-client.virtnet: 1 42890 0.70 MB/sec execute 299 sec latency 26.688 ms oleg443-client.virtnet: 1 cleanup 300 sec oleg443-client.virtnet: 0 cleanup 301 sec oleg443-client.virtnet: oleg443-client.virtnet: Operation Count AvgLat MaxLat oleg443-client.virtnet: ---------------------------------------- oleg443-client.virtnet: NTCreateX 6720 25.552 21427.775 oleg443-client.virtnet: Close 4952 7.271 20711.069 oleg443-client.virtnet: Rename 286 21.646 61.999 oleg443-client.virtnet: Unlink 1360 4.700 19.233 oleg443-client.virtnet: Qpathinfo 6113 11.013 18677.024 oleg443-client.virtnet: Qfileinfo 1067 0.287 19.413 oleg443-client.virtnet: Qfsinfo 1119 0.513 21.422 oleg443-client.virtnet: Sfileinfo 550 14.322 68.260 oleg443-client.virtnet: Find 2358 0.767 36.335 oleg443-client.virtnet: WriteX 3334 2.168 48.229 oleg443-client.virtnet: ReadX 10590 0.038 12.461 oleg443-client.virtnet: LockX 22 3.062 6.069 oleg443-client.virtnet: UnlockX 22 3.469 9.883 oleg443-client.virtnet: Flush 473 18.294 807.498 oleg443-client.virtnet: oleg443-client.virtnet: Throughput 0.702104 MB/sec 1 clients 1 procs max_latency=21427.795 ms oleg443-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg443-client.virtnet at Thu May 1 20:10:32 EDT 2025 with return code 0 oleg443-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg443-client.virtnet oleg443-client.virtnet: /mnt/lustre/d70b.replay-single/oleg443-client.virtnet /mnt/lustre/d70b.replay-single/oleg443-client.virtnet oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg443-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg443-client.virtnet: removed directory: 'clients/client0' oleg443-client.virtnet: removed directory: 'clients' oleg443-client.virtnet: removed 'client.txt' oleg443-client.virtnet: /mnt/lustre/d70b.replay-single/oleg443-client.virtnet oleg443-client.virtnet: dbench successfully finished