== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 16:57:04 (1746133024) Starting client oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre Started clients oleg256-client.virtnet: 192.168.202.156@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg256-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14021 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2504 1285184 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26144 3580876 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7186316 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 16:57:15 (1746133035) shut down facet: mds1 facet_host: oleg256-server facet_failover_host: oleg256-server Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 16:57:33 (1746133053) targets are mounted 16:57:33 (1746133053) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg256-client.virtnet: looking for dbench program oleg256-client.virtnet: /usr/bin/dbench oleg256-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg256-client.virtnet' oleg256-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg256-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg256-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg256-client.virtnet at Thu May 1 16:57:06 EDT 2025 oleg256-client.virtnet: waiting for dbench pid 14046 oleg256-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg256-client.virtnet: oleg256-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg256-client.virtnet: failed to create barrier semaphore oleg256-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg256-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg256-client.virtnet: releasing clients oleg256-client.virtnet: 1 178 7.19 MB/sec warmup 1 sec latency 46.731 ms oleg256-client.virtnet: 1 358 6.39 MB/sec warmup 2 sec latency 37.043 ms oleg256-client.virtnet: 1 589 6.76 MB/sec warmup 3 sec latency 40.999 ms oleg256-client.virtnet: 1 767 5.34 MB/sec warmup 4 sec latency 37.835 ms oleg256-client.virtnet: 1 848 4.31 MB/sec warmup 5 sec latency 492.213 ms oleg256-client.virtnet: 1 962 3.63 MB/sec warmup 6 sec latency 40.474 ms oleg256-client.virtnet: 1 1128 3.28 MB/sec warmup 7 sec latency 38.111 ms oleg256-client.virtnet: 1 1242 2.88 MB/sec warmup 8 sec latency 201.460 ms oleg256-client.virtnet: 1 1242 2.56 MB/sec warmup 9 sec latency 1201.696 ms oleg256-client.virtnet: 1 1242 2.31 MB/sec warmup 10 sec latency 2202.938 ms oleg256-client.virtnet: 1 1242 2.10 MB/sec warmup 11 sec latency 3203.790 ms oleg256-client.virtnet: 1 1242 1.92 MB/sec warmup 12 sec latency 4204.003 ms oleg256-client.virtnet: 1 1242 1.77 MB/sec warmup 13 sec latency 5204.171 ms oleg256-client.virtnet: 1 1242 1.65 MB/sec warmup 14 sec latency 6204.384 ms oleg256-client.virtnet: 1 1242 1.54 MB/sec warmup 15 sec latency 7204.981 ms oleg256-client.virtnet: 1 1242 1.44 MB/sec warmup 16 sec latency 8205.193 ms oleg256-client.virtnet: 1 1242 1.36 MB/sec warmup 17 sec latency 9205.407 ms oleg256-client.virtnet: 1 1242 1.28 MB/sec warmup 18 sec latency 10205.633 ms oleg256-client.virtnet: 1 1242 1.21 MB/sec warmup 19 sec latency 11205.843 ms oleg256-client.virtnet: 1 1242 1.15 MB/sec warmup 20 sec latency 12206.135 ms oleg256-client.virtnet: 1 1242 1.10 MB/sec warmup 21 sec latency 13206.356 ms oleg256-client.virtnet: 1 1242 1.05 MB/sec warmup 22 sec latency 14206.571 ms oleg256-client.virtnet: 1 1242 1.00 MB/sec warmup 23 sec latency 15206.822 ms oleg256-client.virtnet: 1 1242 0.96 MB/sec warmup 24 sec latency 16207.308 ms oleg256-client.virtnet: 1 1242 0.92 MB/sec warmup 25 sec latency 17207.479 ms oleg256-client.virtnet: 1 1242 0.89 MB/sec warmup 26 sec latency 18207.927 ms oleg256-client.virtnet: 1 1242 0.85 MB/sec warmup 27 sec latency 19211.915 ms oleg256-client.virtnet: 1 1242 0.82 MB/sec warmup 28 sec latency 20212.477 ms oleg256-client.virtnet: 1 1242 0.80 MB/sec warmup 29 sec latency 21213.012 ms oleg256-client.virtnet: 1 1248 0.77 MB/sec warmup 30 sec latency 22072.407 ms oleg256-client.virtnet: 1 1463 0.84 MB/sec warmup 31 sec latency 37.101 ms oleg256-client.virtnet: 1 1587 0.82 MB/sec warmup 32 sec latency 44.243 ms oleg256-client.virtnet: 1 1717 0.79 MB/sec warmup 33 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1776 3593020 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 27176 3569868 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28952 7162888 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg256-server Stopping /mnt/lustre-mds2 (opts:) on oleg256-server 16:57:48 (1746133068) shut down facet: mds2 facet_host: oleg256-server facet_failover_host: oleg256-server Failover mds2 to oleg256-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 16:58:06 (1746133086) targets are mounted 16:58:06 (1746133086) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12940 3592448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36700 3570236 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49640 7162684 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 16:58:23 (1746133103) shut down facet: mds1 facet_host: oleg256-server facet_failover_host: oleg256-server latency 60.207 ms oleg256-client.virtnet: 1 1913 0.78 MB/sec warmup 34 sec latency 33.561 ms oleg256-client.virtnet: 1 2128 0.76 MB/sec warmup 35 sec latency 56.691 ms oleg256-client.virtnet: 1 2348 0.77 MB/sec warmup 36 sec latency 31.123 ms oleg256-client.virtnet: 1 2582 0.78 MB/sec warmup 37 sec latency 27.488 ms oleg256-client.virtnet: 1 2990 0.88 MB/sec warmup 38 sec latency 36.933 ms oleg256-client.virtnet: 1 3375 0.91 MB/sec warmup 39 sec latency 25.315 ms oleg256-client.virtnet: 1 3651 0.97 MB/sec warmup 40 sec latency 30.150 ms oleg256-client.virtnet: 1 3812 0.97 MB/sec warmup 41 sec latency 50.158 ms oleg256-client.virtnet: 1 3932 0.95 MB/sec warmup 42 sec latency 40.566 ms oleg256-client.virtnet: 1 4062 0.93 MB/sec warmup 43 sec latency 29.929 ms oleg256-client.virtnet: 1 4083 0.91 MB/sec warmup 44 sec latency 832.133 ms oleg256-client.virtnet: 1 4083 0.89 MB/sec warmup 45 sec latency 1832.387 ms oleg256-client.virtnet: 1 4083 0.87 MB/sec warmup 46 sec latency 2834.796 ms oleg256-client.virtnet: 1 4083 0.85 MB/sec warmup 47 sec latency 3836.316 ms oleg256-client.virtnet: 1 4083 0.83 MB/sec warmup 48 sec latency 4836.627 ms oleg256-client.virtnet: 1 4083 0.82 MB/sec warmup 49 sec latency 5836.851 ms oleg256-client.virtnet: 1 4083 0.80 MB/sec warmup 50 sec latency 6837.979 ms oleg256-client.virtnet: 1 4083 0.79 MB/sec warmup 51 sec latency 7838.187 ms oleg256-client.virtnet: 1 4083 0.77 MB/sec warmup 52 sec latency 8838.468 ms oleg256-client.virtnet: 1 4083 0.76 MB/sec warmup 53 sec latency 9838.645 ms oleg256-client.virtnet: 1 4083 0.74 MB/sec warmup 54 sec latency 10838.854 ms oleg256-client.virtnet: 1 4083 0.73 MB/sec warmup 55 sec latency 11839.264 ms oleg256-client.virtnet: 1 4083 0.72 MB/sec warmup 56 sec latency 12839.395 ms oleg256-client.virtnet: 1 4083 0.70 MB/sec warmup 57 sec latency 13839.561 ms oleg256-client.virtnet: 1 4083 0.69 MB/sec warmup 58 sec latency 14840.230 ms oleg256-client.virtnet: 1 4083 0.68 MB/sec warmup 59 sec latency 15846.506 ms oleg256-client.virtnet: 1 4083 0.00 MB/sec execute 1 sec latency 17846.834 ms oleg256-client.virtnet: 1 4091 0.00 MB/sec execute 2 sec latency 18788.469 ms oleg256-client.virtnet: 1 4281 0.08 MB/sec execute 3 sec latency 40.340 ms oleg256-client.virtnet: 1 4375 0.13 MB/sec execute 4 sec latency 51.141 ms oleg256-client.virtnet: 1 4493 0.16 MB/sec execute 5 sec latency 53.929 ms oleg256-client.virtnet: 1 4652 0.33 MB/sec execute 6 sec latency 47.397 ms oleg256-client.virtnet: 1 4809 0.30 MB/sec execute 7 sec latency 32.732 ms oleg256-client.virtnet: 1 4999 0.63 MB/sec execute 8 sec latency 74.606 ms oleg256-client.virtnet: 1 5123 0.57 MB/sec execute 9 sec latency 38.874 ms oleg256-client.virtnet: 1 5197 0.52 MB/sec execute 10 sec latency 192.361 ms oleg256-client.virtnet: 1 5263 0.48 MB/sec execute 11 sec latency 530.519 ms oleg256-client.virtnet: 1 5385 0.45 MB/sec execute 12 sec latency 37.822 ms oleg256-client.virtnet: 1 5545 0.42 MB/sec execute 13 sec latency 57.349 ms oleg256-client.virtnet: 1 5642 0.40 MB/sec execute 14 sec latency 364.647 ms oleg256-client.virtnet: 1 5642 0.38 MB/sec execute 15 sec latency 1366.101 ms oleg256-client.virtnet: 1 5642 0.35 MB/sec execute 16 sec latency 2369.868 ms oleg256-client.virtnet: 1 5642 0.33 MB/sec execute 17 sec latency 3370.050 ms oleg256-client.virtnet: 1 5642 0.31 MB/sec execute 18 sec latency 4370.366 ms oleg256-client.virtnet: 1 5642 0.Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 16:58:40 (1746133120) targets are mounted 16:58:40 (1746133120) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13064 3593104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36624 3567632 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49688 7160736 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg256-server Stopping /mnt/lustre-mds2 (opts:) on oleg256-server 16:58:56 (1746133136) shut down facet: mds2 facet_host: oleg256-server facet_failover_host: oleg256-server Failover mds2 to oleg256-server mount facets: mds2 30 MB/sec execute 19 sec latency 5370.566 ms oleg256-client.virtnet: 1 5642 0.28 MB/sec execute 20 sec latency 6371.698 ms oleg256-client.virtnet: 1 5642 0.27 MB/sec execute 21 sec latency 7371.984 ms oleg256-client.virtnet: 1 5642 0.26 MB/sec execute 22 sec latency 8372.169 ms oleg256-client.virtnet: 1 5642 0.25 MB/sec execute 23 sec latency 9372.720 ms oleg256-client.virtnet: 1 5642 0.24 MB/sec execute 24 sec latency 10372.931 ms oleg256-client.virtnet: 1 5642 0.23 MB/sec execute 25 sec latency 11373.264 ms oleg256-client.virtnet: 1 5642 0.22 MB/sec execute 26 sec latency 12373.558 ms oleg256-client.virtnet: 1 5642 0.21 MB/sec execute 27 sec latency 13373.953 ms oleg256-client.virtnet: 1 5642 0.20 MB/sec execute 28 sec latency 14374.147 ms oleg256-client.virtnet: 1 5642 0.20 MB/sec execute 29 sec latency 15376.162 ms oleg256-client.virtnet: 1 5642 0.19 MB/sec execute 30 sec latency 16377.929 ms oleg256-client.virtnet: 1 5642 0.18 MB/sec execute 31 sec latency 17378.271 ms oleg256-client.virtnet: 1 5642 0.18 MB/sec execute 32 sec latency 18378.478 ms oleg256-client.virtnet: 1 5642 0.17 MB/sec execute 33 sec latency 19378.808 ms oleg256-client.virtnet: 1 5642 0.17 MB/sec execute 34 sec latency 20379.141 ms oleg256-client.virtnet: 1 5642 0.16 MB/sec execute 35 sec latency 21379.930 ms oleg256-client.virtnet: 1 5642 0.16 MB/sec execute 36 sec latency 22380.659 ms oleg256-client.virtnet: 1 5642 0.15 MB/sec execute 37 sec latency 23380.799 ms oleg256-client.virtnet: 1 5759 0.15 MB/sec execute 38 sec latency 23442.326 ms oleg256-client.virtnet: 1 5936 0.18 MB/sec execute 39 sec latency 37.636 ms oleg256-client.virtnet: 1 6117 0.20 MB/sec execute 40 sec latency 47.009 ms oleg256-client.virtnet: 1 6486 0.31 MB/sec execute 41 sec latency 30.775 ms oleg256-client.virtnet: 1 6907 0.34 MB/sec execute 42 sec latency 32.985 ms oleg256-client.virtnet: 1 7136 0.41 MB/sec execute 43 sec latency 35.748 ms oleg256-client.virtnet: 1 7319 0.43 MB/sec execute 44 sec latency 34.934 ms oleg256-client.virtnet: 1 7472 0.42 MB/sec execute 45 sec latency 30.765 ms oleg256-client.virtnet: 1 7586 0.41 MB/sec execute 46 sec latency 42.076 ms oleg256-client.virtnet: 1 7775 0.41 MB/sec execute 47 sec latency 32.905 ms oleg256-client.virtnet: 1 7882 0.41 MB/sec execute 48 sec latency 73.414 ms oleg256-client.virtnet: 1 7996 0.40 MB/sec execute 49 sec latency 64.156 ms oleg256-client.virtnet: 1 8092 0.40 MB/sec execute 50 sec latency 47.688 ms oleg256-client.virtnet: 1 8269 0.41 MB/sec execute 51 sec latency 29.807 ms oleg256-client.virtnet: 1 8489 0.46 MB/sec execute 52 sec latency 42.421 ms oleg256-client.virtnet: 1 8627 0.46 MB/sec execute 53 sec latency 32.802 ms oleg256-client.virtnet: 1 8775 0.45 MB/sec execute 54 sec latency 40.949 ms oleg256-client.virtnet: 1 8924 0.44 MB/sec execute 55 sec latency 43.510 ms oleg256-client.virtnet: 1 9123 0.44 MB/sec execute 56 sec latency 30.666 ms oleg256-client.virtnet: 1 9287 0.44 MB/sec execute 57 sec latency 40.379 ms oleg256-client.virtnet: 1 9490 0.45 MB/sec execute 58 sec latency 40.697 ms oleg256-client.virtnet: 1 9676 0.46 MB/sec execute 59 sec latency 42.731 ms oleg256-client.virtnet: 1 10021 0.53 MB/sec execute 60 sec latency 32.938 ms oleg256-client.virtnet: 1 10331 0.53 MB/sec execute 61 sec latency 34.360 ms oleg256-client.virtnet: 1 10647 0.59 MB/sec execute 62 sec latency 36.600 ms oleg256-client.virtnet: 1 10819 0.59 MB/sec execute 63 sec latency 3Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 16:59:13 (1746133153) targets are mounted 16:59:13 (1746133153) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12940 3593276 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36624 3569892 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7163168 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 16:59:29 (1746133169) shut down facet: mds1 facet_host: oleg256-server facet_failover_host: oleg256-server Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 16:59:47 (1746133187) targets are mounted 16:59:47 (1746133187) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12940 3592012 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36712 3569852 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49652 7161864 1% /mnt/lustre 2.445 ms oleg256-client.virtnet: 1 10970 0.59 MB/sec execute 64 sec latency 41.482 ms oleg256-client.virtnet: 1 11099 0.59 MB/sec execute 65 sec latency 41.945 ms oleg256-client.virtnet: 1 11277 0.58 MB/sec execute 66 sec latency 34.698 ms oleg256-client.virtnet: 1 11335 0.57 MB/sec execute 67 sec latency 704.448 ms oleg256-client.virtnet: 1 11335 0.57 MB/sec execute 68 sec latency 1704.590 ms oleg256-client.virtnet: 1 11335 0.56 MB/sec execute 69 sec latency 2704.839 ms oleg256-client.virtnet: 1 11400 0.55 MB/sec execute 70 sec latency 3222.734 ms oleg256-client.virtnet: 1 11537 0.55 MB/sec execute 71 sec latency 43.335 ms oleg256-client.virtnet: 1 11635 0.54 MB/sec execute 72 sec latency 45.245 ms oleg256-client.virtnet: 1 11819 0.55 MB/sec execute 73 sec latency 32.349 ms oleg256-client.virtnet: 1 12043 0.58 MB/sec execute 74 sec latency 36.104 ms oleg256-client.virtnet: 1 12158 0.58 MB/sec execute 75 sec latency 48.473 ms oleg256-client.virtnet: 1 12271 0.57 MB/sec execute 76 sec latency 48.525 ms oleg256-client.virtnet: 1 12374 0.56 MB/sec execute 77 sec latency 70.372 ms oleg256-client.virtnet: 1 12484 0.56 MB/sec execute 78 sec latency 45.028 ms oleg256-client.virtnet: 1 12635 0.55 MB/sec execute 79 sec latency 48.555 ms oleg256-client.virtnet: 1 12793 0.55 MB/sec execute 80 sec latency 47.758 ms oleg256-client.virtnet: 1 12982 0.56 MB/sec execute 81 sec latency 38.009 ms oleg256-client.virtnet: 1 13032 0.55 MB/sec execute 82 sec latency 548.025 ms oleg256-client.virtnet: 1 13032 0.54 MB/sec execute 83 sec latency 1548.273 ms oleg256-client.virtnet: 1 13032 0.54 MB/sec execute 84 sec latency 2549.168 ms oleg256-client.virtnet: 1 13032 0.53 MB/sec execute 85 sec latency 3550.978 ms oleg256-client.virtnet: 1 13032 0.53 MB/sec execute 86 sec latency 4551.609 ms oleg256-client.virtnet: 1 13032 0.52 MB/sec execute 87 sec latency 5552.189 ms oleg256-client.virtnet: 1 13032 0.51 MB/sec execute 88 sec latency 6552.442 ms oleg256-client.virtnet: 1 13032 0.51 MB/sec execute 89 sec latency 7554.535 ms oleg256-client.virtnet: 1 13032 0.50 MB/sec execute 90 sec latency 8554.752 ms oleg256-client.virtnet: 1 13032 0.50 MB/sec execute 91 sec latency 9555.066 ms oleg256-client.virtnet: 1 13032 0.49 MB/sec execute 92 sec latency 10556.110 ms oleg256-client.virtnet: 1 13032 0.49 MB/sec execute 93 sec latency 11556.353 ms oleg256-client.virtnet: 1 13032 0.48 MB/sec execute 94 sec latency 12558.310 ms oleg256-client.virtnet: 1 13032 0.48 MB/sec execute 95 sec latency 13558.670 ms oleg256-client.virtnet: 1 13032 0.47 MB/sec execute 96 sec latency 14558.889 ms oleg256-client.virtnet: 1 13032 0.47 MB/sec execute 97 sec latency 15559.566 ms oleg256-client.virtnet: 1 13032 0.46 MB/sec execute 98 sec latency 16559.733 ms oleg256-client.virtnet: 1 13032 0.46 MB/sec execute 99 sec latency 17560.017 ms oleg256-client.virtnet: 1 13032 0.45 MB/sec execute 100 sec latency 18560.160 ms oleg256-client.virtnet: 1 13032 0.45 MB/sec execute 101 sec latency 19560.337 ms oleg256-client.virtnet: 1 13032 0.44 MB/sec execute 102 sec latency 20561.940 ms oleg256-client.virtnet: 1 13042 0.44 MB/sec execute 103 sec latency 21504.046 ms oleg256-client.virtnet: 1 13244 0.44 MB/sec execute 104 sec latency 38.610 ms oleg256-client.virtnet: 1 13632 0.48 MB/sec execute 105 sec latency 29.663 ms oleg256-client.virtnet: 1 14143 0.52 MB/sec execute 106 sec latency 30.197 ms oleg256-client.virtnet: 1 14336 0.52 MB/sec execute 107 sec latency 24.792 ms oleg256-client.virtnetest_70b fail mds2 6 times Failing mds2 on oleg256-server Stopping /mnt/lustre-mds2 (opts:) on oleg256-server 17:00:00 (1746133200) shut down facet: mds2 facet_host: oleg256-server facet_failover_host: oleg256-server Failover mds2 to oleg256-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 17:00:18 (1746133218) targets are mounted 17:00:18 (1746133218) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13176 3593760 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36388 3570352 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7164112 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 17:00:33 (1746133233) shut down facet: mds1 facet_host: oleg256-server facet_failover_host: oleg256-server t: 1 14536 0.53 MB/sec execute 108 sec latency 29.669 ms oleg256-client.virtnet: 1 14709 0.53 MB/sec execute 109 sec latency 27.445 ms oleg256-client.virtnet: 1 14966 0.52 MB/sec execute 110 sec latency 32.844 ms oleg256-client.virtnet: 1 15175 0.52 MB/sec execute 111 sec latency 28.267 ms oleg256-client.virtnet: 1 15392 0.53 MB/sec execute 112 sec latency 25.133 ms oleg256-client.virtnet: 1 15612 0.55 MB/sec execute 113 sec latency 36.572 ms oleg256-client.virtnet: 1 15734 0.55 MB/sec execute 114 sec latency 66.202 ms oleg256-client.virtnet: 1 15910 0.54 MB/sec execute 115 sec latency 32.129 ms oleg256-client.virtnet: 1 16050 0.54 MB/sec execute 116 sec latency 36.399 ms oleg256-client.virtnet: 1 16227 0.54 MB/sec execute 117 sec latency 31.403 ms oleg256-client.virtnet: 1 16395 0.53 MB/sec execute 118 sec latency 34.665 ms oleg256-client.virtnet: 1 16603 0.54 MB/sec execute 119 sec latency 29.992 ms oleg256-client.virtnet: 1 16791 0.54 MB/sec execute 120 sec latency 36.079 ms oleg256-client.virtnet: 1 17123 0.58 MB/sec execute 121 sec latency 40.026 ms oleg256-client.virtnet: 1 17469 0.59 MB/sec execute 122 sec latency 36.100 ms oleg256-client.virtnet: 1 17760 0.61 MB/sec execute 123 sec latency 36.976 ms oleg256-client.virtnet: 1 17891 0.61 MB/sec execute 124 sec latency 46.315 ms oleg256-client.virtnet: 1 18019 0.61 MB/sec execute 125 sec latency 56.424 ms oleg256-client.virtnet: 1 18154 0.61 MB/sec execute 126 sec latency 40.763 ms oleg256-client.virtnet: 1 18251 0.60 MB/sec execute 127 sec latency 50.298 ms oleg256-client.virtnet: 1 18266 0.60 MB/sec execute 128 sec latency 875.184 ms oleg256-client.virtnet: 1 18266 0.59 MB/sec execute 129 sec latency 1876.834 ms oleg256-client.virtnet: 1 18266 0.59 MB/sec execute 130 sec latency 2877.347 ms oleg256-client.virtnet: 1 18266 0.58 MB/sec execute 131 sec latency 3879.213 ms oleg256-client.virtnet: 1 18266 0.58 MB/sec execute 132 sec latency 4879.609 ms oleg256-client.virtnet: 1 18266 0.57 MB/sec execute 133 sec latency 5881.449 ms oleg256-client.virtnet: 1 18346 0.57 MB/sec execute 134 sec latency 6575.918 ms oleg256-client.virtnet: 1 18488 0.57 MB/sec execute 135 sec latency 60.851 ms oleg256-client.virtnet: 1 18602 0.57 MB/sec execute 136 sec latency 47.768 ms oleg256-client.virtnet: 1 18712 0.56 MB/sec execute 137 sec latency 38.056 ms oleg256-client.virtnet: 1 18873 0.57 MB/sec execute 138 sec latency 48.333 ms oleg256-client.virtnet: 1 19013 0.57 MB/sec execute 139 sec latency 41.181 ms oleg256-client.virtnet: 1 19190 0.58 MB/sec execute 140 sec latency 54.104 ms oleg256-client.virtnet: 1 19316 0.58 MB/sec execute 141 sec latency 59.902 ms oleg256-client.virtnet: 1 19382 0.57 MB/sec execute 142 sec latency 470.235 ms oleg256-client.virtnet: 1 19441 0.57 MB/sec execute 143 sec latency 924.651 ms oleg256-client.virtnet: 1 19558 0.57 MB/sec execute 144 sec latency 42.771 ms oleg256-client.virtnet: 1 19716 0.56 MB/sec execute 145 sec latency 49.800 ms oleg256-client.virtnet: 1 19807 0.56 MB/sec execute 146 sec latency 213.891 ms oleg256-client.virtnet: 1 19807 0.56 MB/sec execute 147 sec latency 1214.016 ms oleg256-client.virtnet: 1 19807 0.55 MB/sec execute 148 sec latency 2214.665 ms oleg256-client.virtnet: 1 19807 0.55 MB/sec execute 149 sec latency 3216.288 ms oleg256-client.virtnet: 1 19807 0.55 MB/sec execute 150 sec latency 4218.956 ms oleg256-client.virtnet: 1 19807 0.54 MB/sec execute 151 sec latency 5219.222 ms oleg256-client.virtnet: 1 19807 0.54 MB/sec execute 152 sec latency 6219.5Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 17:00:51 (1746133251) targets are mounted 17:00:51 (1746133251) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13252 3592868 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36400 3568272 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49652 7161140 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg256-server Stopping /mnt/lustre-mds2 (opts:) on oleg256-server 17:01:07 (1746133267) shut down facet: mds2 facet_host: oleg256-server facet_failover_host: oleg256-server Failover mds2 to oleg256-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 91 ms oleg256-client.virtnet: 1 19807 0.54 MB/sec execute 153 sec latency 7219.846 ms oleg256-client.virtnet: 1 19807 0.53 MB/sec execute 154 sec latency 8220.064 ms oleg256-client.virtnet: 1 19807 0.53 MB/sec execute 155 sec latency 9220.284 ms oleg256-client.virtnet: 1 19807 0.53 MB/sec execute 156 sec latency 10221.703 ms oleg256-client.virtnet: 1 19807 0.52 MB/sec execute 157 sec latency 11221.942 ms oleg256-client.virtnet: 1 19807 0.52 MB/sec execute 158 sec latency 12222.154 ms oleg256-client.virtnet: 1 19807 0.52 MB/sec execute 159 sec latency 13222.395 ms oleg256-client.virtnet: 1 19807 0.51 MB/sec execute 160 sec latency 14223.085 ms oleg256-client.virtnet: 1 19807 0.51 MB/sec execute 161 sec latency 15223.705 ms oleg256-client.virtnet: 1 19807 0.51 MB/sec execute 162 sec latency 16223.918 ms oleg256-client.virtnet: 1 19807 0.50 MB/sec execute 163 sec latency 17224.078 ms oleg256-client.virtnet: 1 19807 0.50 MB/sec execute 164 sec latency 18224.391 ms oleg256-client.virtnet: 1 19807 0.50 MB/sec execute 165 sec latency 19224.591 ms oleg256-client.virtnet: 1 19807 0.49 MB/sec execute 166 sec latency 20224.735 ms oleg256-client.virtnet: 1 19807 0.49 MB/sec execute 167 sec latency 21225.613 ms oleg256-client.virtnet: 1 19887 0.49 MB/sec execute 168 sec latency 21641.652 ms oleg256-client.virtnet: 1 19980 0.49 MB/sec execute 169 sec latency 52.953 ms oleg256-client.virtnet: 1 20132 0.49 MB/sec execute 170 sec latency 78.924 ms oleg256-client.virtnet: 1 20221 0.49 MB/sec execute 171 sec latency 72.870 ms oleg256-client.virtnet: 1 20387 0.49 MB/sec execute 172 sec latency 46.183 ms oleg256-client.virtnet: 1 20770 0.51 MB/sec execute 173 sec latency 33.675 ms oleg256-client.virtnet: 1 21122 0.52 MB/sec execute 174 sec latency 38.133 ms oleg256-client.virtnet: 1 21339 0.54 MB/sec execute 175 sec latency 41.018 ms oleg256-client.virtnet: 1 21477 0.54 MB/sec execute 176 sec latency 45.207 ms oleg256-client.virtnet: 1 21621 0.54 MB/sec execute 177 sec latency 34.762 ms oleg256-client.virtnet: 1 21761 0.54 MB/sec execute 178 sec latency 38.009 ms oleg256-client.virtnet: 1 21920 0.53 MB/sec execute 179 sec latency 37.499 ms oleg256-client.virtnet: 1 22101 0.53 MB/sec execute 180 sec latency 38.103 ms oleg256-client.virtnet: 1 22254 0.53 MB/sec execute 181 sec latency 33.913 ms oleg256-client.virtnet: 1 22468 0.54 MB/sec execute 182 sec latency 30.304 ms oleg256-client.virtnet: 1 22649 0.54 MB/sec execute 183 sec latency 40.656 ms oleg256-client.virtnet: 1 22868 0.55 MB/sec execute 184 sec latency 46.673 ms oleg256-client.virtnet: 1 23049 0.55 MB/sec execute 185 sec latency 41.585 ms oleg256-client.virtnet: 1 23232 0.54 MB/sec execute 186 sec latency 34.923 ms oleg256-client.virtnet: 1 23425 0.54 MB/sec execute 187 sec latency 30.924 ms oleg256-client.virtnet: 1 23660 0.54 MB/sec execute 188 sec latency 33.055 ms oleg256-client.virtnet: 1 23921 0.55 MB/sec execute 189 sec latency 22.752 ms oleg256-client.virtnet: 1 24492 0.57 MB/sec execute 190 sec latency 24.885 ms oleg256-client.virtnet: 1 24827 0.59 MB/sec execute 191 sec latency 30.861 ms oleg256-client.virtnet: 1 24998 0.59 MB/sec execute 192 sec latency 30.889 ms oleg256-client.virtnet: 1 25128 0.59 MB/sec execute 193 sec latency 52.385 ms oleg256-client.virtnet: 1 25267 0.59 MB/sec execute 194 sec latency 35.685 ms oleg256-client.virtnet: 1 25374 0.59 MB/sec execute 195 sec latency 57.380 ms oleg256-client.virtnet: 1 25432 0.59 MB/sec execute 196 sec latency 614.798 ms oleg256-client.virtnet: 1 25432 Started lustre-MDT0001 17:01:23 (1746133283) targets are mounted 17:01:23 (1746133283) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12940 3592948 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36708 3569952 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7162900 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 17:01:38 (1746133298) shut down facet: mds1 facet_host: oleg256-server facet_failover_host: oleg256-server Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 17:01:54 (1746133314) targets are mounted 17:01:54 (1746133314) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13064 3593792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36560 3567672 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7161464 1% /mnt/lustre test_70b fail mds2 10 times 0.58 MB/sec execute 197 sec latency 1615.013 ms oleg256-client.virtnet: 1 25432 0.58 MB/sec execute 198 sec latency 2615.279 ms oleg256-client.virtnet: 1 25432 0.58 MB/sec execute 199 sec latency 3615.945 ms oleg256-client.virtnet: 1 25439 0.57 MB/sec execute 200 sec latency 4603.879 ms oleg256-client.virtnet: 1 25624 0.57 MB/sec execute 201 sec latency 36.514 ms oleg256-client.virtnet: 1 25745 0.57 MB/sec execute 202 sec latency 44.144 ms oleg256-client.virtnet: 1 25870 0.57 MB/sec execute 203 sec latency 32.198 ms oleg256-client.virtnet: 1 26037 0.57 MB/sec execute 204 sec latency 38.003 ms oleg256-client.virtnet: 1 26284 0.58 MB/sec execute 205 sec latency 31.283 ms oleg256-client.virtnet: 1 26443 0.58 MB/sec execute 206 sec latency 45.747 ms oleg256-client.virtnet: 1 26608 0.58 MB/sec execute 207 sec latency 32.292 ms oleg256-client.virtnet: 1 26769 0.58 MB/sec execute 208 sec latency 49.263 ms oleg256-client.virtnet: 1 26920 0.58 MB/sec execute 209 sec latency 54.204 ms oleg256-client.virtnet: 1 27044 0.57 MB/sec execute 210 sec latency 67.485 ms oleg256-client.virtnet: 1 27235 0.58 MB/sec execute 211 sec latency 45.804 ms oleg256-client.virtnet: 1 27235 0.57 MB/sec execute 212 sec latency 1028.788 ms oleg256-client.virtnet: 1 27235 0.57 MB/sec execute 213 sec latency 2029.369 ms oleg256-client.virtnet: 1 27235 0.57 MB/sec execute 214 sec latency 3029.843 ms oleg256-client.virtnet: 1 27235 0.57 MB/sec execute 215 sec latency 4031.850 ms oleg256-client.virtnet: 1 27235 0.56 MB/sec execute 216 sec latency 5032.108 ms oleg256-client.virtnet: 1 27235 0.56 MB/sec execute 217 sec latency 6032.350 ms oleg256-client.virtnet: 1 27235 0.56 MB/sec execute 218 sec latency 7032.623 ms oleg256-client.virtnet: 1 27235 0.56 MB/sec execute 219 sec latency 8032.872 ms oleg256-client.virtnet: 1 27235 0.55 MB/sec execute 220 sec latency 9033.178 ms oleg256-client.virtnet: 1 27235 0.55 MB/sec execute 221 sec latency 10033.436 ms oleg256-client.virtnet: 1 27235 0.55 MB/sec execute 222 sec latency 11033.780 ms oleg256-client.virtnet: 1 27235 0.55 MB/sec execute 223 sec latency 12034.034 ms oleg256-client.virtnet: 1 27235 0.54 MB/sec execute 224 sec latency 13034.229 ms oleg256-client.virtnet: 1 27235 0.54 MB/sec execute 225 sec latency 14034.376 ms oleg256-client.virtnet: 1 27235 0.54 MB/sec execute 226 sec latency 15034.544 ms oleg256-client.virtnet: 1 27235 0.54 MB/sec execute 227 sec latency 16034.713 ms oleg256-client.virtnet: 1 27235 0.53 MB/sec execute 228 sec latency 17035.118 ms oleg256-client.virtnet: 1 27235 0.53 MB/sec execute 229 sec latency 18035.754 ms oleg256-client.virtnet: 1 27235 0.53 MB/sec execute 230 sec latency 19036.629 ms oleg256-client.virtnet: 1 27264 0.53 MB/sec execute 231 sec latency 19844.406 ms oleg256-client.virtnet: 1 27449 0.53 MB/sec execute 232 sec latency 56.982 ms oleg256-client.virtnet: 1 27788 0.55 MB/sec execute 233 sec latency 43.211 ms oleg256-client.virtnet: 1 28141 0.55 MB/sec execute 234 sec latency 37.391 ms oleg256-client.virtnet: 1 28422 0.56 MB/sec execute 235 sec latency 40.196 ms oleg256-client.virtnet: 1 28556 0.56 MB/sec execute 236 sec latency 42.540 ms oleg256-client.virtnet: 1 28695 0.56 MB/sec execute 237 sec latency 34.360 ms oleg256-client.virtnet: 1 28834 0.56 MB/sec execute 238 sec latency 30.562 ms oleg256-client.virtnet: 1 28938 0.56 MB/sec execute 239 sec latency 37.760 ms oleg256-client.virtnet: 1 29114 0.56 MB/sec execute 240 sec latency 40.538 ms oleg256-client.virtnet: 1 29214 0.56 MB/sec execute 241 sec Failing mds2 on oleg256-server Stopping /mnt/lustre-mds2 (opts:) on oleg256-server 17:02:10 (1746133330) shut down facet: mds2 facet_host: oleg256-server facet_failover_host: oleg256-server Failover mds2 to oleg256-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 17:02:27 (1746133347) targets are mounted 17:02:27 (1746133347) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec latency 52.719 ms oleg256-client.virtnet: 1 29319 0.56 MB/sec execute 242 sec latency 77.867 ms oleg256-client.virtnet: 1 29406 0.55 MB/sec execute 243 sec latency 67.134 ms oleg256-client.virtnet: 1 29571 0.56 MB/sec execute 244 sec latency 41.519 ms oleg256-client.virtnet: 1 29804 0.57 MB/sec execute 245 sec latency 32.200 ms oleg256-client.virtnet: 1 29946 0.56 MB/sec execute 246 sec latency 35.543 ms oleg256-client.virtnet: 1 30085 0.56 MB/sec execute 247 sec latency 44.383 ms oleg256-client.virtnet: 1 30220 0.56 MB/sec execute 248 sec latency 30.015 ms oleg256-client.virtnet: 1 30388 0.56 MB/sec execute 249 sec latency 33.336 ms oleg256-client.virtnet: 1 30560 0.56 MB/sec execute 250 sec latency 38.716 ms oleg256-client.virtnet: 1 30772 0.56 MB/sec execute 251 sec latency 31.203 ms oleg256-client.virtnet: 1 30975 0.56 MB/sec execute 252 sec latency 34.749 ms oleg256-client.virtnet: 1 31331 0.58 MB/sec execute 253 sec latency 38.839 ms oleg256-client.virtnet: 1 31646 0.58 MB/sec execute 254 sec latency 38.209 ms oleg256-client.virtnet: 1 31968 0.59 MB/sec execute 255 sec latency 28.677 ms oleg256-client.virtnet: 1 32127 0.59 MB/sec execute 256 sec latency 33.443 ms oleg256-client.virtnet: 1 32286 0.59 MB/sec execute 257 sec latency 30.920 ms oleg256-client.virtnet: 1 32401 0.59 MB/sec execute 258 sec latency 48.397 ms oleg256-client.virtnet: 1 32489 0.59 MB/sec execute 259 sec latency 153.815 ms oleg256-client.virtnet: 1 32489 0.59 MB/sec execute 260 sec latency 1154.014 ms oleg256-client.virtnet: 1 32489 0.59 MB/sec execute 261 sec latency 2154.772 ms oleg256-client.virtnet: 1 32489 0.58 MB/sec execute 262 sec latency 3155.052 ms oleg256-client.virtnet: 1 32489 0.58 MB/sec execute 263 sec latency 4155.672 ms oleg256-client.virtnet: 1 32696 0.58 MB/sec execute 264 sec latency 4173.827 ms oleg256-client.virtnet: 1 32888 0.58 MB/sec execute 265 sec latency 27.535 ms oleg256-client.virtnet: 1 33128 0.58 MB/sec execute 266 sec latency 22.659 ms oleg256-client.virtnet: 1 33392 0.59 MB/sec execute 267 sec latency 33.910 ms oleg256-client.virtnet: 1 33613 0.59 MB/sec execute 268 sec latency 21.999 ms oleg256-client.virtnet: 1 33819 0.59 MB/sec execute 269 sec latency 21.983 ms oleg256-client.virtnet: 1 34050 0.59 MB/sec execute 270 sec latency 24.880 ms oleg256-client.virtnet: 1 34298 0.59 MB/sec execute 271 sec latency 33.167 ms oleg256-client.virtnet: 1 34548 0.59 MB/sec execute 272 sec latency 25.826 ms oleg256-client.virtnet: 1 35139 0.61 MB/sec execute 273 sec latency 23.482 ms oleg256-client.virtnet: 1 35571 0.62 MB/sec execute 274 sec latency 24.631 ms oleg256-client.virtnet: 1 35779 0.62 MB/sec execute 275 sec latency 25.956 ms oleg256-client.virtnet: 1 35943 0.62 MB/sec execute 276 sec latency 27.195 ms oleg256-client.virtnet: 1 36110 0.62 MB/sec execute 277 sec latency 33.817 ms oleg256-client.virtnet: 1 36278 0.62 MB/sec execute 278 sec latency 40.513 ms oleg256-client.virtnet: 1 36440 0.62 MB/sec execute 279 sec latency 31.725 ms oleg256-client.virtnet: 1 36650 0.62 MB/sec execute 280 sec latency 31.188 ms oleg256-client.virtnet: 1 36927 0.63 MB/sec execute 281 sec latency 23.999 ms oleg256-client.virtnet: 1 37090 0.63 MB/sec execute 282 sec latency 34.090 ms oleg256-client.virtnet: 1 37260 0.63 MB/sec execute 283 sec latency 32.539 ms oleg256-client.virtnet: 1 37516 0.63 MB/sec execute 284 sec latency 22.695 ms oleg256-client.virtnet: 1 37774 0.62 MB/sec execute 285 sec latency 23.747 ms oleg256-client.virtnet: 1 37975 0.63 MB/sec execute 286 sec latency 32.827 ms oleg256-client.virtnet: 1 38402 0.64 MB/sec execute 287 sec latency 27.230 ms oleg256-client.virtnet: 1 38843 0.65 MB/sec execute 288 sec latency 29.690 ms oleg256-client.virtnet: 1 39121 0.66 MB/sec execute 289 sec latency 24.361 ms oleg256-client.virtnet: 1 39315 0.66 MB/sec execute 290 sec latency 29.512 ms oleg256-client.virtnet: 1 39482 0.66 MB/sec execute 291 sec latency 25.064 ms oleg256-client.virtnet: 1 39658 0.65 MB/sec execute 292 sec latency 27.670 ms oleg256-client.virtnet: 1 39829 0.65 MB/sec execute 293 sec latency 51.105 ms oleg256-client.virtnet: 1 39969 0.65 MB/sec execute 294 sec latency 39.328 ms oleg256-client.virtnet: 1 40148 0.65 MB/sec execute 295 sec latency 30.196 ms oleg256-client.virtnet: 1 40308 0.65 MB/sec execute 296 sec latency 30.596 ms oleg256-client.virtnet: 1 40531 0.66 MB/sec execute 297 sec latency 40.674 ms oleg256-client.virtnet: 1 40689 0.66 MB/sec execute 298 sec latency 27.612 ms oleg256-client.virtnet: 1 40845 0.66 MB/sec execute 299 sec latency 36.299 ms oleg256-client.virtnet: 1 cleanup 300 sec oleg256-client.virtnet: 0 cleanup 301 sec oleg256-client.virtnet: oleg256-client.virtnet: Operation Count AvgLat MaxLat oleg256-client.virtnet: ---------------------------------------- oleg256-client.virtnet: NTCreateX 6401 27.220 23442.304 oleg256-client.virtnet: Close 4734 3.547 26.263 oleg256-client.virtnet: Rename 271 23.601 74.572 oleg256-client.virtnet: Unlink 1295 5.387 20.309 oleg256-client.virtnet: Qpathinfo 5840 9.059 18788.443 oleg256-client.virtnet: Qfileinfo 1013 0.311 11.729 oleg256-client.virtnet: Qfsinfo 1053 0.568 18.370 oleg256-client.virtnet: Sfileinfo 534 62.366 21503.296 oleg256-client.virtnet: Find 2236 2.888 4603.861 oleg256-client.virtnet: WriteX 3162 2.477 54.166 oleg256-client.virtnet: ReadX 9937 0.052 26.919 oleg256-client.virtnet: LockX 20 3.213 6.074 oleg256-client.virtnet: UnlockX 20 3.085 5.164 oleg256-client.virtnet: Flush 460 21.507 924.633 oleg256-client.virtnet: oleg256-client.virtnet: Throughput 0.657066 MB/sec 1 clients 1 procs max_latency=23442.326 ms oleg256-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg256-client.virtnet at Thu May 1 17:03:07 EDT 2025 with return code 0 oleg256-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: /mnt/lustre/d70b.replay-single/oleg256-client.virtnet /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg256-client.virtnet: removed directory: 'clients/client0' oleg256-client.virtnet: removed directory: 'clients' oleg256-client.virtnet: removed 'client.txt' oleg256-client.virtnet: /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: dbench successfully finished