== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 18:17:41 (1746137861) Starting client oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre Started clients oleg338-client.virtnet: 192.168.203.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg338-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14075 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3556144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7161596 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 18:17:51 (1746137871) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds1 to oleg338-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 18:18:06 (1746137886) targets are mounted 18:18:06 (1746137886) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26412 3568896 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1880 3594104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28292 7163000 1% /mnt/lustre oleg338-client.virtnet: looking for dbench program oleg338-client.virtnet: /usr/bin/dbench oleg338-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg338-client.virtnet oleg338-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg338-client.virtnet' oleg338-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg338-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg338-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg338-client.virtnet at Thu May 1 18:17:42 EDT 2025 oleg338-client.virtnet: waiting for dbench pid 14099 oleg338-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg338-client.virtnet: oleg338-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg338-client.virtnet: failed to create barrier semaphore oleg338-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg338-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg338-client.virtnet: releasing clients oleg338-client.virtnet: 1 197 7.84 MB/sec warmup 1 sec latency 42.139 ms oleg338-client.virtnet: 1 374 6.65 MB/sec warmup 2 sec latency 39.693 ms oleg338-client.virtnet: 1 504 5.82 MB/sec warmup 3 sec latency 128.204 ms oleg338-client.virtnet: 1 736 5.33 MB/sec warmup 4 sec latency 36.431 ms oleg338-client.virtnet: 1 930 4.35 MB/sec warmup 5 sec latency 28.178 ms oleg338-client.virtnet: 1 1201 3.84 MB/sec warmup 6 sec latency 25.166 ms oleg338-client.virtnet: 1 1508 3.72 MB/sec warmup 7 sec latency 22.962 ms oleg338-client.virtnet: 1 1696 3.28 MB/sec warmup 8 sec latency 28.758 ms oleg338-client.virtnet: 1 1884 2.93 MB/sec warmup 9 sec latency 39.651 ms oleg338-client.virtnet: 1 2124 2.67 MB/sec warmup 10 sec latency 33.506 ms oleg338-client.virtnet: 1 2389 2.54 MB/sec warmup 11 sec latency 25.321 ms oleg338-client.virtnet: 1 2765 2.66 MB/sec warmup 12 sec latency 25.497 ms oleg338-client.virtnet: 1 3347 2.72 MB/sec warmup 13 sec latency 20.392 ms oleg338-client.virtnet: 1 3769 2.84 MB/sec warmup 14 sec latency 21.940 ms oleg338-client.virtnet: 1 3949 2.67 MB/sec warmup 15 sec latency 25.575 ms oleg338-client.virtnet: 1 4083 2.50 MB/sec warmup 16 sec latency 275.565 ms oleg338-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 1275.710 ms oleg338-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 2275.868 ms oleg338-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 3276.155 ms oleg338-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 4276.338 ms oleg338-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 5276.608 ms oleg338-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 6277.082 ms oleg338-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 7277.336 ms oleg338-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 8277.512 ms oleg338-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 9277.793 ms oleg338-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 10277.923 ms oleg338-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 11278.419 ms oleg338-client.virtnet: 1 4214 1.43 MB/sec warmup 28 sec latency 11827.935 ms oleg338-client.virtnet: 1 4415 1.40 MB/sec warmup 29 sec latency 30.895 ms oleg338-client.virtnet: 1 4618 1.40 MB/sec warmup 30 sec latency 29.835 ms oleg338-client.virtnet: 1 4808 1.36 MB/sec warmup 31 sec latency 32.868 ms oleg338-client.virtnet: 1 5072 1.41 MB/sec warmup 32 sec latency 29.050 ms oleg338-client.virtnet: 1 5245 1.37 MB/sec warmup 33 sec latency 33.822 ms oleg338-clitest_70b fail mds2 2 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server 18:18:21 (1746137901) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 18:18:39 (1746137919) targets are mounted 18:18:39 (1746137919) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3566824 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12188 3593784 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7160608 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 18:18:53 (1746137933) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server ent.virtnet: 1 5364 1.34 MB/sec warmup 34 sec latency 46.812 ms oleg338-client.virtnet: 1 5519 1.30 MB/sec warmup 35 sec latency 234.582 ms oleg338-client.virtnet: 1 5718 1.27 MB/sec warmup 36 sec latency 29.845 ms oleg338-client.virtnet: 1 5916 1.27 MB/sec warmup 37 sec latency 41.420 ms oleg338-client.virtnet: 1 6014 1.24 MB/sec warmup 38 sec latency 164.631 ms oleg338-client.virtnet: 1 6014 1.21 MB/sec warmup 39 sec latency 1165.554 ms oleg338-client.virtnet: 1 6014 1.18 MB/sec warmup 40 sec latency 2166.118 ms oleg338-client.virtnet: 1 6014 1.15 MB/sec warmup 41 sec latency 3166.310 ms oleg338-client.virtnet: 1 6014 1.12 MB/sec warmup 42 sec latency 4166.743 ms oleg338-client.virtnet: 1 6014 1.10 MB/sec warmup 43 sec latency 5166.898 ms oleg338-client.virtnet: 1 6014 1.07 MB/sec warmup 44 sec latency 6168.758 ms oleg338-client.virtnet: 1 6014 1.05 MB/sec warmup 45 sec latency 7168.924 ms oleg338-client.virtnet: 1 6014 1.03 MB/sec warmup 46 sec latency 8169.077 ms oleg338-client.virtnet: 1 6014 1.00 MB/sec warmup 47 sec latency 9169.298 ms oleg338-client.virtnet: 1 6014 0.98 MB/sec warmup 48 sec latency 10169.670 ms oleg338-client.virtnet: 1 6014 0.96 MB/sec warmup 49 sec latency 11170.100 ms oleg338-client.virtnet: 1 6014 0.94 MB/sec warmup 50 sec latency 12170.275 ms oleg338-client.virtnet: 1 6014 0.93 MB/sec warmup 51 sec latency 13173.511 ms oleg338-client.virtnet: 1 6014 0.91 MB/sec warmup 52 sec latency 14177.738 ms oleg338-client.virtnet: 1 6014 0.89 MB/sec warmup 53 sec latency 15178.045 ms oleg338-client.virtnet: 1 6014 0.87 MB/sec warmup 54 sec latency 16179.521 ms oleg338-client.virtnet: 1 6014 0.86 MB/sec warmup 55 sec latency 17179.687 ms oleg338-client.virtnet: 1 6014 0.84 MB/sec warmup 56 sec latency 18179.804 ms oleg338-client.virtnet: 1 6014 0.83 MB/sec warmup 57 sec latency 19179.979 ms oleg338-client.virtnet: 1 6014 0.81 MB/sec warmup 58 sec latency 20180.151 ms oleg338-client.virtnet: 1 6014 0.80 MB/sec warmup 59 sec latency 21180.300 ms oleg338-client.virtnet: 1 6466 4.24 MB/sec execute 1 sec latency 21755.565 ms oleg338-client.virtnet: 1 6762 2.69 MB/sec execute 2 sec latency 63.826 ms oleg338-client.virtnet: 1 7122 3.17 MB/sec execute 3 sec latency 28.228 ms oleg338-client.virtnet: 1 7329 2.71 MB/sec execute 4 sec latency 28.228 ms oleg338-client.virtnet: 1 7451 2.23 MB/sec execute 5 sec latency 39.065 ms oleg338-client.virtnet: 1 7589 1.86 MB/sec execute 6 sec latency 28.972 ms oleg338-client.virtnet: 1 7778 1.62 MB/sec execute 7 sec latency 36.280 ms oleg338-client.virtnet: 1 7882 1.45 MB/sec execute 8 sec latency 42.712 ms oleg338-client.virtnet: 1 8026 1.33 MB/sec execute 9 sec latency 33.049 ms oleg338-client.virtnet: 1 8173 1.31 MB/sec execute 10 sec latency 44.256 ms oleg338-client.virtnet: 1 8352 1.21 MB/sec execute 11 sec latency 29.858 ms oleg338-client.virtnet: 1 8577 1.35 MB/sec execute 12 sec latency 31.985 ms oleg338-client.virtnet: 1 8794 1.26 MB/sec execute 13 sec latency 25.325 ms oleg338-client.virtnet: 1 8959 1.18 MB/sec execute 14 sec latency 24.291 ms oleg338-client.virtnet: 1 9204 1.12 MB/sec execute 15 sec latency 27.982 ms oleg338-client.virtnet: 1 9473 1.13 MB/sec execute 16 sec latency 23.970 ms oleg338-client.virtnet: 1 9698 1.13 MB/sec execute 17 sec latency 29.686 ms oleg338-client.virtnet: 1 10125 1.32 MB/sec execute 18 sec latency 24.566 ms oleg338-client.virtnet: 1 10718 1.52 MB/sec execute 19 seFailover mds1 to oleg338-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 18:19:09 (1746137949) targets are mounted 18:19:09 (1746137949) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2072 1285616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37596 3566364 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12020 3593964 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160328 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server 18:19:24 (1746137964) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 18:19:40 (1746137980) targets are mounted 18:19:40 (1746137980) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c latency 19.354 ms oleg338-client.virtnet: 1 10951 1.50 MB/sec execute 20 sec latency 22.603 ms oleg338-client.virtnet: 1 11155 1.44 MB/sec execute 21 sec latency 21.611 ms oleg338-client.virtnet: 1 11445 1.40 MB/sec execute 22 sec latency 18.504 ms oleg338-client.virtnet: 1 11648 1.36 MB/sec execute 23 sec latency 30.738 ms oleg338-client.virtnet: 1 11881 1.35 MB/sec execute 24 sec latency 31.304 ms oleg338-client.virtnet: 1 12149 1.41 MB/sec execute 25 sec latency 43.000 ms oleg338-client.virtnet: 1 12346 1.37 MB/sec execute 26 sec latency 27.633 ms oleg338-client.virtnet: 1 12495 1.32 MB/sec execute 27 sec latency 37.520 ms oleg338-client.virtnet: 1 12749 1.28 MB/sec execute 28 sec latency 26.477 ms oleg338-client.virtnet: 1 12999 1.28 MB/sec execute 29 sec latency 27.388 ms oleg338-client.virtnet: 1 13144 1.27 MB/sec execute 30 sec latency 31.731 ms oleg338-client.virtnet: 1 13531 1.34 MB/sec execute 31 sec latency 29.501 ms oleg338-client.virtnet: 1 13849 1.36 MB/sec execute 32 sec latency 34.151 ms oleg338-client.virtnet: 1 14195 1.45 MB/sec execute 33 sec latency 40.245 ms oleg338-client.virtnet: 1 14337 1.41 MB/sec execute 34 sec latency 39.379 ms oleg338-client.virtnet: 1 14485 1.40 MB/sec execute 35 sec latency 40.800 ms oleg338-client.virtnet: 1 14591 1.37 MB/sec execute 36 sec latency 50.437 ms oleg338-client.virtnet: 1 14702 1.33 MB/sec execute 37 sec latency 56.279 ms oleg338-client.virtnet: 1 14872 1.31 MB/sec execute 38 sec latency 75.956 ms oleg338-client.virtnet: 1 14987 1.28 MB/sec execute 39 sec latency 167.849 ms oleg338-client.virtnet: 1 15130 1.26 MB/sec execute 40 sec latency 36.614 ms oleg338-client.virtnet: 1 15329 1.25 MB/sec execute 41 sec latency 41.620 ms oleg338-client.virtnet: 1 15350 1.22 MB/sec execute 42 sec latency 858.669 ms oleg338-client.virtnet: 1 15350 1.20 MB/sec execute 43 sec latency 1859.158 ms oleg338-client.virtnet: 1 15350 1.17 MB/sec execute 44 sec latency 2859.414 ms oleg338-client.virtnet: 1 15350 1.14 MB/sec execute 45 sec latency 3859.575 ms oleg338-client.virtnet: 1 15350 1.12 MB/sec execute 46 sec latency 4859.730 ms oleg338-client.virtnet: 1 15350 1.09 MB/sec execute 47 sec latency 5859.973 ms oleg338-client.virtnet: 1 15350 1.07 MB/sec execute 48 sec latency 6860.159 ms oleg338-client.virtnet: 1 15350 1.05 MB/sec execute 49 sec latency 7860.347 ms oleg338-client.virtnet: 1 15350 1.03 MB/sec execute 50 sec latency 8860.967 ms oleg338-client.virtnet: 1 15350 1.01 MB/sec execute 51 sec latency 9862.083 ms oleg338-client.virtnet: 1 15350 0.99 MB/sec execute 52 sec latency 10862.334 ms oleg338-client.virtnet: 1 15350 0.97 MB/sec execute 53 sec latency 11862.802 ms oleg338-client.virtnet: 1 15350 0.95 MB/sec execute 54 sec latency 12862.973 ms oleg338-client.virtnet: 1 15350 0.93 MB/sec execute 55 sec latency 13863.234 ms oleg338-client.virtnet: 1 15350 0.92 MB/sec execute 56 sec latency 14863.984 ms oleg338-client.virtnet: 1 15350 0.90 MB/sec execute 57 sec latency 15864.278 ms oleg338-client.virtnet: 1 15350 0.89 MB/sec execute 58 sec latency 16864.641 ms oleg338-client.virtnet: 1 15350 0.87 MB/sec execute 59 sec latency 17865.686 ms oleg338-client.virtnet: 1 15350 0.86 MB/sec execute 60 sec latency 18866.101 ms oleg338-client.virtnet: 1 15350 0.84 MB/sec execute 61 sec latency 19867.069 ms oleg338-client.virtnet: 1 15379 0.83 MB/sec execute 62 sec latency 20639.816 ms oleg338-client.virtnet: 1 15612 0.86 MB/sec execute 63 sec latency 31.117 ms oleg338-client.virUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37608 3568944 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11908 3594244 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49516 7163188 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 18:19:56 (1746137996) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds1 to oleg338-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 18:20:12 (1746138012) targets are mounted 18:20:12 (1746138012) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569116 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12072 3593428 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7162544 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server 18:20:27 (1746138027) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server tnet: 1 15757 0.85 MB/sec execute 64 sec latency 44.330 ms oleg338-client.virtnet: 1 15924 0.84 MB/sec execute 65 sec latency 28.676 ms oleg338-client.virtnet: 1 16103 0.83 MB/sec execute 66 sec latency 29.752 ms oleg338-client.virtnet: 1 16301 0.82 MB/sec execute 67 sec latency 37.013 ms oleg338-client.virtnet: 1 16430 0.81 MB/sec execute 68 sec latency 38.177 ms oleg338-client.virtnet: 1 16585 0.82 MB/sec execute 69 sec latency 68.332 ms oleg338-client.virtnet: 1 16693 0.82 MB/sec execute 70 sec latency 97.514 ms oleg338-client.virtnet: 1 17096 0.86 MB/sec execute 71 sec latency 29.224 ms oleg338-client.virtnet: 1 17469 0.89 MB/sec execute 72 sec latency 34.481 ms oleg338-client.virtnet: 1 17775 0.92 MB/sec execute 73 sec latency 37.891 ms oleg338-client.virtnet: 1 17969 0.92 MB/sec execute 74 sec latency 25.677 ms oleg338-client.virtnet: 1 18139 0.91 MB/sec execute 75 sec latency 25.445 ms oleg338-client.virtnet: 1 18259 0.90 MB/sec execute 76 sec latency 27.428 ms oleg338-client.virtnet: 1 18266 0.89 MB/sec execute 77 sec latency 970.875 ms oleg338-client.virtnet: 1 18266 0.88 MB/sec execute 78 sec latency 1971.553 ms oleg338-client.virtnet: 1 18266 0.87 MB/sec execute 79 sec latency 2971.758 ms oleg338-client.virtnet: 1 18266 0.86 MB/sec execute 80 sec latency 3971.919 ms oleg338-client.virtnet: 1 18266 0.85 MB/sec execute 81 sec latency 4972.070 ms oleg338-client.virtnet: 1 18266 0.84 MB/sec execute 82 sec latency 5973.393 ms oleg338-client.virtnet: 1 18266 0.83 MB/sec execute 83 sec latency 6973.661 ms oleg338-client.virtnet: 1 18266 0.82 MB/sec execute 84 sec latency 7973.838 ms oleg338-client.virtnet: 1 18266 0.81 MB/sec execute 85 sec latency 8978.436 ms oleg338-client.virtnet: 1 18266 0.80 MB/sec execute 86 sec latency 9979.096 ms oleg338-client.virtnet: 1 18266 0.79 MB/sec execute 87 sec latency 10979.690 ms oleg338-client.virtnet: 1 18266 0.78 MB/sec execute 88 sec latency 11979.844 ms oleg338-client.virtnet: 1 18266 0.77 MB/sec execute 89 sec latency 12979.988 ms oleg338-client.virtnet: 1 18266 0.76 MB/sec execute 90 sec latency 13982.071 ms oleg338-client.virtnet: 1 18266 0.75 MB/sec execute 91 sec latency 14982.293 ms oleg338-client.virtnet: 1 18266 0.74 MB/sec execute 92 sec latency 15982.731 ms oleg338-client.virtnet: 1 18266 0.74 MB/sec execute 93 sec latency 16983.135 ms oleg338-client.virtnet: 1 18430 0.73 MB/sec execute 94 sec latency 17279.434 ms oleg338-client.virtnet: 1 18554 0.73 MB/sec execute 95 sec latency 59.144 ms oleg338-client.virtnet: 1 18743 0.72 MB/sec execute 96 sec latency 25.706 ms oleg338-client.virtnet: 1 18983 0.73 MB/sec execute 97 sec latency 24.728 ms oleg338-client.virtnet: 1 19243 0.75 MB/sec execute 98 sec latency 29.364 ms oleg338-client.virtnet: 1 19396 0.74 MB/sec execute 99 sec latency 29.280 ms oleg338-client.virtnet: 1 19529 0.74 MB/sec execute 100 sec latency 55.679 ms oleg338-client.virtnet: 1 19705 0.73 MB/sec execute 101 sec latency 94.485 ms oleg338-client.virtnet: 1 19939 0.73 MB/sec execute 102 sec latency 25.145 ms oleg338-client.virtnet: 1 20163 0.73 MB/sec execute 103 sec latency 34.460 ms oleg338-client.virtnet: 1 20235 0.73 MB/sec execute 104 sec latency 520.924 ms oleg338-client.virtnet: 1 20235 0.72 MB/sec execute 105 sec latency 1521.117 ms oleg338-client.virtnet: 1 20235 0.71 MB/sec execute 106 sec latency 2521.353 ms oleg338-client.virtnet: 1 20235 0.71 MB/sec execute 107 sec latency 3521.559 ms oleg338-client.virtnet: 1 20235 0.70 MB/sec execFailover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 18:20:42 (1746138042) targets are mounted 18:20:42 (1746138042) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3568872 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12200 3593196 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7162068 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 18:20:55 (1746138055) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds1 to oleg338-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 18:21:11 (1746138071) targets are mounted 18:21:11 (1746138071) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid ute 108 sec latency 4521.777 ms oleg338-client.virtnet: 1 20235 0.69 MB/sec execute 109 sec latency 5521.934 ms oleg338-client.virtnet: 1 20235 0.69 MB/sec execute 110 sec latency 6522.158 ms oleg338-client.virtnet: 1 20235 0.68 MB/sec execute 111 sec latency 7522.474 ms oleg338-client.virtnet: 1 20235 0.68 MB/sec execute 112 sec latency 8523.078 ms oleg338-client.virtnet: 1 20235 0.67 MB/sec execute 113 sec latency 9523.849 ms oleg338-client.virtnet: 1 20235 0.66 MB/sec execute 114 sec latency 10524.670 ms oleg338-client.virtnet: 1 20235 0.66 MB/sec execute 115 sec latency 11524.798 ms oleg338-client.virtnet: 1 20235 0.65 MB/sec execute 116 sec latency 12525.671 ms oleg338-client.virtnet: 1 20235 0.65 MB/sec execute 117 sec latency 13526.022 ms oleg338-client.virtnet: 1 20235 0.64 MB/sec execute 118 sec latency 14526.239 ms oleg338-client.virtnet: 1 20235 0.64 MB/sec execute 119 sec latency 15526.431 ms oleg338-client.virtnet: 1 20235 0.63 MB/sec execute 120 sec latency 16526.647 ms oleg338-client.virtnet: 1 20235 0.63 MB/sec execute 121 sec latency 17526.827 ms oleg338-client.virtnet: 1 20235 0.62 MB/sec execute 122 sec latency 18527.836 ms oleg338-client.virtnet: 1 20235 0.62 MB/sec execute 123 sec latency 19528.000 ms oleg338-client.virtnet: 1 20499 0.64 MB/sec execute 124 sec latency 19825.907 ms oleg338-client.virtnet: 1 20992 0.65 MB/sec execute 125 sec latency 30.305 ms oleg338-client.virtnet: 1 21465 0.69 MB/sec execute 126 sec latency 16.676 ms oleg338-client.virtnet: 1 21759 0.69 MB/sec execute 127 sec latency 19.160 ms oleg338-client.virtnet: 1 22044 0.69 MB/sec execute 128 sec latency 15.645 ms oleg338-client.virtnet: 1 22289 0.69 MB/sec execute 129 sec latency 18.866 ms oleg338-client.virtnet: 1 22599 0.69 MB/sec execute 130 sec latency 19.503 ms oleg338-client.virtnet: 1 22983 0.71 MB/sec execute 131 sec latency 14.117 ms oleg338-client.virtnet: 1 23267 0.71 MB/sec execute 132 sec latency 24.738 ms oleg338-client.virtnet: 1 23522 0.70 MB/sec execute 133 sec latency 19.241 ms oleg338-client.virtnet: 1 23780 0.71 MB/sec execute 134 sec latency 26.107 ms oleg338-client.virtnet: 1 24255 0.74 MB/sec execute 135 sec latency 25.862 ms oleg338-client.virtnet: 1 24818 0.77 MB/sec execute 136 sec latency 22.268 ms oleg338-client.virtnet: 1 25053 0.77 MB/sec execute 137 sec latency 28.963 ms oleg338-client.virtnet: 1 25259 0.77 MB/sec execute 138 sec latency 23.853 ms oleg338-client.virtnet: 1 25498 0.77 MB/sec execute 139 sec latency 28.231 ms oleg338-client.virtnet: 1 25763 0.77 MB/sec execute 140 sec latency 24.641 ms oleg338-client.virtnet: 1 26021 0.77 MB/sec execute 141 sec latency 26.939 ms oleg338-client.virtnet: 1 26307 0.79 MB/sec execute 142 sec latency 23.772 ms oleg338-client.virtnet: 1 26494 0.78 MB/sec execute 143 sec latency 25.415 ms oleg338-client.virtnet: 1 26668 0.78 MB/sec execute 144 sec latency 31.334 ms oleg338-client.virtnet: 1 26861 0.77 MB/sec execute 145 sec latency 30.338 ms oleg338-client.virtnet: 1 27058 0.77 MB/sec execute 146 sec latency 37.340 ms oleg338-client.virtnet: 1 27293 0.77 MB/sec execute 147 sec latency 30.099 ms oleg338-client.virtnet: 1 27471 0.78 MB/sec execute 148 sec latency 37.789 ms oleg338-client.virtnet: 1 27802 0.80 MB/sec execute 149 sec latency 44.833 ms oleg338-client.virtnet: 1 28108 0.80 MB/sec execute 150 sec latency 50.355 ms oleg338-client.virtnet: 1 28430 0.82 MB/sec execute 151 sec latency 30.700 ms oleg338-client.virtnet: 1 28584 0.82 MB/sec execute 152 sec latency 49.637 ms oleg338-client.vmdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2072 1285616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3568112 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12028 3593412 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49452 7161524 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server 18:21:26 (1746138086) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 18:21:42 (1746138102) targets are mounted 18:21:42 (1746138102) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37604 3566468 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12020 3594048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7160516 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 18:21:56 (1746138116) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server irtnet: 1 28743 0.82 MB/sec execute 153 sec latency 32.395 ms oleg338-client.virtnet: 1 28886 0.82 MB/sec execute 154 sec latency 30.463 ms oleg338-client.virtnet: 1 29100 0.81 MB/sec execute 155 sec latency 44.787 ms oleg338-client.virtnet: 1 29248 0.81 MB/sec execute 156 sec latency 41.146 ms oleg338-client.virtnet: 1 29391 0.81 MB/sec execute 157 sec latency 41.373 ms oleg338-client.virtnet: 1 29548 0.81 MB/sec execute 158 sec latency 55.284 ms oleg338-client.virtnet: 1 29674 0.81 MB/sec execute 159 sec latency 50.478 ms oleg338-client.virtnet: 1 29875 0.82 MB/sec execute 160 sec latency 53.362 ms oleg338-client.virtnet: 1 30007 0.81 MB/sec execute 161 sec latency 41.872 ms oleg338-client.virtnet: 1 30142 0.81 MB/sec execute 162 sec latency 46.180 ms oleg338-client.virtnet: 1 30220 0.81 MB/sec execute 163 sec latency 354.107 ms oleg338-client.virtnet: 1 30220 0.80 MB/sec execute 164 sec latency 1354.533 ms oleg338-client.virtnet: 1 30220 0.80 MB/sec execute 165 sec latency 2355.128 ms oleg338-client.virtnet: 1 30220 0.79 MB/sec execute 166 sec latency 3355.355 ms oleg338-client.virtnet: 1 30220 0.79 MB/sec execute 167 sec latency 4355.593 ms oleg338-client.virtnet: 1 30220 0.78 MB/sec execute 168 sec latency 5356.991 ms oleg338-client.virtnet: 1 30220 0.78 MB/sec execute 169 sec latency 6357.157 ms oleg338-client.virtnet: 1 30220 0.77 MB/sec execute 170 sec latency 7357.573 ms oleg338-client.virtnet: 1 30220 0.77 MB/sec execute 171 sec latency 8357.855 ms oleg338-client.virtnet: 1 30220 0.76 MB/sec execute 172 sec latency 9359.957 ms oleg338-client.virtnet: 1 30220 0.76 MB/sec execute 173 sec latency 10361.947 ms oleg338-client.virtnet: 1 30220 0.75 MB/sec execute 174 sec latency 11362.338 ms oleg338-client.virtnet: 1 30220 0.75 MB/sec execute 175 sec latency 12362.517 ms oleg338-client.virtnet: 1 30220 0.75 MB/sec execute 176 sec latency 13363.006 ms oleg338-client.virtnet: 1 30220 0.74 MB/sec execute 177 sec latency 14363.160 ms oleg338-client.virtnet: 1 30220 0.74 MB/sec execute 178 sec latency 15363.640 ms oleg338-client.virtnet: 1 30220 0.73 MB/sec execute 179 sec latency 16363.795 ms oleg338-client.virtnet: 1 30220 0.73 MB/sec execute 180 sec latency 17363.948 ms oleg338-client.virtnet: 1 30220 0.73 MB/sec execute 181 sec latency 18364.252 ms oleg338-client.virtnet: 1 30220 0.72 MB/sec execute 182 sec latency 19364.593 ms oleg338-client.virtnet: 1 30220 0.72 MB/sec execute 183 sec latency 20364.858 ms oleg338-client.virtnet: 1 30376 0.71 MB/sec execute 184 sec latency 20666.562 ms oleg338-client.virtnet: 1 30605 0.71 MB/sec execute 185 sec latency 26.841 ms oleg338-client.virtnet: 1 30873 0.72 MB/sec execute 186 sec latency 21.229 ms oleg338-client.virtnet: 1 31383 0.74 MB/sec execute 187 sec latency 22.885 ms oleg338-client.virtnet: 1 31781 0.75 MB/sec execute 188 sec latency 31.056 ms oleg338-client.virtnet: 1 32060 0.76 MB/sec execute 189 sec latency 27.967 ms oleg338-client.virtnet: 1 32220 0.76 MB/sec execute 190 sec latency 35.393 ms oleg338-client.virtnet: 1 32370 0.76 MB/sec execute 191 sec latency 26.508 ms oleg338-client.virtnet: 1 32513 0.76 MB/sec execute 192 sec latency 27.494 ms oleg338-client.virtnet: 1 32738 0.75 MB/sec execute 193 sec latency 25.506 ms oleg338-client.virtnet: 1 32904 0.75 MB/sec execute 194 sec latency 31.806 ms oleg338-client.virtnet: 1 33144 0.75 MB/sec execute 195 sec latency 28.147 ms oleg338-client.virtnet: 1 33438 0.77 MB/sec execute 196 sec latency 23.805 ms oleg338-client.virtnet: 1 33623 0.76 MB/seFailover mds1 to oleg338-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 18:22:13 (1746138133) targets are mounted 18:22:13 (1746138133) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2072 1285616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37588 3568684 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11904 3595000 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7163684 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server 18:22:27 (1746138147) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 18:22:42 (1746138162) targets are mounted 18:22:42 (1746138162) facet_failover done c execute 197 sec latency 33.744 ms oleg338-client.virtnet: 1 33810 0.76 MB/sec execute 198 sec latency 32.653 ms oleg338-client.virtnet: 1 34024 0.76 MB/sec execute 199 sec latency 26.460 ms oleg338-client.virtnet: 1 34265 0.76 MB/sec execute 200 sec latency 30.073 ms oleg338-client.virtnet: 1 34445 0.76 MB/sec execute 201 sec latency 29.157 ms oleg338-client.virtnet: 1 34851 0.78 MB/sec execute 202 sec latency 30.526 ms oleg338-client.virtnet: 1 35208 0.78 MB/sec execute 203 sec latency 32.109 ms oleg338-client.virtnet: 1 35520 0.80 MB/sec execute 204 sec latency 33.599 ms oleg338-client.virtnet: 1 35640 0.79 MB/sec execute 205 sec latency 85.043 ms oleg338-client.virtnet: 1 35790 0.80 MB/sec execute 206 sec latency 62.006 ms oleg338-client.virtnet: 1 35914 0.79 MB/sec execute 207 sec latency 33.931 ms oleg338-client.virtnet: 1 36030 0.79 MB/sec execute 208 sec latency 34.518 ms oleg338-client.virtnet: 1 36232 0.79 MB/sec execute 209 sec latency 32.825 ms oleg338-client.virtnet: 1 36369 0.78 MB/sec execute 210 sec latency 41.344 ms oleg338-client.virtnet: 1 36504 0.78 MB/sec execute 211 sec latency 34.552 ms oleg338-client.virtnet: 1 36717 0.78 MB/sec execute 212 sec latency 34.767 ms oleg338-client.virtnet: 1 37002 0.79 MB/sec execute 213 sec latency 26.183 ms oleg338-client.virtnet: 1 37194 0.79 MB/sec execute 214 sec latency 29.575 ms oleg338-client.virtnet: 1 37356 0.79 MB/sec execute 215 sec latency 26.751 ms oleg338-client.virtnet: 1 37606 0.79 MB/sec execute 216 sec latency 40.144 ms oleg338-client.virtnet: 1 37867 0.79 MB/sec execute 217 sec latency 32.944 ms oleg338-client.virtnet: 1 38103 0.79 MB/sec execute 218 sec latency 25.407 ms oleg338-client.virtnet: 1 38438 0.81 MB/sec execute 219 sec latency 40.704 ms oleg338-client.virtnet: 1 38849 0.81 MB/sec execute 220 sec latency 40.547 ms oleg338-client.virtnet: 1 39089 0.82 MB/sec execute 221 sec latency 32.939 ms oleg338-client.virtnet: 1 39277 0.82 MB/sec execute 222 sec latency 43.642 ms oleg338-client.virtnet: 1 39415 0.82 MB/sec execute 223 sec latency 49.877 ms oleg338-client.virtnet: 1 39535 0.82 MB/sec execute 224 sec latency 155.371 ms oleg338-client.virtnet: 1 39535 0.81 MB/sec execute 225 sec latency 1155.540 ms oleg338-client.virtnet: 1 39535 0.81 MB/sec execute 226 sec latency 2155.710 ms oleg338-client.virtnet: 1 39535 0.81 MB/sec execute 227 sec latency 3155.875 ms oleg338-client.virtnet: 1 39535 0.80 MB/sec execute 228 sec latency 4156.143 ms oleg338-client.virtnet: 1 39535 0.80 MB/sec execute 229 sec latency 5156.468 ms oleg338-client.virtnet: 1 39535 0.80 MB/sec execute 230 sec latency 6156.968 ms oleg338-client.virtnet: 1 39535 0.79 MB/sec execute 231 sec latency 7157.233 ms oleg338-client.virtnet: 1 39535 0.79 MB/sec execute 232 sec latency 8158.155 ms oleg338-client.virtnet: 1 39535 0.79 MB/sec execute 233 sec latency 9158.945 ms oleg338-client.virtnet: 1 39535 0.78 MB/sec execute 234 sec latency 10159.202 ms oleg338-client.virtnet: 1 39535 0.78 MB/sec execute 235 sec latency 11159.789 ms oleg338-client.virtnet: 1 39535 0.78 MB/sec execute 236 sec latency 12159.961 ms oleg338-client.virtnet: 1 39535 0.77 MB/sec execute 237 sec latency 13160.126 ms oleg338-client.virtnet: 1 39535 0.77 MB/sec execute 238 sec latency 14160.283 ms oleg338-client.virtnet: 1 39535 0.77 MB/sec execute 239 sec latency 15160.778 ms oleg338-client.virtnet: 1 39535 0.76 MB/sec execute 240 sec latency 16160.975 ms oleg338-client.virtnet: 1 39535 0.76 MB/sec execute 241 sec latency 17161.343 ms oleg338-clioleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec ent.virtnet: 1 39535 0.76 MB/sec execute 242 sec latency 18162.113 ms oleg338-client.virtnet: 1 39535 0.75 MB/sec execute 243 sec latency 19162.288 ms oleg338-client.virtnet: 1 39550 0.75 MB/sec execute 244 sec latency 20043.522 ms oleg338-client.virtnet: 1 39762 0.75 MB/sec execute 245 sec latency 123.741 ms oleg338-client.virtnet: 1 39987 0.75 MB/sec execute 246 sec latency 28.859 ms oleg338-client.virtnet: 1 40321 0.75 MB/sec execute 247 sec latency 21.965 ms oleg338-client.virtnet: 1 40694 0.76 MB/sec execute 248 sec latency 17.736 ms oleg338-client.virtnet: 1 41037 0.76 MB/sec execute 249 sec latency 15.875 ms oleg338-client.virtnet: 1 41261 0.76 MB/sec execute 250 sec latency 28.542 ms oleg338-client.virtnet: 1 41542 0.76 MB/sec execute 251 sec latency 24.019 ms oleg338-client.virtnet: 1 42328 0.78 MB/sec execute 252 sec latency 15.575 ms oleg338-client.virtnet: 1 42772 0.80 MB/sec execute 253 sec latency 18.423 ms oleg338-client.virtnet: 1 43001 0.80 MB/sec execute 254 sec latency 21.908 ms oleg338-client.virtnet: 1 43276 0.79 MB/sec execute 255 sec latency 19.658 ms oleg338-client.virtnet: 1 43515 0.79 MB/sec execute 256 sec latency 27.688 ms oleg338-client.virtnet: 1 43848 0.80 MB/sec execute 257 sec latency 19.913 ms oleg338-client.virtnet: 1 44256 0.80 MB/sec execute 258 sec latency 15.522 ms oleg338-client.virtnet: 1 44617 0.80 MB/sec execute 259 sec latency 14.703 ms oleg338-client.virtnet: 1 45040 0.81 MB/sec execute 260 sec latency 16.115 ms oleg338-client.virtnet: 1 45766 0.83 MB/sec execute 261 sec latency 17.058 ms oleg338-client.virtnet: 1 46276 0.84 MB/sec execute 262 sec latency 14.015 ms oleg338-client.virtnet: 1 46508 0.84 MB/sec execute 263 sec latency 23.361 ms oleg338-client.virtnet: 1 46816 0.84 MB/sec execute 264 sec latency 16.968 ms oleg338-client.virtnet: 1 47074 0.84 MB/sec execute 265 sec latency 19.922 ms oleg338-client.virtnet: 1 47317 0.84 MB/sec execute 266 sec latency 22.155 ms oleg338-client.virtnet: 1 47572 0.85 MB/sec execute 267 sec latency 35.994 ms oleg338-client.virtnet: 1 47750 0.85 MB/sec execute 268 sec latency 27.869 ms oleg338-client.virtnet: 1 47958 0.84 MB/sec execute 269 sec latency 24.578 ms oleg338-client.virtnet: 1 48245 0.84 MB/sec execute 270 sec latency 21.412 ms oleg338-client.virtnet: 1 48566 0.84 MB/sec execute 271 sec latency 18.838 ms oleg338-client.virtnet: 1 49040 0.86 MB/sec execute 272 sec latency 28.048 ms oleg338-client.virtnet: 1 49540 0.86 MB/sec execute 273 sec latency 26.700 ms oleg338-client.virtnet: 1 49908 0.88 MB/sec execute 274 sec latency 22.927 ms oleg338-client.virtnet: 1 50106 0.87 MB/sec execute 275 sec latency 25.401 ms oleg338-client.virtnet: 1 50389 0.87 MB/sec execute 276 sec latency 32.029 ms oleg338-client.virtnet: 1 50609 0.87 MB/sec execute 277 sec latency 39.249 ms oleg338-client.virtnet: 1 50888 0.87 MB/sec execute 278 sec latency 21.990 ms oleg338-client.virtnet: 1 51183 0.88 MB/sec execute 279 sec latency 30.849 ms oleg338-client.virtnet: 1 51380 0.88 MB/sec execute 280 sec latency 25.134 ms oleg338-client.virtnet: 1 51620 0.88 MB/sec execute 281 sec latency 21.132 ms oleg338-client.virtnet: 1 51986 0.88 MB/sec execute 282 sec latency 18.037 ms oleg338-client.virtnet: 1 52300 0.88 MB/sec execute 283 sec latency 19.326 ms oleg338-client.virtnet: 1 52897 0.89 MB/sec execute 284 sec latency 30.404 ms oleg338-client.virtnet: 1 53267 0.91 MB/sec execute 285 sec latency 28.212 ms oleg338-client.virtnet: 1 53509 0.91 MB/sec execute 286 sec latency 26.575 ms oleg338-client.virtnet: 1 53694 0.90 MB/sec execute 287 sec latency 23.421 ms oleg338-client.virtnet: 1 53960 0.90 MB/sec execute 288 sec latency 24.164 ms oleg338-client.virtnet: 1 54205 0.90 MB/sec execute 289 sec latency 21.533 ms oleg338-client.virtnet: 1 54443 0.90 MB/sec execute 290 sec latency 23.691 ms oleg338-client.virtnet: 1 54746 0.91 MB/sec execute 291 sec latency 28.493 ms oleg338-client.virtnet: 1 54931 0.91 MB/sec execute 292 sec latency 44.663 ms oleg338-client.virtnet: 1 55063 0.90 MB/sec execute 293 sec latency 46.219 ms oleg338-client.virtnet: 1 55254 0.90 MB/sec execute 294 sec latency 33.376 ms oleg338-client.virtnet: 1 55436 0.90 MB/sec execute 295 sec latency 29.263 ms oleg338-client.virtnet: 1 55666 0.90 MB/sec execute 296 sec latency 26.783 ms oleg338-client.virtnet: 1 55999 0.91 MB/sec execute 297 sec latency 37.948 ms oleg338-client.virtnet: 1 56481 0.92 MB/sec execute 298 sec latency 24.802 ms oleg338-client.virtnet: 1 56899 0.93 MB/sec execute 299 sec latency 26.672 ms oleg338-client.virtnet: 1 cleanup 300 sec oleg338-client.virtnet: 0 cleanup 301 sec oleg338-client.virtnet: oleg338-client.virtnet: Operation Count AvgLat MaxLat oleg338-client.virtnet: ---------------------------------------- oleg338-client.virtnet: NTCreateX 8788 17.622 20666.551 oleg338-client.virtnet: Close 6441 5.755 19825.891 oleg338-client.virtnet: Rename 375 74.404 20639.790 oleg338-client.virtnet: Unlink 1777 4.365 32.854 oleg338-client.virtnet: Qpathinfo 7942 5.396 17279.404 oleg338-client.virtnet: Qfileinfo 1391 0.237 5.618 oleg338-client.virtnet: Qfsinfo 1484 0.442 20.808 oleg338-client.virtnet: Sfileinfo 702 12.235 64.456 oleg338-client.virtnet: Find 3092 0.638 32.075 oleg338-client.virtnet: WriteX 4348 1.893 23.900 oleg338-client.virtnet: ReadX 14011 0.033 12.156 oleg338-client.virtnet: LockX 30 3.047 9.750 oleg338-client.virtnet: UnlockX 30 2.866 9.117 oleg338-client.virtnet: Flush 611 13.813 167.836 oleg338-client.virtnet: oleg338-client.virtnet: Throughput 0.927805 MB/sec 1 clients 1 procs max_latency=21755.565 ms oleg338-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg338-client.virtnet at Thu May 1 18:23:43 EDT 2025 with return code 0 oleg338-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg338-client.virtnet oleg338-client.virtnet: /mnt/lustre/d70b.replay-single/oleg338-client.virtnet /mnt/lustre/d70b.replay-single/oleg338-client.virtnet oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg338-client.virtnet: removed directory: 'clients/client0' oleg338-client.virtnet: removed directory: 'clients' oleg338-client.virtnet: removed 'client.txt' oleg338-client.virtnet: /mnt/lustre/d70b.replay-single/oleg338-client.virtnet oleg338-client.virtnet: dbench successfully finished