== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 21:12:43 (1746148363) Starting client oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre Started clients oleg203-client.virtnet: 192.168.202.103@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg203-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14092 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26160 3556140 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3594048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7150188 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 21:12:53 (1746148373) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 21:13:09 (1746148389) targets are mounted 21:13:09 (1746148389) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 27448 3568912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5564 3594388 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 33012 7163300 1% /mnt/lustre oleg203-client.virtnet: looking for dbench program oleg203-client.virtnet: /usr/bin/dbench oleg203-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg203-client.virtnet oleg203-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg203-client.virtnet' oleg203-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg203-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg203-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg203-client.virtnet at Thu May 1 21:12:44 EDT 2025 oleg203-client.virtnet: waiting for dbench pid 14117 oleg203-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg203-client.virtnet: oleg203-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg203-client.virtnet: failed to create barrier semaphore oleg203-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg203-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg203-client.virtnet: releasing clients oleg203-client.virtnet: 1 182 7.32 MB/sec warmup 1 sec latency 45.676 ms oleg203-client.virtnet: 1 389 6.91 MB/sec warmup 2 sec latency 33.111 ms oleg203-client.virtnet: 1 603 6.99 MB/sec warmup 3 sec latency 52.725 ms oleg203-client.virtnet: 1 738 5.33 MB/sec warmup 4 sec latency 47.021 ms oleg203-client.virtnet: 1 857 4.31 MB/sec warmup 5 sec latency 51.530 ms oleg203-client.virtnet: 1 959 3.64 MB/sec warmup 6 sec latency 63.720 ms oleg203-client.virtnet: 1 1164 3.29 MB/sec warmup 7 sec latency 31.335 ms oleg203-client.virtnet: 1 1417 3.25 MB/sec warmup 8 sec latency 31.558 ms oleg203-client.virtnet: 1 1629 2.91 MB/sec warmup 9 sec latency 32.774 ms oleg203-client.virtnet: 1 1864 2.64 MB/sec warmup 10 sec latency 26.620 ms oleg203-client.virtnet: 1 2150 2.43 MB/sec warmup 11 sec latency 22.783 ms oleg203-client.virtnet: 1 2428 2.34 MB/sec warmup 12 sec latency 24.322 ms oleg203-client.virtnet: 1 2875 2.48 MB/sec warmup 13 sec latency 28.502 ms oleg203-client.virtnet: 1 3339 2.53 MB/sec warmup 14 sec latency 35.824 ms oleg203-client.virtnet: 1 3683 2.58 MB/sec warmup 15 sec latency 22.715 ms oleg203-client.virtnet: 1 3899 2.50 MB/sec warmup 16 sec latency 26.645 ms oleg203-client.virtnet: 1 4055 2.36 MB/sec warmup 17 sec latency 30.083 ms oleg203-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 882.858 ms oleg203-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 1883.474 ms oleg203-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 2884.669 ms oleg203-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 3884.822 ms oleg203-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 4884.992 ms oleg203-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 5885.184 ms oleg203-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 6886.225 ms oleg203-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 7886.491 ms oleg203-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 8888.139 ms oleg203-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 9888.331 ms oleg203-client.virtnet: 1 4083 1.43 MB/sec warmup 28 sec latency 10888.684 ms oleg203-client.virtnet: 1 4301 1.39 MB/sec warmup 29 sec latency 11024.062 ms oleg203-client.virtnet: 1 4473 1.36 MB/sec warmup 30 sec latency 31.676 ms oleg203-client.virtnet: 1 4713 1.36 MB/sec warmup 31 sec latency 39.881 ms oleg203-client.virtnet: 1 5012 1.41 MB/sec warmup 32 sec latency 23.662 ms oleg203-client.virtnet: 1 5212 1.37 MB/sec warmup 33 sec latency 24.314 ms oleg203-client.test_70b fail mds2 2 times Failing mds2 on oleg203-server Stopping /mnt/lustre-mds2 (opts:) on oleg203-server 21:13:23 (1746148403) shut down facet: mds2 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds2 to oleg203-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0001 21:13:39 (1746148419) targets are mounted 21:13:39 (1746148419) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37528 3567292 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12096 3593316 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7160608 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 21:13:55 (1746148435) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server virtnet: 1 5397 1.34 MB/sec warmup 34 sec latency 27.588 ms oleg203-client.virtnet: 1 5538 1.30 MB/sec warmup 35 sec latency 554.777 ms oleg203-client.virtnet: 1 5855 1.30 MB/sec warmup 36 sec latency 28.427 ms oleg203-client.virtnet: 1 6037 1.29 MB/sec warmup 37 sec latency 52.573 ms oleg203-client.virtnet: 1 6161 1.27 MB/sec warmup 38 sec latency 464.067 ms oleg203-client.virtnet: 1 6161 1.23 MB/sec warmup 39 sec latency 1464.394 ms oleg203-client.virtnet: 1 6161 1.20 MB/sec warmup 40 sec latency 2464.587 ms oleg203-client.virtnet: 1 6161 1.17 MB/sec warmup 41 sec latency 3465.258 ms oleg203-client.virtnet: 1 6161 1.15 MB/sec warmup 42 sec latency 4466.415 ms oleg203-client.virtnet: 1 6161 1.12 MB/sec warmup 43 sec latency 5467.765 ms oleg203-client.virtnet: 1 6161 1.09 MB/sec warmup 44 sec latency 6467.949 ms oleg203-client.virtnet: 1 6161 1.07 MB/sec warmup 45 sec latency 7468.459 ms oleg203-client.virtnet: 1 6161 1.05 MB/sec warmup 46 sec latency 8468.784 ms oleg203-client.virtnet: 1 6161 1.02 MB/sec warmup 47 sec latency 9468.953 ms oleg203-client.virtnet: 1 6161 1.00 MB/sec warmup 48 sec latency 10469.124 ms oleg203-client.virtnet: 1 6161 0.98 MB/sec warmup 49 sec latency 11471.158 ms oleg203-client.virtnet: 1 6161 0.96 MB/sec warmup 50 sec latency 12471.405 ms oleg203-client.virtnet: 1 6161 0.94 MB/sec warmup 51 sec latency 13471.639 ms oleg203-client.virtnet: 1 6161 0.92 MB/sec warmup 52 sec latency 14471.804 ms oleg203-client.virtnet: 1 6161 0.91 MB/sec warmup 53 sec latency 15471.972 ms oleg203-client.virtnet: 1 6161 0.89 MB/sec warmup 54 sec latency 16472.608 ms oleg203-client.virtnet: 1 6161 0.87 MB/sec warmup 55 sec latency 17475.388 ms oleg203-client.virtnet: 1 6161 0.86 MB/sec warmup 56 sec latency 18475.773 ms oleg203-client.virtnet: 1 6161 0.84 MB/sec warmup 57 sec latency 19476.958 ms oleg203-client.virtnet: 1 6161 0.83 MB/sec warmup 58 sec latency 20477.175 ms oleg203-client.virtnet: 1 6457 0.88 MB/sec warmup 59 sec latency 20861.498 ms oleg203-client.virtnet: 1 7128 4.14 MB/sec execute 1 sec latency 46.741 ms oleg203-client.virtnet: 1 7360 2.74 MB/sec execute 2 sec latency 42.563 ms oleg203-client.virtnet: 1 7536 1.93 MB/sec execute 3 sec latency 24.445 ms oleg203-client.virtnet: 1 7695 1.45 MB/sec execute 4 sec latency 33.233 ms oleg203-client.virtnet: 1 7871 1.24 MB/sec execute 5 sec latency 38.584 ms oleg203-client.virtnet: 1 7997 1.08 MB/sec execute 6 sec latency 49.800 ms oleg203-client.virtnet: 1 8151 1.10 MB/sec execute 7 sec latency 39.649 ms oleg203-client.virtnet: 1 8319 0.99 MB/sec execute 8 sec latency 40.180 ms oleg203-client.virtnet: 1 8515 1.21 MB/sec execute 9 sec latency 51.175 ms oleg203-client.virtnet: 1 8632 1.09 MB/sec execute 10 sec latency 35.261 ms oleg203-client.virtnet: 1 8800 1.01 MB/sec execute 11 sec latency 48.094 ms oleg203-client.virtnet: 1 8969 0.93 MB/sec execute 12 sec latency 39.470 ms oleg203-client.virtnet: 1 9177 0.88 MB/sec execute 13 sec latency 37.500 ms oleg203-client.virtnet: 1 9399 0.90 MB/sec execute 14 sec latency 49.188 ms oleg203-client.virtnet: 1 9545 0.86 MB/sec execute 15 sec latency 53.018 ms oleg203-client.virtnet: 1 9868 1.05 MB/sec execute 16 sec latency 29.648 ms oleg203-client.virtnet: 1 10123 1.08 MB/sec execute 17 sec latency 34.764 ms oleg203-client.virtnet: 1 10611 1.29 MB/sec execute 18 sec latency 33.730 ms oleg203-client.virtnet: 1 10783 1.24 MB/sec execute 19 sec lateFailover mds1 to oleg203-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 21:14:11 (1746148451) targets are mounted 21:14:11 (1746148451) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2068 1285620 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37656 3568004 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11988 3594068 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49644 7162072 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg203-server Stopping /mnt/lustre-mds2 (opts:) on oleg203-server 21:14:27 (1746148467) shut down facet: mds2 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds2 to oleg203-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0001 21:14:45 (1746148485) targets are mounted 21:14:45 (1746148485) facet_failover done ncy 31.377 ms oleg203-client.virtnet: 1 10961 1.23 MB/sec execute 20 sec latency 25.613 ms oleg203-client.virtnet: 1 11139 1.19 MB/sec execute 21 sec latency 28.274 ms oleg203-client.virtnet: 1 11394 1.15 MB/sec execute 22 sec latency 35.385 ms oleg203-client.virtnet: 1 11533 1.11 MB/sec execute 23 sec latency 61.977 ms oleg203-client.virtnet: 1 11653 1.08 MB/sec execute 24 sec latency 51.081 ms oleg203-client.virtnet: 1 11856 1.08 MB/sec execute 25 sec latency 34.468 ms oleg203-client.virtnet: 1 12111 1.15 MB/sec execute 26 sec latency 47.041 ms oleg203-client.virtnet: 1 12252 1.11 MB/sec execute 27 sec latency 76.835 ms oleg203-client.virtnet: 1 12432 1.08 MB/sec execute 28 sec latency 36.155 ms oleg203-client.virtnet: 1 12581 1.05 MB/sec execute 29 sec latency 40.438 ms oleg203-client.virtnet: 1 12728 1.02 MB/sec execute 30 sec latency 212.355 ms oleg203-client.virtnet: 1 12823 0.99 MB/sec execute 31 sec latency 274.842 ms oleg203-client.virtnet: 1 13001 0.99 MB/sec execute 32 sec latency 122.506 ms oleg203-client.virtnet: 1 13119 0.99 MB/sec execute 33 sec latency 63.007 ms oleg203-client.virtnet: 1 13430 1.06 MB/sec execute 34 sec latency 37.816 ms oleg203-client.virtnet: 1 13847 1.09 MB/sec execute 35 sec latency 37.512 ms oleg203-client.virtnet: 1 14196 1.18 MB/sec execute 36 sec latency 28.046 ms oleg203-client.virtnet: 1 14329 1.15 MB/sec execute 37 sec latency 57.931 ms oleg203-client.virtnet: 1 14452 1.15 MB/sec execute 38 sec latency 44.888 ms oleg203-client.virtnet: 1 14571 1.13 MB/sec execute 39 sec latency 38.172 ms oleg203-client.virtnet: 1 14691 1.10 MB/sec execute 40 sec latency 49.229 ms oleg203-client.virtnet: 1 14863 1.08 MB/sec execute 41 sec latency 39.851 ms oleg203-client.virtnet: 1 14925 1.06 MB/sec execute 42 sec latency 484.156 ms oleg203-client.virtnet: 1 14925 1.03 MB/sec execute 43 sec latency 1484.334 ms oleg203-client.virtnet: 1 14925 1.01 MB/sec execute 44 sec latency 2484.543 ms oleg203-client.virtnet: 1 14925 0.99 MB/sec execute 45 sec latency 3484.711 ms oleg203-client.virtnet: 1 14925 0.96 MB/sec execute 46 sec latency 4484.875 ms oleg203-client.virtnet: 1 14925 0.94 MB/sec execute 47 sec latency 5485.639 ms oleg203-client.virtnet: 1 14925 0.92 MB/sec execute 48 sec latency 6486.182 ms oleg203-client.virtnet: 1 14925 0.90 MB/sec execute 49 sec latency 7487.270 ms oleg203-client.virtnet: 1 14925 0.89 MB/sec execute 50 sec latency 8491.484 ms oleg203-client.virtnet: 1 14925 0.87 MB/sec execute 51 sec latency 9491.643 ms oleg203-client.virtnet: 1 14925 0.85 MB/sec execute 52 sec latency 10491.844 ms oleg203-client.virtnet: 1 14925 0.84 MB/sec execute 53 sec latency 11492.763 ms oleg203-client.virtnet: 1 14925 0.82 MB/sec execute 54 sec latency 12493.896 ms oleg203-client.virtnet: 1 14925 0.81 MB/sec execute 55 sec latency 13494.082 ms oleg203-client.virtnet: 1 14925 0.79 MB/sec execute 56 sec latency 14494.200 ms oleg203-client.virtnet: 1 14925 0.78 MB/sec execute 57 sec latency 15494.373 ms oleg203-client.virtnet: 1 14925 0.76 MB/sec execute 58 sec latency 16494.559 ms oleg203-client.virtnet: 1 14925 0.75 MB/sec execute 59 sec latency 17494.706 ms oleg203-client.virtnet: 1 14925 0.74 MB/sec execute 60 sec latency 18494.855 ms oleg203-client.virtnet: 1 14925 0.73 MB/sec execute 61 sec latency 19495.175 ms oleg203-client.virtnet: 1 14925 0.72 MB/sec execute 62 sec latency 20495.377 ms oleg203-client.virtnet: 1 14925 0.70 MB/sec execute 63 sec latency 21497.854 ms oleg203-client.virtnoleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37432 3569252 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12060 3594048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7163300 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 21:15:00 (1746148500) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 21:15:17 (1746148517) targets are mounted 21:15:17 (1746148517) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37428 3569096 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12060 3594116 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163212 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg203-server Stopping /mnt/lustre-mds2 (opts:) on oleg203-server et: 1 14947 0.69 MB/sec execute 64 sec latency 22297.836 ms oleg203-client.virtnet: 1 15109 0.69 MB/sec execute 65 sec latency 43.166 ms oleg203-client.virtnet: 1 15286 0.70 MB/sec execute 66 sec latency 35.385 ms oleg203-client.virtnet: 1 15470 0.69 MB/sec execute 67 sec latency 47.242 ms oleg203-client.virtnet: 1 15739 0.72 MB/sec execute 68 sec latency 30.349 ms oleg203-client.virtnet: 1 15888 0.71 MB/sec execute 69 sec latency 41.273 ms oleg203-client.virtnet: 1 16000 0.71 MB/sec execute 70 sec latency 42.940 ms oleg203-client.virtnet: 1 16197 0.70 MB/sec execute 71 sec latency 48.778 ms oleg203-client.virtnet: 1 16358 0.69 MB/sec execute 72 sec latency 44.831 ms oleg203-client.virtnet: 1 16565 0.70 MB/sec execute 73 sec latency 54.354 ms oleg203-client.virtnet: 1 16650 0.69 MB/sec execute 74 sec latency 56.546 ms oleg203-client.virtnet: 1 16805 0.69 MB/sec execute 75 sec latency 91.447 ms oleg203-client.virtnet: 1 17204 0.74 MB/sec execute 76 sec latency 43.399 ms oleg203-client.virtnet: 1 17695 0.80 MB/sec execute 77 sec latency 30.216 ms oleg203-client.virtnet: 1 17871 0.79 MB/sec execute 78 sec latency 47.086 ms oleg203-client.virtnet: 1 18031 0.79 MB/sec execute 79 sec latency 31.895 ms oleg203-client.virtnet: 1 18166 0.79 MB/sec execute 80 sec latency 33.759 ms oleg203-client.virtnet: 1 18266 0.78 MB/sec execute 81 sec latency 294.121 ms oleg203-client.virtnet: 1 18266 0.77 MB/sec execute 82 sec latency 1294.352 ms oleg203-client.virtnet: 1 18266 0.76 MB/sec execute 83 sec latency 2294.525 ms oleg203-client.virtnet: 1 18266 0.75 MB/sec execute 84 sec latency 3294.751 ms oleg203-client.virtnet: 1 18266 0.74 MB/sec execute 85 sec latency 4296.448 ms oleg203-client.virtnet: 1 18266 0.73 MB/sec execute 86 sec latency 5297.020 ms oleg203-client.virtnet: 1 18266 0.73 MB/sec execute 87 sec latency 6297.810 ms oleg203-client.virtnet: 1 18266 0.72 MB/sec execute 88 sec latency 7297.976 ms oleg203-client.virtnet: 1 18266 0.71 MB/sec execute 89 sec latency 8298.189 ms oleg203-client.virtnet: 1 18266 0.70 MB/sec execute 90 sec latency 9298.443 ms oleg203-client.virtnet: 1 18266 0.69 MB/sec execute 91 sec latency 10298.934 ms oleg203-client.virtnet: 1 18266 0.69 MB/sec execute 92 sec latency 11299.136 ms oleg203-client.virtnet: 1 18266 0.68 MB/sec execute 93 sec latency 12299.702 ms oleg203-client.virtnet: 1 18266 0.67 MB/sec execute 94 sec latency 13299.908 ms oleg203-client.virtnet: 1 18266 0.66 MB/sec execute 95 sec latency 14300.098 ms oleg203-client.virtnet: 1 18432 0.66 MB/sec execute 96 sec latency 14487.432 ms oleg203-client.virtnet: 1 18532 0.65 MB/sec execute 97 sec latency 54.943 ms oleg203-client.virtnet: 1 18639 0.65 MB/sec execute 98 sec latency 47.461 ms oleg203-client.virtnet: 1 18743 0.65 MB/sec execute 99 sec latency 48.538 ms oleg203-client.virtnet: 1 18940 0.65 MB/sec execute 100 sec latency 44.570 ms oleg203-client.virtnet: 1 19138 0.67 MB/sec execute 101 sec latency 43.203 ms oleg203-client.virtnet: 1 19301 0.67 MB/sec execute 102 sec latency 35.789 ms oleg203-client.virtnet: 1 19382 0.66 MB/sec execute 103 sec latency 216.695 ms oleg203-client.virtnet: 1 19437 0.66 MB/sec execute 104 sec latency 701.532 ms oleg203-client.virtnet: 1 19576 0.65 MB/sec execute 105 sec latency 42.614 ms oleg203-client.virtnet: 1 19735 0.65 MB/sec execute 106 sec latency 52.408 ms oleg203-client.virtnet: 1 19850 0.64 MB/sec execute 107 sec latency 253.166 ms oleg203-client.virtnet: 1 19850 0.64 MB/sec execute 108 s21:15:32 (1746148532) shut down facet: mds2 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds2 to oleg203-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0001 21:15:49 (1746148549) targets are mounted 21:15:49 (1746148549) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37544 3568860 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12112 3592084 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49656 7160944 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 21:16:04 (1746148564) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 ec latency 1253.480 ms oleg203-client.virtnet: 1 19850 0.63 MB/sec execute 109 sec latency 2254.004 ms oleg203-client.virtnet: 1 19850 0.63 MB/sec execute 110 sec latency 3254.291 ms oleg203-client.virtnet: 1 19850 0.62 MB/sec execute 111 sec latency 4257.820 ms oleg203-client.virtnet: 1 19850 0.61 MB/sec execute 112 sec latency 5258.053 ms oleg203-client.virtnet: 1 19850 0.61 MB/sec execute 113 sec latency 6258.268 ms oleg203-client.virtnet: 1 19850 0.60 MB/sec execute 114 sec latency 7259.242 ms oleg203-client.virtnet: 1 19850 0.60 MB/sec execute 115 sec latency 8259.498 ms oleg203-client.virtnet: 1 19850 0.59 MB/sec execute 116 sec latency 9260.027 ms oleg203-client.virtnet: 1 19850 0.59 MB/sec execute 117 sec latency 10260.203 ms oleg203-client.virtnet: 1 19850 0.58 MB/sec execute 118 sec latency 11260.392 ms oleg203-client.virtnet: 1 19850 0.58 MB/sec execute 119 sec latency 12260.574 ms oleg203-client.virtnet: 1 19850 0.57 MB/sec execute 120 sec latency 13260.703 ms oleg203-client.virtnet: 1 19850 0.57 MB/sec execute 121 sec latency 14261.000 ms oleg203-client.virtnet: 1 19850 0.56 MB/sec execute 122 sec latency 15261.653 ms oleg203-client.virtnet: 1 19850 0.56 MB/sec execute 123 sec latency 16262.978 ms oleg203-client.virtnet: 1 19850 0.56 MB/sec execute 124 sec latency 17264.659 ms oleg203-client.virtnet: 1 19850 0.55 MB/sec execute 125 sec latency 18264.900 ms oleg203-client.virtnet: 1 19850 0.55 MB/sec execute 126 sec latency 19265.335 ms oleg203-client.virtnet: 1 19850 0.54 MB/sec execute 127 sec latency 20265.498 ms oleg203-client.virtnet: 1 19850 0.54 MB/sec execute 128 sec latency 21268.235 ms oleg203-client.virtnet: 1 19985 0.53 MB/sec execute 129 sec latency 21275.986 ms oleg203-client.virtnet: 1 20185 0.54 MB/sec execute 130 sec latency 42.826 ms oleg203-client.virtnet: 1 20380 0.54 MB/sec execute 131 sec latency 34.027 ms oleg203-client.virtnet: 1 20768 0.57 MB/sec execute 132 sec latency 33.427 ms oleg203-client.virtnet: 1 21138 0.58 MB/sec execute 133 sec latency 39.232 ms oleg203-client.virtnet: 1 21356 0.60 MB/sec execute 134 sec latency 46.560 ms oleg203-client.virtnet: 1 21508 0.60 MB/sec execute 135 sec latency 46.235 ms oleg203-client.virtnet: 1 21621 0.60 MB/sec execute 136 sec latency 56.731 ms oleg203-client.virtnet: 1 21756 0.60 MB/sec execute 137 sec latency 47.572 ms oleg203-client.virtnet: 1 21922 0.60 MB/sec execute 138 sec latency 33.268 ms oleg203-client.virtnet: 1 22057 0.59 MB/sec execute 139 sec latency 49.879 ms oleg203-client.virtnet: 1 22198 0.59 MB/sec execute 140 sec latency 42.985 ms oleg203-client.virtnet: 1 22393 0.60 MB/sec execute 141 sec latency 44.727 ms oleg203-client.virtnet: 1 22722 0.61 MB/sec execute 142 sec latency 24.205 ms oleg203-client.virtnet: 1 22943 0.61 MB/sec execute 143 sec latency 22.752 ms oleg203-client.virtnet: 1 23123 0.61 MB/sec execute 144 sec latency 35.278 ms oleg203-client.virtnet: 1 23346 0.61 MB/sec execute 145 sec latency 27.502 ms oleg203-client.virtnet: 1 23585 0.60 MB/sec execute 146 sec latency 24.638 ms oleg203-client.virtnet: 1 23774 0.61 MB/sec execute 147 sec latency 39.305 ms oleg203-client.virtnet: 1 24077 0.63 MB/sec execute 148 sec latency 72.028 ms oleg203-client.virtnet: 1 24336 0.64 MB/sec execute 149 sec latency 45.757 ms oleg203-client.virtnet: 1 24733 0.65 MB/sec execute 150 sec latency 36.603 ms oleg203-client.virtnet: 1 24960 0.66 MB/sec execute 151 sec latency 72.609 ms oleg203-client.virtnet: 1 25122 0.67 MB/sec execute 152 sec latency 42.811 ms oleg203-clienStarting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 21:16:22 (1746148582) targets are mounted 21:16:22 (1746148582) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2068 1285620 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3569104 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12064 3593956 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7163060 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg203-server Stopping /mnt/lustre-mds2 (opts:) on oleg203-server 21:16:36 (1746148596) shut down facet: mds2 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds2 to oleg203-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0001 21:16:54 (1746148614) targets are mounted 21:16:54 (1746148614) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec t.virtnet: 1 25242 0.66 MB/sec execute 153 sec latency 37.550 ms oleg203-client.virtnet: 1 25326 0.66 MB/sec execute 154 sec latency 50.847 ms oleg203-client.virtnet: 1 25468 0.65 MB/sec execute 155 sec latency 55.297 ms oleg203-client.virtnet: 1 25629 0.65 MB/sec execute 156 sec latency 39.547 ms oleg203-client.virtnet: 1 25770 0.65 MB/sec execute 157 sec latency 42.728 ms oleg203-client.virtnet: 1 25921 0.65 MB/sec execute 158 sec latency 34.436 ms oleg203-client.virtnet: 1 26104 0.65 MB/sec execute 159 sec latency 30.017 ms oleg203-client.virtnet: 1 26350 0.67 MB/sec execute 160 sec latency 33.552 ms oleg203-client.virtnet: 1 26500 0.66 MB/sec execute 161 sec latency 54.144 ms oleg203-client.virtnet: 1 26632 0.66 MB/sec execute 162 sec latency 42.533 ms oleg203-client.virtnet: 1 26767 0.66 MB/sec execute 163 sec latency 44.517 ms oleg203-client.virtnet: 1 26976 0.65 MB/sec execute 164 sec latency 41.736 ms oleg203-client.virtnet: 1 27245 0.66 MB/sec execute 165 sec latency 64.308 ms oleg203-client.virtnet: 1 27436 0.66 MB/sec execute 166 sec latency 31.188 ms oleg203-client.virtnet: 1 27769 0.68 MB/sec execute 167 sec latency 37.724 ms oleg203-client.virtnet: 1 27853 0.68 MB/sec execute 168 sec latency 539.165 ms oleg203-client.virtnet: 1 28147 0.69 MB/sec execute 169 sec latency 682.170 ms oleg203-client.virtnet: 1 28449 0.70 MB/sec execute 170 sec latency 36.048 ms oleg203-client.virtnet: 1 28626 0.70 MB/sec execute 171 sec latency 45.041 ms oleg203-client.virtnet: 1 28651 0.70 MB/sec execute 172 sec latency 847.141 ms oleg203-client.virtnet: 1 28651 0.69 MB/sec execute 173 sec latency 1848.013 ms oleg203-client.virtnet: 1 28651 0.69 MB/sec execute 174 sec latency 2848.182 ms oleg203-client.virtnet: 1 28651 0.69 MB/sec execute 175 sec latency 3848.352 ms oleg203-client.virtnet: 1 28651 0.68 MB/sec execute 176 sec latency 4848.524 ms oleg203-client.virtnet: 1 28651 0.68 MB/sec execute 177 sec latency 5849.985 ms oleg203-client.virtnet: 1 28651 0.68 MB/sec execute 178 sec latency 6850.494 ms oleg203-client.virtnet: 1 28651 0.67 MB/sec execute 179 sec latency 7851.446 ms oleg203-client.virtnet: 1 28651 0.67 MB/sec execute 180 sec latency 8851.851 ms oleg203-client.virtnet: 1 28651 0.66 MB/sec execute 181 sec latency 9852.477 ms oleg203-client.virtnet: 1 28651 0.66 MB/sec execute 182 sec latency 10852.653 ms oleg203-client.virtnet: 1 28651 0.66 MB/sec execute 183 sec latency 11853.027 ms oleg203-client.virtnet: 1 28651 0.65 MB/sec execute 184 sec latency 12853.600 ms oleg203-client.virtnet: 1 28651 0.65 MB/sec execute 185 sec latency 13853.939 ms oleg203-client.virtnet: 1 28651 0.65 MB/sec execute 186 sec latency 14854.420 ms oleg203-client.virtnet: 1 28651 0.64 MB/sec execute 187 sec latency 15854.596 ms oleg203-client.virtnet: 1 28651 0.64 MB/sec execute 188 sec latency 16854.969 ms oleg203-client.virtnet: 1 28651 0.64 MB/sec execute 189 sec latency 17855.945 ms oleg203-client.virtnet: 1 28651 0.63 MB/sec execute 190 sec latency 18856.431 ms oleg203-client.virtnet: 1 28651 0.63 MB/sec execute 191 sec latency 19856.841 ms oleg203-client.virtnet: 1 28651 0.63 MB/sec execute 192 sec latency 20859.039 ms oleg203-client.virtnet: 1 28651 0.62 MB/sec execute 193 sec latency 21859.386 ms oleg203-client.virtnet: 1 28738 0.62 MB/sec execute 194 sec latency 21943.425 ms oleg203-client.virtnet: 1 28830 0.62 MB/sec execute 195 sec latency 50.259 ms oleg203-client.virtnet: 1 28918 0.61 MB/sec execute 196 sec latency 61.362 ms oleg203-client.virtnet: 1 29071 0.UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37596 3569268 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11900 3595120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7164388 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 21:17:10 (1746148630) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 21:17:28 (1746148648) targets are mounted 21:17:28 (1746148648) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37588 3569004 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11900 3594096 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163100 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg203-server Stopping /mnt/lustre-mds2 (opts:) on oleg203-server 21:17:44 (1746148664) shut down facet: mds2 facet_host: oleg203-server facet_failover_host: oleg203-server 61 MB/sec execute 197 sec latency 52.571 ms oleg203-client.virtnet: 1 29195 0.61 MB/sec execute 198 sec latency 55.080 ms oleg203-client.virtnet: 1 29314 0.61 MB/sec execute 199 sec latency 51.954 ms oleg203-client.virtnet: 1 29421 0.61 MB/sec execute 200 sec latency 41.458 ms oleg203-client.virtnet: 1 29579 0.61 MB/sec execute 201 sec latency 44.987 ms oleg203-client.virtnet: 1 29709 0.61 MB/sec execute 202 sec latency 49.244 ms oleg203-client.virtnet: 1 29917 0.62 MB/sec execute 203 sec latency 51.452 ms oleg203-client.virtnet: 1 30031 0.62 MB/sec execute 204 sec latency 57.800 ms oleg203-client.virtnet: 1 30140 0.61 MB/sec execute 205 sec latency 49.176 ms oleg203-client.virtnet: 1 30268 0.61 MB/sec execute 206 sec latency 65.181 ms oleg203-client.virtnet: 1 30455 0.61 MB/sec execute 207 sec latency 37.966 ms oleg203-client.virtnet: 1 30679 0.61 MB/sec execute 208 sec latency 30.243 ms oleg203-client.virtnet: 1 30853 0.61 MB/sec execute 209 sec latency 51.628 ms oleg203-client.virtnet: 1 31170 0.63 MB/sec execute 210 sec latency 34.695 ms oleg203-client.virtnet: 1 31584 0.63 MB/sec execute 211 sec latency 41.388 ms oleg203-client.virtnet: 1 31929 0.65 MB/sec execute 212 sec latency 30.506 ms oleg203-client.virtnet: 1 32109 0.65 MB/sec execute 213 sec latency 47.697 ms oleg203-client.virtnet: 1 32269 0.65 MB/sec execute 214 sec latency 33.457 ms oleg203-client.virtnet: 1 32427 0.65 MB/sec execute 215 sec latency 48.158 ms oleg203-client.virtnet: 1 32489 0.65 MB/sec execute 216 sec latency 625.053 ms oleg203-client.virtnet: 1 32489 0.64 MB/sec execute 217 sec latency 1625.206 ms oleg203-client.virtnet: 1 32489 0.64 MB/sec execute 218 sec latency 2626.957 ms oleg203-client.virtnet: 1 32489 0.64 MB/sec execute 219 sec latency 3627.423 ms oleg203-client.virtnet: 1 32489 0.63 MB/sec execute 220 sec latency 4629.120 ms oleg203-client.virtnet: 1 32489 0.63 MB/sec execute 221 sec latency 5629.426 ms oleg203-client.virtnet: 1 32489 0.63 MB/sec execute 222 sec latency 6629.542 ms oleg203-client.virtnet: 1 32489 0.63 MB/sec execute 223 sec latency 7629.809 ms oleg203-client.virtnet: 1 32489 0.62 MB/sec execute 224 sec latency 8630.011 ms oleg203-client.virtnet: 1 32489 0.62 MB/sec execute 225 sec latency 9630.465 ms oleg203-client.virtnet: 1 32489 0.62 MB/sec execute 226 sec latency 10630.844 ms oleg203-client.virtnet: 1 32613 0.62 MB/sec execute 227 sec latency 10970.668 ms oleg203-client.virtnet: 1 32704 0.61 MB/sec execute 228 sec latency 65.560 ms oleg203-client.virtnet: 1 32839 0.61 MB/sec execute 229 sec latency 58.978 ms oleg203-client.virtnet: 1 32964 0.61 MB/sec execute 230 sec latency 40.118 ms oleg203-client.virtnet: 1 33148 0.61 MB/sec execute 231 sec latency 48.771 ms oleg203-client.virtnet: 1 33360 0.62 MB/sec execute 232 sec latency 41.097 ms oleg203-client.virtnet: 1 33494 0.62 MB/sec execute 233 sec latency 63.857 ms oleg203-client.virtnet: 1 33602 0.62 MB/sec execute 234 sec latency 51.062 ms oleg203-client.virtnet: 1 33697 0.62 MB/sec execute 235 sec latency 58.095 ms oleg203-client.virtnet: 1 33819 0.61 MB/sec execute 236 sec latency 43.441 ms oleg203-client.virtnet: 1 33996 0.61 MB/sec execute 237 sec latency 40.287 ms oleg203-client.virtnet: 1 34143 0.61 MB/sec execute 238 sec latency 41.711 ms oleg203-client.virtnet: 1 34225 0.61 MB/sec execute 239 sec latency 570.716 ms oleg203-client.virtnet: 1 34225 0.61 MB/sec execute 240 sec latency 1571.334 ms oleg203-client.virtnet: 1 34225 0.60 MB/sec execute 241 sec latency 2571.999 ms oleg203-client.vFailover mds2 to oleg203-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0001 21:18:01 (1746148681) targets are mounted 21:18:01 (1746148681) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec irtnet: 1 34225 0.60 MB/sec execute 242 sec latency 3572.624 ms oleg203-client.virtnet: 1 34225 0.60 MB/sec execute 243 sec latency 4573.333 ms oleg203-client.virtnet: 1 34225 0.60 MB/sec execute 244 sec latency 5573.597 ms oleg203-client.virtnet: 1 34225 0.59 MB/sec execute 245 sec latency 6574.342 ms oleg203-client.virtnet: 1 34225 0.59 MB/sec execute 246 sec latency 7574.509 ms oleg203-client.virtnet: 1 34225 0.59 MB/sec execute 247 sec latency 8574.814 ms oleg203-client.virtnet: 1 34225 0.59 MB/sec execute 248 sec latency 9580.600 ms oleg203-client.virtnet: 1 34225 0.58 MB/sec execute 249 sec latency 10580.934 ms oleg203-client.virtnet: 1 34225 0.58 MB/sec execute 250 sec latency 11581.155 ms oleg203-client.virtnet: 1 34225 0.58 MB/sec execute 251 sec latency 12581.799 ms oleg203-client.virtnet: 1 34225 0.58 MB/sec execute 252 sec latency 13581.960 ms oleg203-client.virtnet: 1 34225 0.57 MB/sec execute 253 sec latency 14582.359 ms oleg203-client.virtnet: 1 34225 0.57 MB/sec execute 254 sec latency 15582.531 ms oleg203-client.virtnet: 1 34225 0.57 MB/sec execute 255 sec latency 16582.728 ms oleg203-client.virtnet: 1 34225 0.57 MB/sec execute 256 sec latency 17583.925 ms oleg203-client.virtnet: 1 34225 0.57 MB/sec execute 257 sec latency 18584.867 ms oleg203-client.virtnet: 1 34225 0.56 MB/sec execute 258 sec latency 19585.081 ms oleg203-client.virtnet: 1 34225 0.56 MB/sec execute 259 sec latency 20586.294 ms oleg203-client.virtnet: 1 34311 0.56 MB/sec execute 260 sec latency 21106.259 ms oleg203-client.virtnet: 1 34432 0.56 MB/sec execute 261 sec latency 71.768 ms oleg203-client.virtnet: 1 34573 0.56 MB/sec execute 262 sec latency 82.409 ms oleg203-client.virtnet: 1 34962 0.58 MB/sec execute 263 sec latency 64.689 ms oleg203-client.virtnet: 1 35351 0.58 MB/sec execute 264 sec latency 52.223 ms oleg203-client.virtnet: 1 35645 0.59 MB/sec execute 265 sec latency 33.198 ms oleg203-client.virtnet: 1 35818 0.60 MB/sec execute 266 sec latency 36.368 ms oleg203-client.virtnet: 1 35968 0.59 MB/sec execute 267 sec latency 34.979 ms oleg203-client.virtnet: 1 36175 0.59 MB/sec execute 268 sec latency 28.086 ms oleg203-client.virtnet: 1 36308 0.59 MB/sec execute 269 sec latency 48.282 ms oleg203-client.virtnet: 1 36473 0.59 MB/sec execute 270 sec latency 46.242 ms oleg203-client.virtnet: 1 36692 0.59 MB/sec execute 271 sec latency 26.399 ms oleg203-client.virtnet: 1 36960 0.60 MB/sec execute 272 sec latency 28.921 ms oleg203-client.virtnet: 1 37135 0.60 MB/sec execute 273 sec latency 34.230 ms oleg203-client.virtnet: 1 37307 0.60 MB/sec execute 274 sec latency 33.478 ms oleg203-client.virtnet: 1 37508 0.60 MB/sec execute 275 sec latency 37.834 ms oleg203-client.virtnet: 1 37681 0.60 MB/sec execute 276 sec latency 29.662 ms oleg203-client.virtnet: 1 37909 0.60 MB/sec execute 277 sec latency 47.515 ms oleg203-client.virtnet: 1 38099 0.60 MB/sec execute 278 sec latency 34.496 ms oleg203-client.virtnet: 1 38497 0.61 MB/sec execute 279 sec latency 31.851 ms oleg203-client.virtnet: 1 38994 0.63 MB/sec execute 280 sec latency 31.969 ms oleg203-client.virtnet: 1 39186 0.63 MB/sec execute 281 sec latency 35.533 ms oleg203-client.virtnet: 1 39350 0.63 MB/sec execute 282 sec latency 36.677 ms oleg203-client.virtnet: 1 39496 0.63 MB/sec execute 283 sec latency 25.851 ms oleg203-client.virtnet: 1 39704 0.63 MB/sec execute 284 sec latency 30.474 ms oleg203-client.virtnet: 1 39846 0.63 MB/sec execute 285 sec latency 42.483 ms oleg203-client.virtnet: 1 40020 0.62 MB/sec execute 286 sec latency 29.441 ms oleg203-client.virtnet: 1 40210 0.63 MB/sec execute 287 sec latency 35.142 ms oleg203-client.virtnet: 1 40450 0.63 MB/sec execute 288 sec latency 39.436 ms oleg203-client.virtnet: 1 40622 0.63 MB/sec execute 289 sec latency 34.629 ms oleg203-client.virtnet: 1 40777 0.63 MB/sec execute 290 sec latency 36.771 ms oleg203-client.virtnet: 1 40949 0.63 MB/sec execute 291 sec latency 32.861 ms oleg203-client.virtnet: 1 41166 0.63 MB/sec execute 292 sec latency 30.283 ms oleg203-client.virtnet: 1 41453 0.63 MB/sec execute 293 sec latency 22.890 ms oleg203-client.virtnet: 1 41825 0.64 MB/sec execute 294 sec latency 28.703 ms oleg203-client.virtnet: 1 42266 0.65 MB/sec execute 295 sec latency 25.164 ms oleg203-client.virtnet: 1 42672 0.66 MB/sec execute 296 sec latency 34.954 ms oleg203-client.virtnet: 1 42886 0.66 MB/sec execute 297 sec latency 26.561 ms oleg203-client.virtnet: 1 43065 0.66 MB/sec execute 298 sec latency 26.673 ms oleg203-client.virtnet: 1 43301 0.66 MB/sec execute 299 sec latency 44.702 ms oleg203-client.virtnet: 1 cleanup 300 sec oleg203-client.virtnet: 0 cleanup 301 sec oleg203-client.virtnet: oleg203-client.virtnet: Operation Count AvgLat MaxLat oleg203-client.virtnet: ---------------------------------------- oleg203-client.virtnet: NTCreateX 6391 26.841 22297.826 oleg203-client.virtnet: Close 4659 3.391 22.791 oleg203-client.virtnet: Rename 267 23.510 68.127 oleg203-client.virtnet: Unlink 1320 5.437 39.829 oleg203-client.virtnet: Qpathinfo 5742 12.143 21275.967 oleg203-client.virtnet: Qfileinfo 985 0.245 7.876 oleg203-client.virtnet: Qfsinfo 1078 0.608 22.653 oleg203-client.virtnet: Sfileinfo 511 15.819 65.548 oleg203-client.virtnet: Find 2246 0.880 35.011 oleg203-client.virtnet: WriteX 3123 2.415 41.751 oleg203-client.virtnet: ReadX 9888 0.040 13.403 oleg203-client.virtnet: LockX 20 3.608 9.175 oleg203-client.virtnet: UnlockX 20 2.556 6.700 oleg203-client.virtnet: Flush 452 21.728 701.518 oleg203-client.virtnet: oleg203-client.virtnet: Throughput 0.659379 MB/sec 1 clients 1 procs max_latency=22297.836 ms oleg203-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg203-client.virtnet at Thu May 1 21:18:45 EDT 2025 with return code 0 oleg203-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg203-client.virtnet oleg203-client.virtnet: /mnt/lustre/d70b.replay-single/oleg203-client.virtnet /mnt/lustre/d70b.replay-single/oleg203-client.virtnet oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg203-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg203-client.virtnet: removed directory: 'clients/client0' oleg203-client.virtnet: removed directory: 'clients' oleg203-client.virtnet: removed 'client.txt' oleg203-client.virtnet: /mnt/lustre/d70b.replay-single/oleg203-client.virtnet oleg203-client.virtnet: dbench successfully finished