== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 23:01:23 (1746068483) Starting client oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre Started clients oleg356-client.virtnet: 192.168.203.156@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg356-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14048 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3580864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7186316 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg356-server Stopping /mnt/lustre-mds1 (opts:) on oleg356-server 23:01:34 (1746068494) shut down facet: mds1 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds1 to oleg356-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 23:01:51 (1746068511) targets are mounted 23:01:51 (1746068511) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 27440 3567840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1728 3594256 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 29168 7162096 1% /mnt/lustre oleg356-client.virtnet: looking for dbench program oleg356-client.virtnet: /usr/bin/dbench oleg356-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg356-client.virtnet oleg356-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg356-client.virtnet' oleg356-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg356-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg356-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg356-client.virtnet at Wed Apr 30 23:01:24 EDT 2025 oleg356-client.virtnet: waiting for dbench pid 14073 oleg356-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg356-client.virtnet: oleg356-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg356-client.virtnet: failed to create barrier semaphore oleg356-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg356-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg356-client.virtnet: releasing clients oleg356-client.virtnet: 1 180 7.32 MB/sec warmup 1 sec latency 46.163 ms oleg356-client.virtnet: 1 325 5.87 MB/sec warmup 2 sec latency 51.942 ms oleg356-client.virtnet: 1 503 5.82 MB/sec warmup 3 sec latency 71.071 ms oleg356-client.virtnet: 1 708 5.30 MB/sec warmup 4 sec latency 72.579 ms oleg356-client.virtnet: 1 863 4.31 MB/sec warmup 5 sec latency 57.795 ms oleg356-client.virtnet: 1 984 3.65 MB/sec warmup 6 sec latency 40.788 ms oleg356-client.virtnet: 1 1145 3.28 MB/sec warmup 7 sec latency 44.410 ms oleg356-client.virtnet: 1 1263 2.89 MB/sec warmup 8 sec latency 61.079 ms oleg356-client.virtnet: 1 1489 2.89 MB/sec warmup 9 sec latency 38.738 ms oleg356-client.virtnet: 1 1662 2.62 MB/sec warmup 10 sec latency 39.305 ms oleg356-client.virtnet: 1 1873 2.39 MB/sec warmup 11 sec latency 34.123 ms oleg356-client.virtnet: 1 2145 2.22 MB/sec warmup 12 sec latency 23.109 ms oleg356-client.virtnet: 1 2384 2.15 MB/sec warmup 13 sec latency 25.944 ms oleg356-client.virtnet: 1 2586 2.07 MB/sec warmup 14 sec latency 36.767 ms oleg356-client.virtnet: 1 2978 2.23 MB/sec warmup 15 sec latency 59.441 ms oleg356-client.virtnet: 1 3352 2.21 MB/sec warmup 16 sec latency 39.184 ms oleg356-client.virtnet: 1 3689 2.28 MB/sec warmup 17 sec latency 25.035 ms oleg356-client.virtnet: 1 3856 2.21 MB/sec warmup 18 sec latency 31.261 ms oleg356-client.virtnet: 1 3999 2.11 MB/sec warmup 19 sec latency 37.867 ms oleg356-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 244.512 ms oleg356-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 1245.005 ms oleg356-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 2245.287 ms oleg356-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 3246.057 ms oleg356-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 4247.027 ms oleg356-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 5247.200 ms oleg356-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 6247.436 ms oleg356-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 7247.630 ms oleg356-client.virtnet: 1 4083 1.43 MB/sec warmup 28 sec latency 8248.525 ms oleg356-client.virtnet: 1 4094 1.38 MB/sec warmup 29 sec latency 9145.500 ms oleg356-client.virtnet: 1 4304 1.35 MB/sec warmup 30 sec latency 26.615 ms oleg356-client.virtnet: 1 4455 1.32 MB/sec warmup 31 sec latency 47.744 ms oleg356-client.virtnet: 1 4670 1.31 MB/sec warmup 32 sec latency 28.678 ms oleg356-client.virtnet: 1 4969 1.37 MB/sec warmup 33 sec latency 26.666 ms oleg356-client.virtnetest_70b fail mds2 2 times Failing mds2 on oleg356-server Stopping /mnt/lustre-mds2 (opts:) on oleg356-server 23:02:05 (1746068525) shut down facet: mds2 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds2 to oleg356-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 23:02:23 (1746068543) targets are mounted 23:02:23 (1746068543) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37548 3569340 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12120 3591272 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49668 7160612 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg356-server Stopping /mnt/lustre-mds1 (opts:) on oleg356-server 23:02:38 (1746068558) shut down facet: mds1 facet_host: oleg356-server facet_failover_host: oleg356-server t: 1 5129 1.33 MB/sec warmup 34 sec latency 31.528 ms oleg356-client.virtnet: 1 5235 1.29 MB/sec warmup 35 sec latency 54.133 ms oleg356-client.virtnet: 1 5356 1.26 MB/sec warmup 36 sec latency 45.045 ms oleg356-client.virtnet: 1 5530 1.23 MB/sec warmup 37 sec latency 57.782 ms oleg356-client.virtnet: 1 5724 1.21 MB/sec warmup 38 sec latency 31.953 ms oleg356-client.virtnet: 1 5933 1.21 MB/sec warmup 39 sec latency 35.959 ms oleg356-client.virtnet: 1 5994 1.18 MB/sec warmup 40 sec latency 446.663 ms oleg356-client.virtnet: 1 5994 1.15 MB/sec warmup 41 sec latency 1447.598 ms oleg356-client.virtnet: 1 5994 1.12 MB/sec warmup 42 sec latency 2449.140 ms oleg356-client.virtnet: 1 5994 1.10 MB/sec warmup 43 sec latency 3450.024 ms oleg356-client.virtnet: 1 5994 1.07 MB/sec warmup 44 sec latency 4450.237 ms oleg356-client.virtnet: 1 5994 1.05 MB/sec warmup 45 sec latency 5452.291 ms oleg356-client.virtnet: 1 5994 1.03 MB/sec warmup 46 sec latency 6452.469 ms oleg356-client.virtnet: 1 5994 1.00 MB/sec warmup 47 sec latency 7452.717 ms oleg356-client.virtnet: 1 5994 0.98 MB/sec warmup 48 sec latency 8453.159 ms oleg356-client.virtnet: 1 5994 0.96 MB/sec warmup 49 sec latency 9453.329 ms oleg356-client.virtnet: 1 5994 0.94 MB/sec warmup 50 sec latency 10454.012 ms oleg356-client.virtnet: 1 5994 0.93 MB/sec warmup 51 sec latency 11454.180 ms oleg356-client.virtnet: 1 5994 0.91 MB/sec warmup 52 sec latency 12454.409 ms oleg356-client.virtnet: 1 5994 0.89 MB/sec warmup 53 sec latency 13454.656 ms oleg356-client.virtnet: 1 5994 0.87 MB/sec warmup 54 sec latency 14455.385 ms oleg356-client.virtnet: 1 5994 0.86 MB/sec warmup 55 sec latency 15455.615 ms oleg356-client.virtnet: 1 5994 0.84 MB/sec warmup 56 sec latency 16456.879 ms oleg356-client.virtnet: 1 5994 0.83 MB/sec warmup 57 sec latency 17457.242 ms oleg356-client.virtnet: 1 5994 0.81 MB/sec warmup 58 sec latency 18458.827 ms oleg356-client.virtnet: 1 5994 0.80 MB/sec warmup 59 sec latency 19459.071 ms oleg356-client.virtnet: 1 5994 0.00 MB/sec execute 1 sec latency 21459.534 ms oleg356-client.virtnet: 1 6141 0.43 MB/sec execute 2 sec latency 21528.050 ms oleg356-client.virtnet: 1 6451 1.57 MB/sec execute 3 sec latency 50.983 ms oleg356-client.virtnet: 1 6579 1.36 MB/sec execute 4 sec latency 56.594 ms oleg356-client.virtnet: 1 7077 2.05 MB/sec execute 5 sec latency 49.646 ms oleg356-client.virtnet: 1 7252 1.78 MB/sec execute 6 sec latency 28.642 ms oleg356-client.virtnet: 1 7381 1.66 MB/sec execute 7 sec latency 55.320 ms oleg356-client.virtnet: 1 7507 1.49 MB/sec execute 8 sec latency 38.149 ms oleg356-client.virtnet: 1 7607 1.33 MB/sec execute 9 sec latency 55.128 ms oleg356-client.virtnet: 1 7769 1.21 MB/sec execute 10 sec latency 50.220 ms oleg356-client.virtnet: 1 7892 1.12 MB/sec execute 11 sec latency 47.739 ms oleg356-client.virtnet: 1 8030 1.06 MB/sec execute 12 sec latency 51.239 ms oleg356-client.virtnet: 1 8187 1.07 MB/sec execute 13 sec latency 44.893 ms oleg356-client.virtnet: 1 8309 1.00 MB/sec execute 14 sec latency 49.312 ms oleg356-client.virtnet: 1 8568 1.14 MB/sec execute 15 sec latency 25.492 ms oleg356-client.virtnet: 1 8724 1.07 MB/sec execute 16 sec latency 27.175 ms oleg356-client.virtnet: 1 8889 1.02 MB/sec execute 17 sec latency 36.149 ms oleg356-client.virtnet: 1 9083 0.97 MB/sec execute 18 sec latency 36.259 ms oleg356-client.virtnet: 1 9254 0.93 MB/sec execute 19 sec latency 36.Failover mds1 to oleg356-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 23:02:57 (1746068577) targets are mounted 23:02:57 (1746068577) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2068 1285620 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37428 3569448 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12068 3594672 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7164120 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg356-server Stopping /mnt/lustre-mds2 (opts:) on oleg356-server 23:03:11 (1746068591) shut down facet: mds2 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds2 to oleg356-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 23:03:28 (1746068608) targets are mounted 23:03:28 (1746068608) facet_failover done 965 ms oleg356-client.virtnet: 1 9451 0.94 MB/sec execute 20 sec latency 35.109 ms oleg356-client.virtnet: 1 9601 0.94 MB/sec execute 21 sec latency 31.490 ms oleg356-client.virtnet: 1 10021 1.11 MB/sec execute 22 sec latency 25.993 ms oleg356-client.virtnet: 1 10411 1.15 MB/sec execute 23 sec latency 35.485 ms oleg356-client.virtnet: 1 10682 1.23 MB/sec execute 24 sec latency 35.694 ms oleg356-client.virtnet: 1 10797 1.19 MB/sec execute 25 sec latency 46.731 ms oleg356-client.virtnet: 1 10926 1.18 MB/sec execute 26 sec latency 58.668 ms oleg356-client.virtnet: 1 11022 1.15 MB/sec execute 27 sec latency 60.843 ms oleg356-client.virtnet: 1 11100 1.11 MB/sec execute 28 sec latency 78.360 ms oleg356-client.virtnet: 1 11169 1.07 MB/sec execute 29 sec latency 60.038 ms oleg356-client.virtnet: 1 11346 1.04 MB/sec execute 30 sec latency 41.475 ms oleg356-client.virtnet: 1 11436 1.02 MB/sec execute 31 sec latency 65.482 ms oleg356-client.virtnet: 1 11547 0.99 MB/sec execute 32 sec latency 99.996 ms oleg356-client.virtnet: 1 11643 0.97 MB/sec execute 33 sec latency 61.868 ms oleg356-client.virtnet: 1 11846 0.97 MB/sec execute 34 sec latency 30.993 ms oleg356-client.virtnet: 1 12074 1.03 MB/sec execute 35 sec latency 65.125 ms oleg356-client.virtnet: 1 12217 1.01 MB/sec execute 36 sec latency 50.885 ms oleg356-client.virtnet: 1 12326 0.98 MB/sec execute 37 sec latency 53.057 ms oleg356-client.virtnet: 1 12470 0.96 MB/sec execute 38 sec latency 39.920 ms oleg356-client.virtnet: 1 12654 0.94 MB/sec execute 39 sec latency 33.580 ms oleg356-client.virtnet: 1 12823 0.92 MB/sec execute 40 sec latency 62.159 ms oleg356-client.virtnet: 1 13002 0.93 MB/sec execute 41 sec latency 53.732 ms oleg356-client.virtnet: 1 13122 0.92 MB/sec execute 42 sec latency 38.548 ms oleg356-client.virtnet: 1 13421 0.98 MB/sec execute 43 sec latency 118.543 ms oleg356-client.virtnet: 1 13826 1.01 MB/sec execute 44 sec latency 49.545 ms oleg356-client.virtnet: 1 14164 1.08 MB/sec execute 45 sec latency 39.448 ms oleg356-client.virtnet: 1 14325 1.06 MB/sec execute 46 sec latency 160.654 ms oleg356-client.virtnet: 1 14325 1.04 MB/sec execute 47 sec latency 1160.919 ms oleg356-client.virtnet: 1 14325 1.02 MB/sec execute 48 sec latency 2161.326 ms oleg356-client.virtnet: 1 14325 1.00 MB/sec execute 49 sec latency 3161.594 ms oleg356-client.virtnet: 1 14325 0.98 MB/sec execute 50 sec latency 4161.827 ms oleg356-client.virtnet: 1 14325 0.96 MB/sec execute 51 sec latency 5162.098 ms oleg356-client.virtnet: 1 14325 0.94 MB/sec execute 52 sec latency 6162.354 ms oleg356-client.virtnet: 1 14325 0.92 MB/sec execute 53 sec latency 7162.721 ms oleg356-client.virtnet: 1 14325 0.90 MB/sec execute 54 sec latency 8163.207 ms oleg356-client.virtnet: 1 14325 0.89 MB/sec execute 55 sec latency 9163.476 ms oleg356-client.virtnet: 1 14325 0.87 MB/sec execute 56 sec latency 10164.359 ms oleg356-client.virtnet: 1 14325 0.86 MB/sec execute 57 sec latency 11164.793 ms oleg356-client.virtnet: 1 14325 0.84 MB/sec execute 58 sec latency 12164.938 ms oleg356-client.virtnet: 1 14325 0.83 MB/sec execute 59 sec latency 13165.111 ms oleg356-client.virtnet: 1 14325 0.81 MB/sec execute 60 sec latency 14165.723 ms oleg356-client.virtnet: 1 14325 0.80 MB/sec execute 61 sec latency 15165.894 ms oleg356-client.virtnet: 1 14325 0.79 MB/sec execute 62 sec latency 16167.439 ms oleg356-client.virtnet: 1 14325 0.78 MB/sec execute 63 sec latency 17167.742 ms oleg356-client.virtnet: 1 14325 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37508 3569084 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13832 3589372 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 51340 7158456 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg356-server Stopping /mnt/lustre-mds1 (opts:) on oleg356-server 23:03:42 (1746068622) shut down facet: mds1 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds1 to oleg356-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 23:04:01 (1746068641) targets are mounted 23:04:01 (1746068641) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569540 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12140 3594824 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7164364 1% /mnt/lustre test_70b fail mds2 6 times 0.76 MB/sec execute 64 sec latency 18167.952 ms oleg356-client.virtnet: 1 14325 0.75 MB/sec execute 65 sec latency 19168.191 ms oleg356-client.virtnet: 1 14325 0.74 MB/sec execute 66 sec latency 20169.549 ms oleg356-client.virtnet: 1 14348 0.73 MB/sec execute 67 sec latency 20921.985 ms oleg356-client.virtnet: 1 14495 0.73 MB/sec execute 68 sec latency 32.565 ms oleg356-client.virtnet: 1 14645 0.73 MB/sec execute 69 sec latency 38.626 ms oleg356-client.virtnet: 1 14850 0.72 MB/sec execute 70 sec latency 23.534 ms oleg356-client.virtnet: 1 15010 0.71 MB/sec execute 71 sec latency 44.643 ms oleg356-client.virtnet: 1 15177 0.71 MB/sec execute 72 sec latency 31.474 ms oleg356-client.virtnet: 1 15425 0.72 MB/sec execute 73 sec latency 40.808 ms oleg356-client.virtnet: 1 15689 0.75 MB/sec execute 74 sec latency 25.563 ms oleg356-client.virtnet: 1 15861 0.74 MB/sec execute 75 sec latency 29.678 ms oleg356-client.virtnet: 1 16009 0.73 MB/sec execute 76 sec latency 35.328 ms oleg356-client.virtnet: 1 16187 0.72 MB/sec execute 77 sec latency 46.356 ms oleg356-client.virtnet: 1 16360 0.72 MB/sec execute 78 sec latency 37.885 ms oleg356-client.virtnet: 1 16592 0.72 MB/sec execute 79 sec latency 30.281 ms oleg356-client.virtnet: 1 16809 0.73 MB/sec execute 80 sec latency 32.747 ms oleg356-client.virtnet: 1 17216 0.77 MB/sec execute 81 sec latency 26.528 ms oleg356-client.virtnet: 1 17715 0.82 MB/sec execute 82 sec latency 27.621 ms oleg356-client.virtnet: 1 17912 0.82 MB/sec execute 83 sec latency 31.830 ms oleg356-client.virtnet: 1 18086 0.82 MB/sec execute 84 sec latency 41.207 ms oleg356-client.virtnet: 1 18227 0.82 MB/sec execute 85 sec latency 34.753 ms oleg356-client.virtnet: 1 18266 0.81 MB/sec execute 86 sec latency 758.125 ms oleg356-client.virtnet: 1 18266 0.80 MB/sec execute 87 sec latency 1758.296 ms oleg356-client.virtnet: 1 18266 0.79 MB/sec execute 88 sec latency 2758.414 ms oleg356-client.virtnet: 1 18266 0.78 MB/sec execute 89 sec latency 3758.536 ms oleg356-client.virtnet: 1 18266 0.77 MB/sec execute 90 sec latency 4758.997 ms oleg356-client.virtnet: 1 18266 0.76 MB/sec execute 91 sec latency 5764.760 ms oleg356-client.virtnet: 1 18266 0.75 MB/sec execute 92 sec latency 6764.949 ms oleg356-client.virtnet: 1 18266 0.75 MB/sec execute 93 sec latency 7765.554 ms oleg356-client.virtnet: 1 18266 0.74 MB/sec execute 94 sec latency 8765.706 ms oleg356-client.virtnet: 1 18266 0.73 MB/sec execute 95 sec latency 9765.931 ms oleg356-client.virtnet: 1 18266 0.72 MB/sec execute 96 sec latency 10766.126 ms oleg356-client.virtnet: 1 18266 0.71 MB/sec execute 97 sec latency 11766.603 ms oleg356-client.virtnet: 1 18266 0.71 MB/sec execute 98 sec latency 12769.962 ms oleg356-client.virtnet: 1 18395 0.70 MB/sec execute 99 sec latency 13176.445 ms oleg356-client.virtnet: 1 18495 0.70 MB/sec execute 100 sec latency 67.491 ms oleg356-client.virtnet: 1 18628 0.69 MB/sec execute 101 sec latency 53.443 ms oleg356-client.virtnet: 1 18826 0.70 MB/sec execute 102 sec latency 26.089 ms oleg356-client.virtnet: 1 19013 0.69 MB/sec execute 103 sec latency 38.294 ms oleg356-client.virtnet: 1 19243 0.72 MB/sec execute 104 sec latency 43.851 ms oleg356-client.virtnet: 1 19371 0.71 MB/sec execute 105 sec latency 37.096 ms oleg356-client.virtnet: 1 19527 0.70 MB/sec execute 106 sec latency 39.887 ms oleg356-client.virtnet: 1 19712 0.70 MB/sec execute 107 sec latency 36.425 ms oleg356-client.virtnet: 1 19893 0.69 MB/sec execute 108 sec latency 40.236 msFailing mds2 on oleg356-server Stopping /mnt/lustre-mds2 (opts:) on oleg356-server 23:04:15 (1746068655) shut down facet: mds2 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds2 to oleg356-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 23:04:33 (1746068673) targets are mounted 23:04:33 (1746068673) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37512 3569256 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12188 3592536 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49700 7161792 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg356-server Stopping /mnt/lustre-mds1 (opts:) on oleg356-server 23:04:47 (1746068687) shut down facet: mds1 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds1 to oleg356-server mount facets: mds1 oleg356-client.virtnet: 1 19989 0.69 MB/sec execute 109 sec latency 59.562 ms oleg356-client.virtnet: 1 20127 0.69 MB/sec execute 110 sec latency 430.693 ms oleg356-client.virtnet: 1 20127 0.69 MB/sec execute 111 sec latency 1431.447 ms oleg356-client.virtnet: 1 20127 0.68 MB/sec execute 112 sec latency 2432.084 ms oleg356-client.virtnet: 1 20127 0.67 MB/sec execute 113 sec latency 3432.618 ms oleg356-client.virtnet: 1 20127 0.67 MB/sec execute 114 sec latency 4432.955 ms oleg356-client.virtnet: 1 20127 0.66 MB/sec execute 115 sec latency 5439.503 ms oleg356-client.virtnet: 1 20127 0.66 MB/sec execute 116 sec latency 6440.082 ms oleg356-client.virtnet: 1 20127 0.65 MB/sec execute 117 sec latency 7440.317 ms oleg356-client.virtnet: 1 20127 0.65 MB/sec execute 118 sec latency 8440.591 ms oleg356-client.virtnet: 1 20127 0.64 MB/sec execute 119 sec latency 9440.864 ms oleg356-client.virtnet: 1 20127 0.64 MB/sec execute 120 sec latency 10441.098 ms oleg356-client.virtnet: 1 20127 0.63 MB/sec execute 121 sec latency 11441.591 ms oleg356-client.virtnet: 1 20127 0.62 MB/sec execute 122 sec latency 12441.763 ms oleg356-client.virtnet: 1 20127 0.62 MB/sec execute 123 sec latency 13444.324 ms oleg356-client.virtnet: 1 20127 0.61 MB/sec execute 124 sec latency 14445.128 ms oleg356-client.virtnet: 1 20127 0.61 MB/sec execute 125 sec latency 15445.404 ms oleg356-client.virtnet: 1 20127 0.60 MB/sec execute 126 sec latency 16446.021 ms oleg356-client.virtnet: 1 20127 0.60 MB/sec execute 127 sec latency 17446.173 ms oleg356-client.virtnet: 1 20127 0.60 MB/sec execute 128 sec latency 18446.416 ms oleg356-client.virtnet: 1 20127 0.59 MB/sec execute 129 sec latency 19447.619 ms oleg356-client.virtnet: 1 20127 0.59 MB/sec execute 130 sec latency 20447.956 ms oleg356-client.virtnet: 1 20219 0.58 MB/sec execute 131 sec latency 20920.204 ms oleg356-client.virtnet: 1 20499 0.61 MB/sec execute 132 sec latency 43.385 ms oleg356-client.virtnet: 1 20753 0.62 MB/sec execute 133 sec latency 54.419 ms oleg356-client.virtnet: 1 21122 0.63 MB/sec execute 134 sec latency 42.977 ms oleg356-client.virtnet: 1 21404 0.64 MB/sec execute 135 sec latency 24.983 ms oleg356-client.virtnet: 1 21555 0.65 MB/sec execute 136 sec latency 41.426 ms oleg356-client.virtnet: 1 21697 0.65 MB/sec execute 137 sec latency 33.329 ms oleg356-client.virtnet: 1 21803 0.64 MB/sec execute 138 sec latency 37.143 ms oleg356-client.virtnet: 1 21945 0.64 MB/sec execute 139 sec latency 40.925 ms oleg356-client.virtnet: 1 22097 0.63 MB/sec execute 140 sec latency 56.710 ms oleg356-client.virtnet: 1 22237 0.63 MB/sec execute 141 sec latency 51.802 ms oleg356-client.virtnet: 1 22409 0.64 MB/sec execute 142 sec latency 34.126 ms oleg356-client.virtnet: 1 22603 0.63 MB/sec execute 143 sec latency 27.597 ms oleg356-client.virtnet: 1 22894 0.65 MB/sec execute 144 sec latency 20.296 ms oleg356-client.virtnet: 1 23066 0.65 MB/sec execute 145 sec latency 30.020 ms oleg356-client.virtnet: 1 23326 0.64 MB/sec execute 146 sec latency 18.866 ms oleg356-client.virtnet: 1 23554 0.64 MB/sec execute 147 sec latency 28.928 ms oleg356-client.virtnet: 1 23792 0.65 MB/sec execute 148 sec latency 38.420 ms oleg356-client.virtnet: 1 24227 0.67 MB/sec execute 149 sec latency 21.911 ms oleg356-client.virtnet: 1 24666 0.69 MB/sec execute 150 sec latency 25.151 ms oleg356-client.virtnet: 1 24967 0.70 MB/sec execute 151 sec latency 23.406 ms oleg356-client.virtnet: 1 25167 0.71 MB/sec execute 152 sec latency 31.767 ms oleg356-client.virtnet: 1 253Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 23:05:03 (1746068703) targets are mounted 23:05:03 (1746068703) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2072 1285616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37560 3568900 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12076 3591952 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160852 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg356-server Stopping /mnt/lustre-mds2 (opts:) on oleg356-server 23:05:22 (1746068722) shut down facet: mds2 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds2 to oleg356-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 23:05:39 (1746068739) targets are mounted 23:05:39 (1746068739) facet_failover done 33 0.70 MB/sec execute 153 sec latency 29.254 ms oleg356-client.virtnet: 1 25553 0.70 MB/sec execute 154 sec latency 42.303 ms oleg356-client.virtnet: 1 25758 0.70 MB/sec execute 155 sec latency 28.774 ms oleg356-client.virtnet: 1 25965 0.70 MB/sec execute 156 sec latency 26.386 ms oleg356-client.virtnet: 1 26149 0.70 MB/sec execute 157 sec latency 34.937 ms oleg356-client.virtnet: 1 26387 0.71 MB/sec execute 158 sec latency 27.730 ms oleg356-client.virtnet: 1 26568 0.71 MB/sec execute 159 sec latency 36.971 ms oleg356-client.virtnet: 1 26726 0.71 MB/sec execute 160 sec latency 25.339 ms oleg356-client.virtnet: 1 26932 0.70 MB/sec execute 161 sec latency 32.044 ms oleg356-client.virtnet: 1 27128 0.70 MB/sec execute 162 sec latency 37.054 ms oleg356-client.virtnet: 1 27305 0.70 MB/sec execute 163 sec latency 39.981 ms oleg356-client.virtnet: 1 27645 0.72 MB/sec execute 164 sec latency 33.860 ms oleg356-client.virtnet: 1 28081 0.73 MB/sec execute 165 sec latency 31.379 ms oleg356-client.virtnet: 1 28452 0.75 MB/sec execute 166 sec latency 33.204 ms oleg356-client.virtnet: 1 28671 0.76 MB/sec execute 167 sec latency 26.180 ms oleg356-client.virtnet: 1 28817 0.75 MB/sec execute 168 sec latency 33.161 ms oleg356-client.virtnet: 1 28974 0.75 MB/sec execute 169 sec latency 30.255 ms oleg356-client.virtnet: 1 29193 0.75 MB/sec execute 170 sec latency 26.771 ms oleg356-client.virtnet: 1 29348 0.75 MB/sec execute 171 sec latency 121.984 ms oleg356-client.virtnet: 1 29348 0.74 MB/sec execute 172 sec latency 1122.237 ms oleg356-client.virtnet: 1 29348 0.74 MB/sec execute 173 sec latency 2122.608 ms oleg356-client.virtnet: 1 29348 0.73 MB/sec execute 174 sec latency 3122.951 ms oleg356-client.virtnet: 1 29348 0.73 MB/sec execute 175 sec latency 4123.163 ms oleg356-client.virtnet: 1 29348 0.72 MB/sec execute 176 sec latency 5123.340 ms oleg356-client.virtnet: 1 29348 0.72 MB/sec execute 177 sec latency 6123.536 ms oleg356-client.virtnet: 1 29348 0.72 MB/sec execute 178 sec latency 7125.928 ms oleg356-client.virtnet: 1 29348 0.71 MB/sec execute 179 sec latency 8126.103 ms oleg356-client.virtnet: 1 29348 0.71 MB/sec execute 180 sec latency 9126.289 ms oleg356-client.virtnet: 1 29348 0.70 MB/sec execute 181 sec latency 10126.457 ms oleg356-client.virtnet: 1 29348 0.70 MB/sec execute 182 sec latency 11126.956 ms oleg356-client.virtnet: 1 29348 0.70 MB/sec execute 183 sec latency 12129.231 ms oleg356-client.virtnet: 1 29348 0.69 MB/sec execute 184 sec latency 13129.634 ms oleg356-client.virtnet: 1 29348 0.69 MB/sec execute 185 sec latency 14130.093 ms oleg356-client.virtnet: 1 29348 0.69 MB/sec execute 186 sec latency 15130.290 ms oleg356-client.virtnet: 1 29348 0.68 MB/sec execute 187 sec latency 16130.461 ms oleg356-client.virtnet: 1 29348 0.68 MB/sec execute 188 sec latency 17132.012 ms oleg356-client.virtnet: 1 29348 0.67 MB/sec execute 189 sec latency 18133.191 ms oleg356-client.virtnet: 1 29348 0.67 MB/sec execute 190 sec latency 19134.171 ms oleg356-client.virtnet: 1 29348 0.67 MB/sec execute 191 sec latency 20134.357 ms oleg356-client.virtnet: 1 29348 0.66 MB/sec execute 192 sec latency 21134.534 ms oleg356-client.virtnet: 1 29348 0.66 MB/sec execute 193 sec latency 22134.807 ms oleg356-client.virtnet: 1 29348 0.66 MB/sec execute 194 sec latency 23135.190 ms oleg356-client.virtnet: 1 29348 0.65 MB/sec execute 195 sec latency 24135.403 ms oleg356-client.virtnet: 1 29348 0.65 MB/sec execute 196 sec latency 25135.556 ms oleg356-client.virtnet: 1 29348 0.65 MB/sec execuoleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37500 3569012 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11988 3594316 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163328 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg356-server Stopping /mnt/lustre-mds1 (opts:) on oleg356-server 23:05:54 (1746068754) shut down facet: mds1 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds1 to oleg356-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 23:06:11 (1746068771) targets are mounted 23:06:11 (1746068771) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37592 3568988 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11904 3594860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7163848 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg356-server Stopping /mnt/lustre-mds2 (opts:) on oleg356-server 23:06:25 (1746068785) shut down facet: mds2 facet_host: oleg356-server facet_failover_host: oleg356-server te 197 sec latency 26136.174 ms oleg356-client.virtnet: 1 29391 0.64 MB/sec execute 198 sec latency 26886.248 ms oleg356-client.virtnet: 1 29616 0.65 MB/sec execute 199 sec latency 37.973 ms oleg356-client.virtnet: 1 29845 0.66 MB/sec execute 200 sec latency 57.514 ms oleg356-client.virtnet: 1 29999 0.66 MB/sec execute 201 sec latency 46.959 ms oleg356-client.virtnet: 1 30198 0.65 MB/sec execute 202 sec latency 27.860 ms oleg356-client.virtnet: 1 30363 0.65 MB/sec execute 203 sec latency 39.171 ms oleg356-client.virtnet: 1 30559 0.65 MB/sec execute 204 sec latency 38.587 ms oleg356-client.virtnet: 1 30757 0.65 MB/sec execute 205 sec latency 53.124 ms oleg356-client.virtnet: 1 30884 0.65 MB/sec execute 206 sec latency 46.084 ms oleg356-client.virtnet: 1 31167 0.67 MB/sec execute 207 sec latency 69.851 ms oleg356-client.virtnet: 1 31388 0.67 MB/sec execute 208 sec latency 56.997 ms oleg356-client.virtnet: 1 31691 0.67 MB/sec execute 209 sec latency 51.050 ms oleg356-client.virtnet: 1 31979 0.69 MB/sec execute 210 sec latency 46.235 ms oleg356-client.virtnet: 1 32166 0.69 MB/sec execute 211 sec latency 31.345 ms oleg356-client.virtnet: 1 32311 0.69 MB/sec execute 212 sec latency 33.941 ms oleg356-client.virtnet: 1 32468 0.68 MB/sec execute 213 sec latency 26.937 ms oleg356-client.virtnet: 1 32489 0.68 MB/sec execute 214 sec latency 850.664 ms oleg356-client.virtnet: 1 32489 0.68 MB/sec execute 215 sec latency 1850.995 ms oleg356-client.virtnet: 1 32489 0.67 MB/sec execute 216 sec latency 2851.187 ms oleg356-client.virtnet: 1 32489 0.67 MB/sec execute 217 sec latency 3852.748 ms oleg356-client.virtnet: 1 32489 0.67 MB/sec execute 218 sec latency 4852.978 ms oleg356-client.virtnet: 1 32489 0.67 MB/sec execute 219 sec latency 5853.998 ms oleg356-client.virtnet: 1 32489 0.66 MB/sec execute 220 sec latency 6854.621 ms oleg356-client.virtnet: 1 32489 0.66 MB/sec execute 221 sec latency 7854.799 ms oleg356-client.virtnet: 1 32489 0.66 MB/sec execute 222 sec latency 8855.380 ms oleg356-client.virtnet: 1 32489 0.65 MB/sec execute 223 sec latency 9856.153 ms oleg356-client.virtnet: 1 32489 0.65 MB/sec execute 224 sec latency 10856.318 ms oleg356-client.virtnet: 1 32489 0.65 MB/sec execute 225 sec latency 11856.995 ms oleg356-client.virtnet: 1 32489 0.65 MB/sec execute 226 sec latency 12857.218 ms oleg356-client.virtnet: 1 32489 0.64 MB/sec execute 227 sec latency 13858.870 ms oleg356-client.virtnet: 1 32489 0.64 MB/sec execute 228 sec latency 14859.076 ms oleg356-client.virtnet: 1 32489 0.64 MB/sec execute 229 sec latency 15859.263 ms oleg356-client.virtnet: 1 32668 0.63 MB/sec execute 230 sec latency 15922.081 ms oleg356-client.virtnet: 1 32823 0.63 MB/sec execute 231 sec latency 38.019 ms oleg356-client.virtnet: 1 32962 0.63 MB/sec execute 232 sec latency 43.999 ms oleg356-client.virtnet: 1 33194 0.63 MB/sec execute 233 sec latency 39.469 ms oleg356-client.virtnet: 1 33471 0.64 MB/sec execute 234 sec latency 24.209 ms oleg356-client.virtnet: 1 33623 0.64 MB/sec execute 235 sec latency 31.323 ms oleg356-client.virtnet: 1 33780 0.64 MB/sec execute 236 sec latency 35.173 ms oleg356-client.virtnet: 1 33978 0.64 MB/sec execute 237 sec latency 45.987 ms oleg356-client.virtnet: 1 34192 0.64 MB/sec execute 238 sec latency 27.026 ms oleg356-client.virtnet: 1 34407 0.64 MB/sec execute 239 sec latency 34.273 ms oleg356-client.virtnet: 1 34462 0.64 MB/sec execute 240 sec latency 685.312 ms oleg356-client.virtnet: 1 34462 0.64 MB/sec execute 241 sec latency 1685.471 ms oleg356-clienFailover mds2 to oleg356-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 23:06:42 (1746068802) targets are mounted 23:06:42 (1746068802) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec t.virtnet: 1 34462 0.63 MB/sec execute 242 sec latency 2686.306 ms oleg356-client.virtnet: 1 34462 0.63 MB/sec execute 243 sec latency 3686.785 ms oleg356-client.virtnet: 1 34462 0.63 MB/sec execute 244 sec latency 4687.315 ms oleg356-client.virtnet: 1 34462 0.63 MB/sec execute 245 sec latency 5687.781 ms oleg356-client.virtnet: 1 34462 0.62 MB/sec execute 246 sec latency 6688.156 ms oleg356-client.virtnet: 1 34462 0.62 MB/sec execute 247 sec latency 7691.687 ms oleg356-client.virtnet: 1 34462 0.62 MB/sec execute 248 sec latency 8692.381 ms oleg356-client.virtnet: 1 34462 0.62 MB/sec execute 249 sec latency 9693.917 ms oleg356-client.virtnet: 1 34462 0.61 MB/sec execute 250 sec latency 10694.089 ms oleg356-client.virtnet: 1 34462 0.61 MB/sec execute 251 sec latency 11694.882 ms oleg356-client.virtnet: 1 34462 0.61 MB/sec execute 252 sec latency 12697.290 ms oleg356-client.virtnet: 1 34462 0.61 MB/sec execute 253 sec latency 13698.259 ms oleg356-client.virtnet: 1 34462 0.60 MB/sec execute 254 sec latency 14698.442 ms oleg356-client.virtnet: 1 34462 0.60 MB/sec execute 255 sec latency 15699.155 ms oleg356-client.virtnet: 1 34462 0.60 MB/sec execute 256 sec latency 16700.877 ms oleg356-client.virtnet: 1 34462 0.60 MB/sec execute 257 sec latency 17701.643 ms oleg356-client.virtnet: 1 34462 0.60 MB/sec execute 258 sec latency 18701.939 ms oleg356-client.virtnet: 1 34462 0.59 MB/sec execute 259 sec latency 19702.555 ms oleg356-client.virtnet: 1 34462 0.59 MB/sec execute 260 sec latency 20703.313 ms oleg356-client.virtnet: 1 34744 0.60 MB/sec execute 261 sec latency 20783.831 ms oleg356-client.virtnet: 1 34987 0.60 MB/sec execute 262 sec latency 47.274 ms oleg356-client.virtnet: 1 35467 0.62 MB/sec execute 263 sec latency 36.745 ms oleg356-client.virtnet: 1 35640 0.62 MB/sec execute 264 sec latency 44.794 ms oleg356-client.virtnet: 1 35791 0.62 MB/sec execute 265 sec latency 36.449 ms oleg356-client.virtnet: 1 35950 0.62 MB/sec execute 266 sec latency 30.104 ms oleg356-client.virtnet: 1 36136 0.62 MB/sec execute 267 sec latency 25.434 ms oleg356-client.virtnet: 1 36317 0.62 MB/sec execute 268 sec latency 34.901 ms oleg356-client.virtnet: 1 36475 0.62 MB/sec execute 269 sec latency 29.960 ms oleg356-client.virtnet: 1 36655 0.62 MB/sec execute 270 sec latency 40.086 ms oleg356-client.virtnet: 1 36918 0.63 MB/sec execute 271 sec latency 23.979 ms oleg356-client.virtnet: 1 37090 0.63 MB/sec execute 272 sec latency 28.909 ms oleg356-client.virtnet: 1 37237 0.62 MB/sec execute 273 sec latency 35.294 ms oleg356-client.virtnet: 1 37379 0.62 MB/sec execute 274 sec latency 35.086 ms oleg356-client.virtnet: 1 37568 0.62 MB/sec execute 275 sec latency 31.109 ms oleg356-client.virtnet: 1 37852 0.62 MB/sec execute 276 sec latency 25.184 ms oleg356-client.virtnet: 1 38011 0.62 MB/sec execute 277 sec latency 37.840 ms oleg356-client.virtnet: 1 38402 0.64 MB/sec execute 278 sec latency 29.908 ms oleg356-client.virtnet: 1 38741 0.64 MB/sec execute 279 sec latency 30.862 ms oleg356-client.virtnet: 1 39051 0.65 MB/sec execute 280 sec latency 67.874 ms oleg356-client.virtnet: 1 39211 0.65 MB/sec execute 281 sec latency 37.725 ms oleg356-client.virtnet: 1 39398 0.65 MB/sec execute 282 sec latency 28.304 ms oleg356-client.virtnet: 1 39531 0.65 MB/sec execute 283 sec latency 33.680 ms oleg356-client.virtnet: 1 39732 0.65 MB/sec execute 284 sec latency 42.416 ms oleg356-client.virtnet: 1 39873 0.65 MB/sec execute 285 sec latency 38.129 ms oleg356-client.virtnet: 1 40035 0.65 MB/sec execute 286 sec latency 36.641 ms oleg356-client.virtnet: 1 40223 0.65 MB/sec execute 287 sec latency 37.726 ms oleg356-client.virtnet: 1 40468 0.66 MB/sec execute 288 sec latency 40.323 ms oleg356-client.virtnet: 1 40626 0.65 MB/sec execute 289 sec latency 55.443 ms oleg356-client.virtnet: 1 40753 0.65 MB/sec execute 290 sec latency 34.566 ms oleg356-client.virtnet: 1 40928 0.65 MB/sec execute 291 sec latency 35.528 ms oleg356-client.virtnet: 1 41169 0.65 MB/sec execute 292 sec latency 36.612 ms oleg356-client.virtnet: 1 41415 0.65 MB/sec execute 293 sec latency 46.989 ms oleg356-client.virtnet: 1 41616 0.65 MB/sec execute 294 sec latency 36.875 ms oleg356-client.virtnet: 1 41976 0.67 MB/sec execute 295 sec latency 33.626 ms oleg356-client.virtnet: 1 42324 0.67 MB/sec execute 296 sec latency 54.171 ms oleg356-client.virtnet: 1 42618 0.68 MB/sec execute 297 sec latency 32.272 ms oleg356-client.virtnet: 1 42764 0.68 MB/sec execute 298 sec latency 32.124 ms oleg356-client.virtnet: 1 42922 0.68 MB/sec execute 299 sec latency 42.985 ms oleg356-client.virtnet: 1 cleanup 300 sec oleg356-client.virtnet: 0 cleanup 301 sec oleg356-client.virtnet: oleg356-client.virtnet: Operation Count AvgLat MaxLat oleg356-client.virtnet: ---------------------------------------- oleg356-client.virtnet: NTCreateX 6388 23.649 26886.239 oleg356-client.virtnet: Close 4664 3.258 37.353 oleg356-client.virtnet: Rename 272 22.942 57.498 oleg356-client.virtnet: Unlink 1298 5.209 19.789 oleg356-client.virtnet: Qpathinfo 5755 16.336 21527.980 oleg356-client.virtnet: Qfileinfo 1008 0.270 5.160 oleg356-client.virtnet: Qfsinfo 1079 0.614 29.957 oleg356-client.virtnet: Sfileinfo 506 56.081 20920.191 oleg356-client.virtnet: Find 2239 0.849 47.664 oleg356-client.virtnet: WriteX 3154 2.324 46.342 oleg356-client.virtnet: ReadX 10208 0.033 17.713 oleg356-client.virtnet: LockX 22 3.827 8.737 oleg356-client.virtnet: UnlockX 22 5.133 25.373 oleg356-client.virtnet: Flush 440 17.568 118.524 oleg356-client.virtnet: oleg356-client.virtnet: Throughput 0.678208 MB/sec 1 clients 1 procs max_latency=26886.248 ms oleg356-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg356-client.virtnet at Wed Apr 30 23:07:26 EDT 2025 with return code 0 oleg356-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg356-client.virtnet oleg356-client.virtnet: /mnt/lustre/d70b.replay-single/oleg356-client.virtnet /mnt/lustre/d70b.replay-single/oleg356-client.virtnet oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg356-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg356-client.virtnet: removed directory: 'clients/client0' oleg356-client.virtnet: removed directory: 'clients' oleg356-client.virtnet: removed 'client.txt' oleg356-client.virtnet: /mnt/lustre/d70b.replay-single/oleg356-client.virtnet oleg356-client.virtnet: dbench successfully finished