== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 00:03:01 (1746158581) Starting client oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre Started clients oleg338-client.virtnet: 192.168.203.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg338-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14091 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1285568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3556144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7161596 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 00:03:11 (1746158591) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds1 to oleg338-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 00:03:28 (1746158608) targets are mounted 00:03:28 (1746158608) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2396 1285292 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 27440 3567840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1880 3594104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 29320 7161944 1% /mnt/lustre oleg338-client.virtnet: looking for dbench program oleg338-client.virtnet: /usr/bin/dbench oleg338-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg338-client.virtnet oleg338-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg338-client.virtnet' oleg338-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg338-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg338-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg338-client.virtnet at Fri May 2 00:03:03 EDT 2025 oleg338-client.virtnet: waiting for dbench pid 14115 oleg338-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg338-client.virtnet: oleg338-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg338-client.virtnet: failed to create barrier semaphore oleg338-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg338-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg338-client.virtnet: releasing clients oleg338-client.virtnet: 1 199 7.84 MB/sec warmup 1 sec latency 51.511 ms oleg338-client.virtnet: 1 421 7.43 MB/sec warmup 2 sec latency 42.236 ms oleg338-client.virtnet: 1 730 7.10 MB/sec warmup 3 sec latency 27.158 ms oleg338-client.virtnet: 1 902 5.42 MB/sec warmup 4 sec latency 28.480 ms oleg338-client.virtnet: 1 1169 4.60 MB/sec warmup 5 sec latency 24.208 ms oleg338-client.virtnet: 1 1465 4.34 MB/sec warmup 6 sec latency 35.750 ms oleg338-client.virtnet: 1 1655 3.74 MB/sec warmup 7 sec latency 49.312 ms oleg338-client.virtnet: 1 1843 3.29 MB/sec warmup 8 sec latency 41.012 ms oleg338-client.virtnet: 1 2072 2.96 MB/sec warmup 9 sec latency 27.885 ms oleg338-client.virtnet: 1 2376 2.80 MB/sec warmup 10 sec latency 23.633 ms oleg338-client.virtnet: 1 2608 2.64 MB/sec warmup 11 sec latency 24.526 ms oleg338-client.virtnet: 1 3026 2.79 MB/sec warmup 12 sec latency 29.515 ms oleg338-client.virtnet: 1 3543 2.95 MB/sec warmup 13 sec latency 29.050 ms oleg338-client.virtnet: 1 3773 2.84 MB/sec warmup 14 sec latency 26.160 ms oleg338-client.virtnet: 1 3929 2.67 MB/sec warmup 15 sec latency 30.767 ms oleg338-client.virtnet: 1 4083 2.50 MB/sec warmup 16 sec latency 134.692 ms oleg338-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 1136.363 ms oleg338-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 2137.607 ms oleg338-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 3138.329 ms oleg338-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 4138.488 ms oleg338-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 5138.646 ms oleg338-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 6146.104 ms oleg338-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 7146.400 ms oleg338-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 8146.761 ms oleg338-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 9146.928 ms oleg338-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 10147.124 ms oleg338-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 11147.334 ms oleg338-client.virtnet: 1 4283 1.44 MB/sec warmup 28 sec latency 11323.915 ms oleg338-client.virtnet: 1 4455 1.41 MB/sec warmup 29 sec latency 50.572 ms oleg338-client.virtnet: 1 4693 1.40 MB/sec warmup 30 sec latency 29.143 ms oleg338-client.virtnet: 1 5013 1.46 MB/sec warmup 31 sec latency 20.705 ms oleg338-client.virtnet: 1 5215 1.42 MB/sec warmup 32 sec latency 28.608 ms oleg338-client.virtnet: 1 5346 1.38 MB/sec warmup 33 sec latency 47.563 ms oleg338-clietest_70b fail mds2 2 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server 00:03:48 (1746158628) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 00:04:05 (1746158645) targets are mounted 00:04:05 (1746158645) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1285668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37528 3567396 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12176 3594260 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49704 7161656 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 00:04:19 (1746158659) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server nt.virtnet: 1 5449 1.34 MB/sec warmup 34 sec latency 308.620 ms oleg338-client.virtnet: 1 5579 1.30 MB/sec warmup 35 sec latency 582.085 ms oleg338-client.virtnet: 1 5804 1.27 MB/sec warmup 36 sec latency 26.924 ms oleg338-client.virtnet: 1 6028 1.29 MB/sec warmup 37 sec latency 24.723 ms oleg338-client.virtnet: 1 6047 1.26 MB/sec warmup 38 sec latency 903.130 ms oleg338-client.virtnet: 1 6047 1.23 MB/sec warmup 39 sec latency 1903.333 ms oleg338-client.virtnet: 1 6047 1.20 MB/sec warmup 40 sec latency 2903.495 ms oleg338-client.virtnet: 1 6047 1.17 MB/sec warmup 41 sec latency 3904.941 ms oleg338-client.virtnet: 1 6047 1.14 MB/sec warmup 42 sec latency 4906.213 ms oleg338-client.virtnet: 1 6047 1.11 MB/sec warmup 43 sec latency 5906.378 ms oleg338-client.virtnet: 1 6047 1.09 MB/sec warmup 44 sec latency 6906.537 ms oleg338-client.virtnet: 1 6047 1.06 MB/sec warmup 45 sec latency 7906.761 ms oleg338-client.virtnet: 1 6047 1.04 MB/sec warmup 46 sec latency 8906.925 ms oleg338-client.virtnet: 1 6047 1.02 MB/sec warmup 47 sec latency 9907.251 ms oleg338-client.virtnet: 1 6047 1.00 MB/sec warmup 48 sec latency 10907.602 ms oleg338-client.virtnet: 1 6047 0.98 MB/sec warmup 49 sec latency 11907.816 ms oleg338-client.virtnet: 1 6047 0.96 MB/sec warmup 50 sec latency 12907.995 ms oleg338-client.virtnet: 1 6047 0.94 MB/sec warmup 51 sec latency 13909.484 ms oleg338-client.virtnet: 1 6047 0.92 MB/sec warmup 52 sec latency 14910.003 ms oleg338-client.virtnet: 1 6047 0.90 MB/sec warmup 53 sec latency 15910.154 ms oleg338-client.virtnet: 1 6047 0.89 MB/sec warmup 54 sec latency 16910.333 ms oleg338-client.virtnet: 1 6047 0.87 MB/sec warmup 55 sec latency 17913.128 ms oleg338-client.virtnet: 1 6047 0.85 MB/sec warmup 56 sec latency 18913.339 ms oleg338-client.virtnet: 1 6047 0.84 MB/sec warmup 57 sec latency 19913.576 ms oleg338-client.virtnet: 1 6047 0.82 MB/sec warmup 58 sec latency 20914.254 ms oleg338-client.virtnet: 1 6047 0.81 MB/sec warmup 59 sec latency 21914.419 ms oleg338-client.virtnet: 1 6047 0.00 MB/sec execute 1 sec latency 23915.991 ms oleg338-client.virtnet: 1 6047 0.00 MB/sec execute 2 sec latency 24917.594 ms oleg338-client.virtnet: 1 6047 0.00 MB/sec execute 3 sec latency 25918.533 ms oleg338-client.virtnet: 1 6047 0.00 MB/sec execute 4 sec latency 26918.751 ms oleg338-client.virtnet: 1 6191 0.14 MB/sec execute 5 sec latency 27230.780 ms oleg338-client.virtnet: 1 6573 0.80 MB/sec execute 6 sec latency 38.033 ms oleg338-client.virtnet: 1 7101 1.39 MB/sec execute 7 sec latency 33.793 ms oleg338-client.virtnet: 1 7312 1.38 MB/sec execute 8 sec latency 30.152 ms oleg338-client.virtnet: 1 7476 1.26 MB/sec execute 9 sec latency 27.166 ms oleg338-client.virtnet: 1 7599 1.13 MB/sec execute 10 sec latency 40.447 ms oleg338-client.virtnet: 1 7808 1.05 MB/sec execute 11 sec latency 38.245 ms oleg338-client.virtnet: 1 7976 1.00 MB/sec execute 12 sec latency 42.822 ms oleg338-client.virtnet: 1 8192 1.03 MB/sec execute 13 sec latency 27.783 ms oleg338-client.virtnet: 1 8376 0.97 MB/sec execute 14 sec latency 37.208 ms oleg338-client.virtnet: 1 8668 1.10 MB/sec execute 15 sec latency 24.450 ms oleg338-client.virtnet: 1 8844 1.04 MB/sec execute 16 sec latency 29.082 ms oleg338-client.virtnet: 1 9089 0.99 MB/sec execute 17 sec latency 24.276 ms oleg338-client.virtnet: 1 9315 0.95 MB/sec execute 18 sec latency 25.001 ms oleg338-client.virtnet: 1 9570 1.00 MB/sec exFailover mds1 to oleg338-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 00:04:35 (1746158675) targets are mounted 00:04:35 (1746158675) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1285652 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37596 3566484 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12020 3594768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7161252 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server 00:04:49 (1746158689) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 00:05:07 (1746158707) targets are mounted 00:05:07 (1746158707) facet_failover done ecute 19 sec latency 35.762 ms oleg338-client.virtnet: 1 10003 1.16 MB/sec execute 20 sec latency 26.177 ms oleg338-client.virtnet: 1 10464 1.23 MB/sec execute 21 sec latency 28.525 ms oleg338-client.virtnet: 1 10768 1.32 MB/sec execute 22 sec latency 31.446 ms oleg338-client.virtnet: 1 10946 1.31 MB/sec execute 23 sec latency 27.266 ms oleg338-client.virtnet: 1 11099 1.27 MB/sec execute 24 sec latency 27.494 ms oleg338-client.virtnet: 1 11316 1.22 MB/sec execute 25 sec latency 28.201 ms oleg338-client.virtnet: 1 11516 1.20 MB/sec execute 26 sec latency 32.742 ms oleg338-client.virtnet: 1 11699 1.20 MB/sec execute 27 sec latency 31.265 ms oleg338-client.virtnet: 1 11887 1.16 MB/sec execute 28 sec latency 37.188 ms oleg338-client.virtnet: 1 12161 1.23 MB/sec execute 29 sec latency 31.915 ms oleg338-client.virtnet: 1 12341 1.19 MB/sec execute 30 sec latency 31.310 ms oleg338-client.virtnet: 1 12455 1.16 MB/sec execute 31 sec latency 57.564 ms oleg338-client.virtnet: 1 12670 1.13 MB/sec execute 32 sec latency 30.957 ms oleg338-client.virtnet: 1 12899 1.10 MB/sec execute 33 sec latency 29.441 ms oleg338-client.virtnet: 1 13132 1.12 MB/sec execute 34 sec latency 26.044 ms oleg338-client.virtnet: 1 13530 1.19 MB/sec execute 35 sec latency 37.658 ms oleg338-client.virtnet: 1 13847 1.22 MB/sec execute 36 sec latency 34.287 ms oleg338-client.virtnet: 1 14189 1.29 MB/sec execute 37 sec latency 42.231 ms oleg338-client.virtnet: 1 14369 1.27 MB/sec execute 38 sec latency 35.945 ms oleg338-client.virtnet: 1 14523 1.26 MB/sec execute 39 sec latency 41.421 ms oleg338-client.virtnet: 1 14632 1.24 MB/sec execute 40 sec latency 44.235 ms oleg338-client.virtnet: 1 14784 1.21 MB/sec execute 41 sec latency 47.830 ms oleg338-client.virtnet: 1 14913 1.19 MB/sec execute 42 sec latency 433.092 ms oleg338-client.virtnet: 1 15108 1.17 MB/sec execute 43 sec latency 24.802 ms oleg338-client.virtnet: 1 15261 1.17 MB/sec execute 44 sec latency 40.959 ms oleg338-client.virtnet: 1 15365 1.15 MB/sec execute 45 sec latency 290.522 ms oleg338-client.virtnet: 1 15365 1.12 MB/sec execute 46 sec latency 1291.140 ms oleg338-client.virtnet: 1 15365 1.10 MB/sec execute 47 sec latency 2291.819 ms oleg338-client.virtnet: 1 15365 1.08 MB/sec execute 48 sec latency 3292.373 ms oleg338-client.virtnet: 1 15365 1.05 MB/sec execute 49 sec latency 4292.529 ms oleg338-client.virtnet: 1 15365 1.03 MB/sec execute 50 sec latency 5301.862 ms oleg338-client.virtnet: 1 15365 1.01 MB/sec execute 51 sec latency 6302.050 ms oleg338-client.virtnet: 1 15365 0.99 MB/sec execute 52 sec latency 7306.965 ms oleg338-client.virtnet: 1 15365 0.97 MB/sec execute 53 sec latency 8307.620 ms oleg338-client.virtnet: 1 15365 0.96 MB/sec execute 54 sec latency 9312.663 ms oleg338-client.virtnet: 1 15365 0.94 MB/sec execute 55 sec latency 10312.818 ms oleg338-client.virtnet: 1 15365 0.92 MB/sec execute 56 sec latency 11316.181 ms oleg338-client.virtnet: 1 15365 0.91 MB/sec execute 57 sec latency 12316.372 ms oleg338-client.virtnet: 1 15365 0.89 MB/sec execute 58 sec latency 13317.643 ms oleg338-client.virtnet: 1 15365 0.87 MB/sec execute 59 sec latency 14321.161 ms oleg338-client.virtnet: 1 15365 0.86 MB/sec execute 60 sec latency 15321.445 ms oleg338-client.virtnet: 1 15365 0.85 MB/sec execute 61 sec latency 16321.687 ms oleg338-client.virtnet: 1 15365 0.83 MB/sec execute 62 sec latency 17322.640 ms oleg338-client.virtnet: 1 15365 0.82 MB/sec execute 63 sec latency 18324.313 ms oleg338-clieoleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1285668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37600 3569192 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11912 3594996 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49512 7164188 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 00:05:21 (1746158721) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds1 to oleg338-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 00:05:37 (1746158737) targets are mounted 00:05:37 (1746158737) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1285660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37592 3568752 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11904 3594888 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7163640 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server nt.virtnet: 1 15365 0.81 MB/sec execute 64 sec latency 19325.263 ms oleg338-client.virtnet: 1 15365 0.79 MB/sec execute 65 sec latency 20325.433 ms oleg338-client.virtnet: 1 15365 0.78 MB/sec execute 66 sec latency 21325.698 ms oleg338-client.virtnet: 1 15466 0.77 MB/sec execute 67 sec latency 21746.636 ms oleg338-client.virtnet: 1 15690 0.80 MB/sec execute 68 sec latency 53.455 ms oleg338-client.virtnet: 1 15814 0.79 MB/sec execute 69 sec latency 43.174 ms oleg338-client.virtnet: 1 15971 0.78 MB/sec execute 70 sec latency 40.902 ms oleg338-client.virtnet: 1 16175 0.78 MB/sec execute 71 sec latency 30.012 ms oleg338-client.virtnet: 1 16350 0.77 MB/sec execute 72 sec latency 47.945 ms oleg338-client.virtnet: 1 16555 0.77 MB/sec execute 73 sec latency 43.911 ms oleg338-client.virtnet: 1 16748 0.78 MB/sec execute 74 sec latency 39.858 ms oleg338-client.virtnet: 1 17181 0.83 MB/sec execute 75 sec latency 21.539 ms oleg338-client.virtnet: 1 17691 0.88 MB/sec execute 76 sec latency 32.147 ms oleg338-client.virtnet: 1 17868 0.87 MB/sec execute 77 sec latency 27.219 ms oleg338-client.virtnet: 1 18003 0.88 MB/sec execute 78 sec latency 38.742 ms oleg338-client.virtnet: 1 18168 0.87 MB/sec execute 79 sec latency 28.012 ms oleg338-client.virtnet: 1 18266 0.86 MB/sec execute 80 sec latency 433.347 ms oleg338-client.virtnet: 1 18266 0.85 MB/sec execute 81 sec latency 1434.817 ms oleg338-client.virtnet: 1 18266 0.84 MB/sec execute 82 sec latency 2435.023 ms oleg338-client.virtnet: 1 18266 0.83 MB/sec execute 83 sec latency 3435.241 ms oleg338-client.virtnet: 1 18266 0.82 MB/sec execute 84 sec latency 4435.399 ms oleg338-client.virtnet: 1 18266 0.81 MB/sec execute 85 sec latency 5435.550 ms oleg338-client.virtnet: 1 18266 0.80 MB/sec execute 86 sec latency 6439.895 ms oleg338-client.virtnet: 1 18266 0.79 MB/sec execute 87 sec latency 7441.300 ms oleg338-client.virtnet: 1 18266 0.78 MB/sec execute 88 sec latency 8441.508 ms oleg338-client.virtnet: 1 18266 0.77 MB/sec execute 89 sec latency 9441.925 ms oleg338-client.virtnet: 1 18266 0.76 MB/sec execute 90 sec latency 10443.404 ms oleg338-client.virtnet: 1 18266 0.75 MB/sec execute 91 sec latency 11453.725 ms oleg338-client.virtnet: 1 18266 0.75 MB/sec execute 92 sec latency 12454.005 ms oleg338-client.virtnet: 1 18266 0.74 MB/sec execute 93 sec latency 13454.173 ms oleg338-client.virtnet: 1 18266 0.73 MB/sec execute 94 sec latency 14456.747 ms oleg338-client.virtnet: 1 18266 0.72 MB/sec execute 95 sec latency 15456.870 ms oleg338-client.virtnet: 1 18266 0.72 MB/sec execute 96 sec latency 16457.037 ms oleg338-client.virtnet: 1 18266 0.71 MB/sec execute 97 sec latency 17457.214 ms oleg338-client.virtnet: 1 18476 0.70 MB/sec execute 98 sec latency 17484.406 ms oleg338-client.virtnet: 1 18636 0.70 MB/sec execute 99 sec latency 43.717 ms oleg338-client.virtnet: 1 18878 0.71 MB/sec execute 100 sec latency 27.151 ms oleg338-client.virtnet: 1 19135 0.73 MB/sec execute 101 sec latency 26.618 ms oleg338-client.virtnet: 1 19274 0.72 MB/sec execute 102 sec latency 39.426 ms oleg338-client.virtnet: 1 19396 0.72 MB/sec execute 103 sec latency 47.387 ms oleg338-client.virtnet: 1 19520 0.71 MB/sec execute 104 sec latency 46.911 ms oleg338-client.virtnet: 1 19610 0.71 MB/sec execute 105 sec latency 62.093 ms oleg338-client.virtnet: 1 19803 0.70 MB/sec execute 106 sec latency 44.208 ms oleg338-client.virtnet: 1 19987 0.70 MB/sec execute 107 sec latency 39.056 ms oleg338-client.virtnet: 1 20185 0.700:05:52 (1746158752) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 00:06:07 (1746158767) targets are mounted 00:06:07 (1746158767) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1285668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37596 3566636 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12028 3594632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7161268 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 00:06:24 (1746158784) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds1 to oleg338-server mount facets: mds1 0 MB/sec execute 108 sec latency 212.379 ms oleg338-client.virtnet: 1 20185 0.69 MB/sec execute 109 sec latency 1212.541 ms oleg338-client.virtnet: 1 20185 0.69 MB/sec execute 110 sec latency 2213.380 ms oleg338-client.virtnet: 1 20185 0.68 MB/sec execute 111 sec latency 3213.577 ms oleg338-client.virtnet: 1 20185 0.68 MB/sec execute 112 sec latency 4213.766 ms oleg338-client.virtnet: 1 20185 0.67 MB/sec execute 113 sec latency 5215.374 ms oleg338-client.virtnet: 1 20185 0.66 MB/sec execute 114 sec latency 6215.562 ms oleg338-client.virtnet: 1 20185 0.66 MB/sec execute 115 sec latency 7215.824 ms oleg338-client.virtnet: 1 20185 0.65 MB/sec execute 116 sec latency 8216.412 ms oleg338-client.virtnet: 1 20185 0.65 MB/sec execute 117 sec latency 9216.583 ms oleg338-client.virtnet: 1 20185 0.64 MB/sec execute 118 sec latency 10216.863 ms oleg338-client.virtnet: 1 20185 0.64 MB/sec execute 119 sec latency 11217.150 ms oleg338-client.virtnet: 1 20185 0.63 MB/sec execute 120 sec latency 12217.322 ms oleg338-client.virtnet: 1 20185 0.63 MB/sec execute 121 sec latency 13217.541 ms oleg338-client.virtnet: 1 20185 0.62 MB/sec execute 122 sec latency 14217.710 ms oleg338-client.virtnet: 1 20185 0.62 MB/sec execute 123 sec latency 15217.879 ms oleg338-client.virtnet: 1 20185 0.61 MB/sec execute 124 sec latency 16218.049 ms oleg338-client.virtnet: 1 20185 0.61 MB/sec execute 125 sec latency 17218.295 ms oleg338-client.virtnet: 1 20185 0.60 MB/sec execute 126 sec latency 18218.500 ms oleg338-client.virtnet: 1 20185 0.60 MB/sec execute 127 sec latency 19218.668 ms oleg338-client.virtnet: 1 20281 0.60 MB/sec execute 128 sec latency 19530.972 ms oleg338-client.virtnet: 1 20702 0.63 MB/sec execute 129 sec latency 40.342 ms oleg338-client.virtnet: 1 21152 0.64 MB/sec execute 130 sec latency 29.883 ms oleg338-client.virtnet: 1 21494 0.66 MB/sec execute 131 sec latency 19.878 ms oleg338-client.virtnet: 1 21693 0.67 MB/sec execute 132 sec latency 30.932 ms oleg338-client.virtnet: 1 21837 0.66 MB/sec execute 133 sec latency 29.828 ms oleg338-client.virtnet: 1 22044 0.66 MB/sec execute 134 sec latency 35.029 ms oleg338-client.virtnet: 1 22209 0.66 MB/sec execute 135 sec latency 26.628 ms oleg338-client.virtnet: 1 22362 0.66 MB/sec execute 136 sec latency 57.102 ms oleg338-client.virtnet: 1 22533 0.66 MB/sec execute 137 sec latency 39.323 ms oleg338-client.virtnet: 1 22752 0.67 MB/sec execute 138 sec latency 38.233 ms oleg338-client.virtnet: 1 22933 0.67 MB/sec execute 139 sec latency 36.851 ms oleg338-client.virtnet: 1 23121 0.67 MB/sec execute 140 sec latency 34.083 ms oleg338-client.virtnet: 1 23356 0.66 MB/sec execute 141 sec latency 28.792 ms oleg338-client.virtnet: 1 23654 0.67 MB/sec execute 142 sec latency 29.366 ms oleg338-client.virtnet: 1 23901 0.67 MB/sec execute 143 sec latency 23.232 ms oleg338-client.virtnet: 1 24341 0.70 MB/sec execute 144 sec latency 22.923 ms oleg338-client.virtnet: 1 24864 0.73 MB/sec execute 145 sec latency 27.682 ms oleg338-client.virtnet: 1 25090 0.73 MB/sec execute 146 sec latency 28.258 ms oleg338-client.virtnet: 1 25270 0.73 MB/sec execute 147 sec latency 24.057 ms oleg338-client.virtnet: 1 25494 0.72 MB/sec execute 148 sec latency 24.940 ms oleg338-client.virtnet: 1 25702 0.72 MB/sec execute 149 sec latency 28.457 ms oleg338-client.virtnet: 1 25874 0.72 MB/sec execute 150 sec latency 33.372 ms oleg338-client.virtnet: 1 26083 0.72 MB/sec execute 151 sec latency 32.078 ms oleg338-client.virtnet: 1 26350 0.74 MB/sec execute 152 sec latency 30.19Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 00:06:40 (1746158800) targets are mounted 00:06:40 (1746158800) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1285652 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37596 3568780 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12028 3593884 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7162664 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server 00:06:54 (1746158814) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 00:07:10 (1746158830) targets are mounted 00:07:10 (1746158830) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1285668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37592 3569288 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11912 3594968 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49504 7164256 1% /mnt/lustre 7 ms oleg338-client.virtnet: 1 26489 0.73 MB/sec execute 153 sec latency 37.629 ms oleg338-client.virtnet: 1 26608 0.73 MB/sec execute 154 sec latency 63.483 ms oleg338-client.virtnet: 1 26752 0.72 MB/sec execute 155 sec latency 74.323 ms oleg338-client.virtnet: 1 26961 0.72 MB/sec execute 156 sec latency 37.726 ms oleg338-client.virtnet: 1 27171 0.72 MB/sec execute 157 sec latency 46.354 ms oleg338-client.virtnet: 1 27372 0.73 MB/sec execute 158 sec latency 30.005 ms oleg338-client.virtnet: 1 27793 0.75 MB/sec execute 159 sec latency 41.183 ms oleg338-client.virtnet: 1 28242 0.76 MB/sec execute 160 sec latency 38.667 ms oleg338-client.virtnet: 1 28514 0.77 MB/sec execute 161 sec latency 30.548 ms oleg338-client.virtnet: 1 28702 0.78 MB/sec execute 162 sec latency 30.191 ms oleg338-client.virtnet: 1 28851 0.77 MB/sec execute 163 sec latency 36.199 ms oleg338-client.virtnet: 1 29041 0.77 MB/sec execute 164 sec latency 28.533 ms oleg338-client.virtnet: 1 29226 0.77 MB/sec execute 165 sec latency 49.682 ms oleg338-client.virtnet: 1 29375 0.76 MB/sec execute 166 sec latency 58.567 ms oleg338-client.virtnet: 1 29463 0.77 MB/sec execute 167 sec latency 562.195 ms oleg338-client.virtnet: 1 29691 0.76 MB/sec execute 168 sec latency 20.899 ms oleg338-client.virtnet: 1 29956 0.78 MB/sec execute 169 sec latency 34.675 ms oleg338-client.virtnet: 1 30057 0.77 MB/sec execute 170 sec latency 528.795 ms oleg338-client.virtnet: 1 30057 0.77 MB/sec execute 171 sec latency 1529.242 ms oleg338-client.virtnet: 1 30057 0.76 MB/sec execute 172 sec latency 2529.435 ms oleg338-client.virtnet: 1 30057 0.76 MB/sec execute 173 sec latency 3529.600 ms oleg338-client.virtnet: 1 30057 0.75 MB/sec execute 174 sec latency 4531.275 ms oleg338-client.virtnet: 1 30057 0.75 MB/sec execute 175 sec latency 5531.433 ms oleg338-client.virtnet: 1 30057 0.75 MB/sec execute 176 sec latency 6531.700 ms oleg338-client.virtnet: 1 30057 0.74 MB/sec execute 177 sec latency 7531.890 ms oleg338-client.virtnet: 1 30057 0.74 MB/sec execute 178 sec latency 8532.167 ms oleg338-client.virtnet: 1 30057 0.73 MB/sec execute 179 sec latency 9533.661 ms oleg338-client.virtnet: 1 30057 0.73 MB/sec execute 180 sec latency 10533.833 ms oleg338-client.virtnet: 1 30057 0.73 MB/sec execute 181 sec latency 11534.080 ms oleg338-client.virtnet: 1 30057 0.72 MB/sec execute 182 sec latency 12534.370 ms oleg338-client.virtnet: 1 30057 0.72 MB/sec execute 183 sec latency 13534.587 ms oleg338-client.virtnet: 1 30057 0.71 MB/sec execute 184 sec latency 14534.762 ms oleg338-client.virtnet: 1 30057 0.71 MB/sec execute 185 sec latency 15535.637 ms oleg338-client.virtnet: 1 30057 0.71 MB/sec execute 186 sec latency 16535.906 ms oleg338-client.virtnet: 1 30057 0.70 MB/sec execute 187 sec latency 17536.819 ms oleg338-client.virtnet: 1 30057 0.70 MB/sec execute 188 sec latency 18537.103 ms oleg338-client.virtnet: 1 30057 0.69 MB/sec execute 189 sec latency 19537.641 ms oleg338-client.virtnet: 1 30115 0.69 MB/sec execute 190 sec latency 19879.216 ms oleg338-client.virtnet: 1 30216 0.69 MB/sec execute 191 sec latency 45.373 ms oleg338-client.virtnet: 1 30347 0.69 MB/sec execute 192 sec latency 43.701 ms oleg338-client.virtnet: 1 30569 0.68 MB/sec execute 193 sec latency 34.964 ms oleg338-client.virtnet: 1 30788 0.69 MB/sec execute 194 sec latency 34.230 ms oleg338-client.virtnet: 1 30979 0.69 MB/sec execute 195 sec latency 66.435 ms oleg338-client.virtnet: 1 31345 0.71 MB/sec execute 196 sec latency 29.238 ms oleg338-client.virtnet: 1 3test_70b fail mds1 9 times Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 00:07:24 (1746158844) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds1 to oleg338-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 00:07:40 (1746158860) targets are mounted 00:07:40 (1746158860) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1285660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37668 3568924 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11904 3594776 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49572 7163700 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg338-server Stopping /mnt/lustre-mds2 (opts:) on oleg338-server 00:07:55 (1746158875) shut down facet: mds2 facet_host: oleg338-server facet_failover_host: oleg338-server 1774 0.71 MB/sec execute 197 sec latency 32.652 ms oleg338-client.virtnet: 1 32089 0.73 MB/sec execute 198 sec latency 24.438 ms oleg338-client.virtnet: 1 32262 0.73 MB/sec execute 199 sec latency 35.348 ms oleg338-client.virtnet: 1 32401 0.73 MB/sec execute 200 sec latency 49.979 ms oleg338-client.virtnet: 1 32489 0.72 MB/sec execute 201 sec latency 324.954 ms oleg338-client.virtnet: 1 32489 0.72 MB/sec execute 202 sec latency 1325.148 ms oleg338-client.virtnet: 1 32489 0.72 MB/sec execute 203 sec latency 2325.348 ms oleg338-client.virtnet: 1 32489 0.71 MB/sec execute 204 sec latency 3325.516 ms oleg338-client.virtnet: 1 32489 0.71 MB/sec execute 205 sec latency 4326.188 ms oleg338-client.virtnet: 1 32489 0.70 MB/sec execute 206 sec latency 5326.970 ms oleg338-client.virtnet: 1 32489 0.70 MB/sec execute 207 sec latency 6327.218 ms oleg338-client.virtnet: 1 32489 0.70 MB/sec execute 208 sec latency 7327.889 ms oleg338-client.virtnet: 1 32489 0.69 MB/sec execute 209 sec latency 8328.269 ms oleg338-client.virtnet: 1 32489 0.69 MB/sec execute 210 sec latency 9328.717 ms oleg338-client.virtnet: 1 32489 0.69 MB/sec execute 211 sec latency 10328.944 ms oleg338-client.virtnet: 1 32489 0.68 MB/sec execute 212 sec latency 11330.590 ms oleg338-client.virtnet: 1 32489 0.68 MB/sec execute 213 sec latency 12330.721 ms oleg338-client.virtnet: 1 32489 0.68 MB/sec execute 214 sec latency 13330.943 ms oleg338-client.virtnet: 1 32489 0.68 MB/sec execute 215 sec latency 14331.360 ms oleg338-client.virtnet: 1 32489 0.67 MB/sec execute 216 sec latency 15331.731 ms oleg338-client.virtnet: 1 32489 0.67 MB/sec execute 217 sec latency 16332.134 ms oleg338-client.virtnet: 1 32489 0.67 MB/sec execute 218 sec latency 17332.340 ms oleg338-client.virtnet: 1 32489 0.66 MB/sec execute 219 sec latency 18332.500 ms oleg338-client.virtnet: 1 32548 0.66 MB/sec execute 220 sec latency 19086.671 ms oleg338-client.virtnet: 1 32745 0.66 MB/sec execute 221 sec latency 48.469 ms oleg338-client.virtnet: 1 32878 0.66 MB/sec execute 222 sec latency 52.335 ms oleg338-client.virtnet: 1 33075 0.66 MB/sec execute 223 sec latency 31.333 ms oleg338-client.virtnet: 1 33228 0.66 MB/sec execute 224 sec latency 33.115 ms oleg338-client.virtnet: 1 33459 0.67 MB/sec execute 225 sec latency 60.438 ms oleg338-client.virtnet: 1 33616 0.67 MB/sec execute 226 sec latency 33.747 ms oleg338-client.virtnet: 1 33753 0.66 MB/sec execute 227 sec latency 51.733 ms oleg338-client.virtnet: 1 33827 0.66 MB/sec execute 228 sec latency 169.571 ms oleg338-client.virtnet: 1 34071 0.66 MB/sec execute 229 sec latency 28.922 ms oleg338-client.virtnet: 1 34270 0.66 MB/sec execute 230 sec latency 48.314 ms oleg338-client.virtnet: 1 34354 0.66 MB/sec execute 231 sec latency 471.943 ms oleg338-client.virtnet: 1 34354 0.66 MB/sec execute 232 sec latency 1473.174 ms oleg338-client.virtnet: 1 34354 0.65 MB/sec execute 233 sec latency 2473.591 ms oleg338-client.virtnet: 1 34354 0.65 MB/sec execute 234 sec latency 3474.345 ms oleg338-client.virtnet: 1 34354 0.65 MB/sec execute 235 sec latency 4474.569 ms oleg338-client.virtnet: 1 34354 0.64 MB/sec execute 236 sec latency 5474.792 ms oleg338-client.virtnet: 1 34354 0.64 MB/sec execute 237 sec latency 6474.998 ms oleg338-client.virtnet: 1 34354 0.64 MB/sec execute 238 sec latency 7475.164 ms oleg338-client.virtnet: 1 34354 0.64 MB/sec execute 239 sec latency 8475.336 ms oleg338-client.virtnet: 1 34354 0.63 MB/sec execute 240 sec latency 9475.523 ms oleg338-client.virtnet: 1 34354 0.63 MB/sec eFailover mds2 to oleg338-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 00:08:12 (1746158892) targets are mounted 00:08:12 (1746158892) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec xecute 241 sec latency 10476.804 ms oleg338-client.virtnet: 1 34354 0.63 MB/sec execute 242 sec latency 11476.976 ms oleg338-client.virtnet: 1 34354 0.63 MB/sec execute 243 sec latency 12477.461 ms oleg338-client.virtnet: 1 34354 0.62 MB/sec execute 244 sec latency 13477.846 ms oleg338-client.virtnet: 1 34354 0.62 MB/sec execute 245 sec latency 14479.795 ms oleg338-client.virtnet: 1 34354 0.62 MB/sec execute 246 sec latency 15480.414 ms oleg338-client.virtnet: 1 34354 0.62 MB/sec execute 247 sec latency 16480.575 ms oleg338-client.virtnet: 1 34354 0.61 MB/sec execute 248 sec latency 17480.753 ms oleg338-client.virtnet: 1 34354 0.61 MB/sec execute 249 sec latency 18481.347 ms oleg338-client.virtnet: 1 34354 0.61 MB/sec execute 250 sec latency 19481.953 ms oleg338-client.virtnet: 1 34354 0.61 MB/sec execute 251 sec latency 20482.134 ms oleg338-client.virtnet: 1 34404 0.60 MB/sec execute 252 sec latency 20962.058 ms oleg338-client.virtnet: 1 34597 0.61 MB/sec execute 253 sec latency 45.480 ms oleg338-client.virtnet: 1 34968 0.62 MB/sec execute 254 sec latency 34.318 ms oleg338-client.virtnet: 1 35454 0.64 MB/sec execute 255 sec latency 45.990 ms oleg338-client.virtnet: 1 35640 0.64 MB/sec execute 256 sec latency 28.622 ms oleg338-client.virtnet: 1 35819 0.64 MB/sec execute 257 sec latency 27.064 ms oleg338-client.virtnet: 1 35979 0.64 MB/sec execute 258 sec latency 35.808 ms oleg338-client.virtnet: 1 36169 0.63 MB/sec execute 259 sec latency 44.319 ms oleg338-client.virtnet: 1 36317 0.63 MB/sec execute 260 sec latency 40.778 ms oleg338-client.virtnet: 1 36453 0.63 MB/sec execute 261 sec latency 47.807 ms oleg338-client.virtnet: 1 36665 0.63 MB/sec execute 262 sec latency 39.045 ms oleg338-client.virtnet: 1 36914 0.64 MB/sec execute 263 sec latency 33.638 ms oleg338-client.virtnet: 1 37076 0.64 MB/sec execute 264 sec latency 35.080 ms oleg338-client.virtnet: 1 37243 0.64 MB/sec execute 265 sec latency 31.616 ms oleg338-client.virtnet: 1 37419 0.64 MB/sec execute 266 sec latency 39.309 ms oleg338-client.virtnet: 1 37609 0.64 MB/sec execute 267 sec latency 38.121 ms oleg338-client.virtnet: 1 37846 0.64 MB/sec execute 268 sec latency 40.470 ms oleg338-client.virtnet: 1 38074 0.64 MB/sec execute 269 sec latency 34.217 ms oleg338-client.virtnet: 1 38426 0.65 MB/sec execute 270 sec latency 32.255 ms oleg338-client.virtnet: 1 38736 0.66 MB/sec execute 271 sec latency 42.475 ms oleg338-client.virtnet: 1 39072 0.67 MB/sec execute 272 sec latency 38.934 ms oleg338-client.virtnet: 1 39233 0.67 MB/sec execute 273 sec latency 33.491 ms oleg338-client.virtnet: 1 39380 0.67 MB/sec execute 274 sec latency 37.363 ms oleg338-client.virtnet: 1 39523 0.67 MB/sec execute 275 sec latency 32.752 ms oleg338-client.virtnet: 1 39707 0.66 MB/sec execute 276 sec latency 67.795 ms oleg338-client.virtnet: 1 39875 0.66 MB/sec execute 277 sec latency 29.309 ms oleg338-client.virtnet: 1 40056 0.66 MB/sec execute 278 sec latency 26.530 ms oleg338-client.virtnet: 1 40313 0.66 MB/sec execute 279 sec latency 33.151 ms oleg338-client.virtnet: 1 40630 0.67 MB/sec execute 280 sec latency 32.387 ms oleg338-client.virtnet: 1 40870 0.67 MB/sec execute 281 sec latency 21.515 ms oleg338-client.virtnet: 1 41122 0.67 MB/sec execute 282 sec latency 27.875 ms oleg338-client.virtnet: 1 41393 0.67 MB/sec execute 283 sec latency 27.117 ms oleg338-client.virtnet: 1 41555 0.67 MB/sec execute 284 sec latency 31.643 ms oleg338-client.virtnet: 1 41985 0.69 MB/sec execute 285 sec latency 31.965 ms oleg338-client.virtnet: 1 42419 0.69 MB/sec execute 286 sec latency 25.651 ms oleg338-client.virtnet: 1 42689 0.70 MB/sec execute 287 sec latency 39.747 ms oleg338-client.virtnet: 1 42866 0.70 MB/sec execute 288 sec latency 33.609 ms oleg338-client.virtnet: 1 43036 0.70 MB/sec execute 289 sec latency 24.137 ms oleg338-client.virtnet: 1 43252 0.70 MB/sec execute 290 sec latency 29.670 ms oleg338-client.virtnet: 1 43408 0.70 MB/sec execute 291 sec latency 45.928 ms oleg338-client.virtnet: 1 43587 0.70 MB/sec execute 292 sec latency 29.739 ms oleg338-client.virtnet: 1 43767 0.70 MB/sec execute 293 sec latency 46.654 ms oleg338-client.virtnet: 1 44026 0.71 MB/sec execute 294 sec latency 33.797 ms oleg338-client.virtnet: 1 44189 0.70 MB/sec execute 295 sec latency 36.910 ms oleg338-client.virtnet: 1 44365 0.70 MB/sec execute 296 sec latency 38.079 ms oleg338-client.virtnet: 1 44512 0.70 MB/sec execute 297 sec latency 31.810 ms oleg338-client.virtnet: 1 44739 0.70 MB/sec execute 298 sec latency 31.762 ms oleg338-client.virtnet: 1 44972 0.70 MB/sec execute 299 sec latency 46.161 ms oleg338-client.virtnet: 1 cleanup 300 sec oleg338-client.virtnet: 0 cleanup 301 sec oleg338-client.virtnet: oleg338-client.virtnet: Operation Count AvgLat MaxLat oleg338-client.virtnet: ---------------------------------------- oleg338-client.virtnet: NTCreateX 6765 25.123 27230.769 oleg338-client.virtnet: Close 4971 3.002 38.187 oleg338-client.virtnet: Rename 288 93.693 20962.044 oleg338-client.virtnet: Unlink 1365 4.667 17.451 oleg338-client.virtnet: Qpathinfo 6131 9.505 19086.649 oleg338-client.virtnet: Qfileinfo 1068 0.246 7.319 oleg338-client.virtnet: Qfsinfo 1124 0.495 20.227 oleg338-client.virtnet: Sfileinfo 552 13.631 48.458 oleg338-client.virtnet: Find 2373 0.736 33.966 oleg338-client.virtnet: WriteX 3337 2.136 34.916 oleg338-client.virtnet: ReadX 10642 0.053 24.090 oleg338-client.virtnet: LockX 22 2.849 7.142 oleg338-client.virtnet: UnlockX 22 2.738 6.996 oleg338-client.virtnet: Flush 473 59.230 19879.200 oleg338-client.virtnet: oleg338-client.virtnet: Throughput 0.700404 MB/sec 1 clients 1 procs max_latency=27230.780 ms oleg338-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg338-client.virtnet at Fri May 2 00:09:04 EDT 2025 with return code 0 oleg338-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg338-client.virtnet oleg338-client.virtnet: /mnt/lustre/d70b.replay-single/oleg338-client.virtnet /mnt/lustre/d70b.replay-single/oleg338-client.virtnet oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg338-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg338-client.virtnet: removed directory: 'clients/client0' oleg338-client.virtnet: removed directory: 'clients' oleg338-client.virtnet: removed 'client.txt' oleg338-client.virtnet: /mnt/lustre/d70b.replay-single/oleg338-client.virtnet oleg338-client.virtnet: dbench successfully finished