== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 18:25:50 (1746138350) Starting client oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre Started clients oleg325-client.virtnet: 192.168.203.125@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg325-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14106 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1285568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1584 3605436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26060 3556216 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27644 7161652 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg325-server Stopping /mnt/lustre-mds1 (opts:) on oleg325-server 18:26:01 (1746138361) shut down facet: mds1 facet_host: oleg325-server facet_failover_host: oleg325-server Failover mds1 to oleg325-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0000 18:26:17 (1746138377) targets are mounted 18:26:17 (1746138377) facet_failover done oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2396 1285292 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1744 3592936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 27424 3568828 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 29168 7161764 1% /mnt/lustre oleg325-client.virtnet: looking for dbench program oleg325-client.virtnet: /usr/bin/dbench oleg325-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg325-client.virtnet oleg325-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg325-client.virtnet' oleg325-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg325-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg325-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg325-client.virtnet at Thu May 1 18:25:52 EDT 2025 oleg325-client.virtnet: waiting for dbench pid 14131 oleg325-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg325-client.virtnet: oleg325-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg325-client.virtnet: failed to create barrier semaphore oleg325-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg325-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg325-client.virtnet: releasing clients oleg325-client.virtnet: 1 190 7.58 MB/sec warmup 1 sec latency 43.552 ms oleg325-client.virtnet: 1 389 6.90 MB/sec warmup 2 sec latency 32.287 ms oleg325-client.virtnet: 1 649 7.01 MB/sec warmup 3 sec latency 47.061 ms oleg325-client.virtnet: 1 799 5.36 MB/sec warmup 4 sec latency 38.536 ms oleg325-client.virtnet: 1 933 4.35 MB/sec warmup 5 sec latency 62.408 ms oleg325-client.virtnet: 1 1155 3.83 MB/sec warmup 6 sec latency 33.584 ms oleg325-client.virtnet: 1 1406 3.71 MB/sec warmup 7 sec latency 39.042 ms oleg325-client.virtnet: 1 1534 3.26 MB/sec warmup 8 sec latency 56.453 ms oleg325-client.virtnet: 1 1721 2.92 MB/sec warmup 9 sec latency 46.580 ms oleg325-client.virtnet: 1 2004 2.65 MB/sec warmup 10 sec latency 24.883 ms oleg325-client.virtnet: 1 2289 2.52 MB/sec warmup 11 sec latency 22.044 ms oleg325-client.virtnet: 1 2487 2.39 MB/sec warmup 12 sec latency 26.238 ms oleg325-client.virtnet: 1 2973 2.58 MB/sec warmup 13 sec latency 23.735 ms oleg325-client.virtnet: 1 3373 2.53 MB/sec warmup 14 sec latency 29.471 ms oleg325-client.virtnet: 1 3729 2.59 MB/sec warmup 15 sec latency 21.803 ms oleg325-client.virtnet: 1 3948 2.50 MB/sec warmup 16 sec latency 28.465 ms oleg325-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 155.437 ms oleg325-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 1156.356 ms oleg325-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 2156.523 ms oleg325-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 3156.728 ms oleg325-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 4156.916 ms oleg325-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 5157.581 ms oleg325-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 6157.943 ms oleg325-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 7158.090 ms oleg325-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 8158.352 ms oleg325-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 9158.680 ms oleg325-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 10158.871 ms oleg325-client.virtnet: 1 4119 1.43 MB/sec warmup 28 sec latency 11027.702 ms oleg325-client.virtnet: 1 4374 1.40 MB/sec warmup 29 sec latency 33.211 ms oleg325-client.virtnet: 1 4552 1.37 MB/sec warmup 30 sec latency 34.158 ms oleg325-client.virtnet: 1 4805 1.36 MB/sec warmup 31 sec latency 28.277 ms oleg325-client.virtnet: 1 5093 1.41 MB/sec warmup 32 sec latency 24.366 ms oleg325-client.virtnet: 1 5260 1.37 MB/sec warmup 33 sec latency 46.144 ms oleg325-client.test_70b fail mds2 2 times Failing mds2 on oleg325-server Stopping /mnt/lustre-mds2 (opts:) on oleg325-server 18:26:30 (1746138390) shut down facet: mds2 facet_host: oleg325-server facet_failover_host: oleg325-server Failover mds2 to oleg325-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0001 18:26:45 (1746138405) targets are mounted 18:26:45 (1746138405) facet_failover done oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1285668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13140 3591544 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36484 3569064 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7160608 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg325-server Stopping /mnt/lustre-mds1 (opts:) on oleg325-server 18:27:01 (1746138421) shut down facet: mds1 facet_host: oleg325-server facet_failover_host: oleg325-server Failover mds1 to oleg325-server virtnet: 1 5442 1.34 MB/sec warmup 34 sec latency 34.739 ms oleg325-client.virtnet: 1 5587 1.30 MB/sec warmup 35 sec latency 469.996 ms oleg325-client.virtnet: 1 5948 1.31 MB/sec warmup 36 sec latency 22.589 ms oleg325-client.virtnet: 1 6451 1.40 MB/sec warmup 37 sec latency 116.068 ms oleg325-client.virtnet: 1 6451 1.37 MB/sec warmup 38 sec latency 1116.933 ms oleg325-client.virtnet: 1 6451 1.33 MB/sec warmup 39 sec latency 2117.103 ms oleg325-client.virtnet: 1 6451 1.30 MB/sec warmup 40 sec latency 3117.279 ms oleg325-client.virtnet: 1 6451 1.27 MB/sec warmup 41 sec latency 4117.576 ms oleg325-client.virtnet: 1 6451 1.24 MB/sec warmup 42 sec latency 5118.923 ms oleg325-client.virtnet: 1 6451 1.21 MB/sec warmup 43 sec latency 6119.191 ms oleg325-client.virtnet: 1 6451 1.18 MB/sec warmup 44 sec latency 7119.413 ms oleg325-client.virtnet: 1 6451 1.15 MB/sec warmup 45 sec latency 8119.622 ms oleg325-client.virtnet: 1 6451 1.13 MB/sec warmup 46 sec latency 9120.006 ms oleg325-client.virtnet: 1 6451 1.10 MB/sec warmup 47 sec latency 10120.452 ms oleg325-client.virtnet: 1 6451 1.08 MB/sec warmup 48 sec latency 11127.402 ms oleg325-client.virtnet: 1 6451 1.06 MB/sec warmup 49 sec latency 12127.584 ms oleg325-client.virtnet: 1 6451 1.04 MB/sec warmup 50 sec latency 13127.751 ms oleg325-client.virtnet: 1 6451 1.02 MB/sec warmup 51 sec latency 14127.949 ms oleg325-client.virtnet: 1 6451 1.00 MB/sec warmup 52 sec latency 15128.441 ms oleg325-client.virtnet: 1 6451 0.98 MB/sec warmup 53 sec latency 16128.596 ms oleg325-client.virtnet: 1 6451 0.96 MB/sec warmup 54 sec latency 17128.787 ms oleg325-client.virtnet: 1 6451 0.94 MB/sec warmup 55 sec latency 18129.281 ms oleg325-client.virtnet: 1 6451 0.93 MB/sec warmup 56 sec latency 19129.445 ms oleg325-client.virtnet: 1 6472 0.92 MB/sec warmup 57 sec latency 20021.696 ms oleg325-client.virtnet: 1 6801 0.93 MB/sec warmup 58 sec latency 38.294 ms oleg325-client.virtnet: 1 7114 0.97 MB/sec warmup 59 sec latency 40.942 ms oleg325-client.virtnet: 1 7438 0.29 MB/sec execute 1 sec latency 58.445 ms oleg325-client.virtnet: 1 7566 0.15 MB/sec execute 2 sec latency 42.313 ms oleg325-client.virtnet: 1 7685 0.11 MB/sec execute 3 sec latency 40.841 ms oleg325-client.virtnet: 1 7825 0.14 MB/sec execute 4 sec latency 48.032 ms oleg325-client.virtnet: 1 7986 0.20 MB/sec execute 5 sec latency 38.893 ms oleg325-client.virtnet: 1 8161 0.37 MB/sec execute 6 sec latency 34.568 ms oleg325-client.virtnet: 1 8327 0.35 MB/sec execute 7 sec latency 34.124 ms oleg325-client.virtnet: 1 8562 0.67 MB/sec execute 8 sec latency 35.165 ms oleg325-client.virtnet: 1 8711 0.61 MB/sec execute 9 sec latency 60.753 ms oleg325-client.virtnet: 1 8885 0.56 MB/sec execute 10 sec latency 34.202 ms oleg325-client.virtnet: 1 9093 0.53 MB/sec execute 11 sec latency 32.380 ms oleg325-client.virtnet: 1 9290 0.51 MB/sec execute 12 sec latency 27.360 ms oleg325-client.virtnet: 1 9530 0.57 MB/sec execute 13 sec latency 27.350 ms oleg325-client.virtnet: 1 9735 0.63 MB/sec execute 14 sec latency 46.037 ms oleg325-client.virtnet: 1 10099 0.86 MB/sec execute 15 sec latency 35.482 ms oleg325-client.virtnet: 1 10473 0.92 MB/sec execute 16 sec latency 45.761 ms oleg325-client.virtnet: 1 10732 1.06 MB/sec execute 17 sec latency 29.927 ms oleg325-client.virtnet: 1 10914 1.06 MB/sec execute 18 sec latency 30.477 ms oleg325-client.virtnet: 1 11065 1.02 MB/sec execute 19 sec latencymount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0000 18:27:18 (1746138438) targets are mounted 18:27:18 (1746138438) facet_failover done oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1285652 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13060 3592924 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36556 3567404 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160328 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg325-server Stopping /mnt/lustre-mds2 (opts:) on oleg325-server 18:27:35 (1746138455) shut down facet: mds2 facet_host: oleg325-server facet_failover_host: oleg325-server Failover mds2 to oleg325-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0001 18:27:52 (1746138472) targets are mounted 18:27:52 (1746138472) facet_failover done oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 36.456 ms oleg325-client.virtnet: 1 11196 0.97 MB/sec execute 20 sec latency 30.223 ms oleg325-client.virtnet: 1 11372 0.94 MB/sec execute 21 sec latency 83.250 ms oleg325-client.virtnet: 1 11464 0.91 MB/sec execute 22 sec latency 50.537 ms oleg325-client.virtnet: 1 11578 0.88 MB/sec execute 23 sec latency 48.320 ms oleg325-client.virtnet: 1 11720 0.89 MB/sec execute 24 sec latency 52.159 ms oleg325-client.virtnet: 1 11891 0.86 MB/sec execute 25 sec latency 32.611 ms oleg325-client.virtnet: 1 12160 0.94 MB/sec execute 26 sec latency 24.925 ms oleg325-client.virtnet: 1 12344 0.91 MB/sec execute 27 sec latency 32.276 ms oleg325-client.virtnet: 1 12526 0.89 MB/sec execute 28 sec latency 24.359 ms oleg325-client.virtnet: 1 12781 0.87 MB/sec execute 29 sec latency 28.709 ms oleg325-client.virtnet: 1 13004 0.88 MB/sec execute 30 sec latency 51.637 ms oleg325-client.virtnet: 1 13141 0.88 MB/sec execute 31 sec latency 44.805 ms oleg325-client.virtnet: 1 13581 1.00 MB/sec execute 32 sec latency 32.760 ms oleg325-client.virtnet: 1 14152 1.12 MB/sec execute 33 sec latency 24.270 ms oleg325-client.virtnet: 1 14364 1.10 MB/sec execute 34 sec latency 22.134 ms oleg325-client.virtnet: 1 14544 1.10 MB/sec execute 35 sec latency 28.259 ms oleg325-client.virtnet: 1 14670 1.07 MB/sec execute 36 sec latency 36.477 ms oleg325-client.virtnet: 1 14850 1.04 MB/sec execute 37 sec latency 39.699 ms oleg325-client.virtnet: 1 15005 1.03 MB/sec execute 38 sec latency 41.845 ms oleg325-client.virtnet: 1 15118 1.01 MB/sec execute 39 sec latency 43.718 ms oleg325-client.virtnet: 1 15193 0.99 MB/sec execute 40 sec latency 365.254 ms oleg325-client.virtnet: 1 15193 0.96 MB/sec execute 41 sec latency 1367.341 ms oleg325-client.virtnet: 1 15193 0.94 MB/sec execute 42 sec latency 2367.463 ms oleg325-client.virtnet: 1 15193 0.92 MB/sec execute 43 sec latency 3367.965 ms oleg325-client.virtnet: 1 15193 0.90 MB/sec execute 44 sec latency 4368.412 ms oleg325-client.virtnet: 1 15193 0.88 MB/sec execute 45 sec latency 5368.583 ms oleg325-client.virtnet: 1 15193 0.86 MB/sec execute 46 sec latency 6368.754 ms oleg325-client.virtnet: 1 15193 0.84 MB/sec execute 47 sec latency 7368.942 ms oleg325-client.virtnet: 1 15193 0.82 MB/sec execute 48 sec latency 8369.188 ms oleg325-client.virtnet: 1 15193 0.81 MB/sec execute 49 sec latency 9369.607 ms oleg325-client.virtnet: 1 15193 0.79 MB/sec execute 50 sec latency 10370.324 ms oleg325-client.virtnet: 1 15193 0.77 MB/sec execute 51 sec latency 11370.765 ms oleg325-client.virtnet: 1 15193 0.76 MB/sec execute 52 sec latency 12371.666 ms oleg325-client.virtnet: 1 15193 0.74 MB/sec execute 53 sec latency 13373.892 ms oleg325-client.virtnet: 1 15193 0.73 MB/sec execute 54 sec latency 14374.251 ms oleg325-client.virtnet: 1 15193 0.72 MB/sec execute 55 sec latency 15375.635 ms oleg325-client.virtnet: 1 15193 0.71 MB/sec execute 56 sec latency 16376.044 ms oleg325-client.virtnet: 1 15193 0.69 MB/sec execute 57 sec latency 17376.660 ms oleg325-client.virtnet: 1 15193 0.68 MB/sec execute 58 sec latency 18376.905 ms oleg325-client.virtnet: 1 15193 0.67 MB/sec execute 59 sec latency 19377.390 ms oleg325-client.virtnet: 1 15193 0.66 MB/sec execute 60 sec latency 20377.567 ms oleg325-client.virtnet: 1 15193 0.65 MB/sec execute 61 sec latency 21379.367 ms oleg325-client.virtnet: 1 15193 0.64 MB/sec execute 62 sec latency 22380.314 ms oleg325-client.virtnet: 1 15199 0.63 MB/sec execute 63 sec latency 23281.970 ms oleg325-client.virtnmdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1285668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12952 3593956 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36548 3570216 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7164172 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg325-server Stopping /mnt/lustre-mds1 (opts:) on oleg325-server 18:28:07 (1746138487) shut down facet: mds1 facet_host: oleg325-server facet_failover_host: oleg325-server Failover mds1 to oleg325-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0000 18:28:24 (1746138504) targets are mounted 18:28:25 (1746138505) facet_failover done oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1285660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12944 3592868 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36620 3570168 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7163036 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg325-server Stopping /mnt/lustre-mds2 (opts:) on oleg325-server 18:28:39 (1746138519) shut down facet: mds2 facet_host: oleg325-server facet_failover_host: oleg325-server et: 1 15392 0.63 MB/sec execute 64 sec latency 40.207 ms oleg325-client.virtnet: 1 15623 0.67 MB/sec execute 65 sec latency 34.226 ms oleg325-client.virtnet: 1 15788 0.66 MB/sec execute 66 sec latency 35.605 ms oleg325-client.virtnet: 1 15950 0.65 MB/sec execute 67 sec latency 38.991 ms oleg325-client.virtnet: 1 16087 0.65 MB/sec execute 68 sec latency 44.985 ms oleg325-client.virtnet: 1 16281 0.64 MB/sec execute 69 sec latency 42.646 ms oleg325-client.virtnet: 1 16461 0.64 MB/sec execute 70 sec latency 56.026 ms oleg325-client.virtnet: 1 16625 0.64 MB/sec execute 71 sec latency 45.168 ms oleg325-client.virtnet: 1 16911 0.69 MB/sec execute 72 sec latency 49.131 ms oleg325-client.virtnet: 1 17362 0.71 MB/sec execute 73 sec latency 32.296 ms oleg325-client.virtnet: 1 17582 0.72 MB/sec execute 74 sec latency 39.429 ms oleg325-client.virtnet: 1 17807 0.75 MB/sec execute 75 sec latency 49.549 ms oleg325-client.virtnet: 1 17968 0.75 MB/sec execute 76 sec latency 37.554 ms oleg325-client.virtnet: 1 18115 0.75 MB/sec execute 77 sec latency 44.125 ms oleg325-client.virtnet: 1 18258 0.74 MB/sec execute 78 sec latency 45.213 ms oleg325-client.virtnet: 1 18266 0.73 MB/sec execute 79 sec latency 949.977 ms oleg325-client.virtnet: 1 18266 0.72 MB/sec execute 80 sec latency 1950.224 ms oleg325-client.virtnet: 1 18266 0.71 MB/sec execute 81 sec latency 2951.815 ms oleg325-client.virtnet: 1 18266 0.70 MB/sec execute 82 sec latency 3952.002 ms oleg325-client.virtnet: 1 18266 0.69 MB/sec execute 83 sec latency 4952.942 ms oleg325-client.virtnet: 1 18266 0.69 MB/sec execute 84 sec latency 5953.330 ms oleg325-client.virtnet: 1 18266 0.68 MB/sec execute 85 sec latency 6953.563 ms oleg325-client.virtnet: 1 18266 0.67 MB/sec execute 86 sec latency 7953.750 ms oleg325-client.virtnet: 1 18266 0.66 MB/sec execute 87 sec latency 8953.977 ms oleg325-client.virtnet: 1 18266 0.66 MB/sec execute 88 sec latency 9954.131 ms oleg325-client.virtnet: 1 18266 0.65 MB/sec execute 89 sec latency 10954.274 ms oleg325-client.virtnet: 1 18266 0.64 MB/sec execute 90 sec latency 11954.779 ms oleg325-client.virtnet: 1 18266 0.63 MB/sec execute 91 sec latency 12957.425 ms oleg325-client.virtnet: 1 18266 0.63 MB/sec execute 92 sec latency 13958.717 ms oleg325-client.virtnet: 1 18266 0.62 MB/sec execute 93 sec latency 14959.050 ms oleg325-client.virtnet: 1 18266 0.61 MB/sec execute 94 sec latency 15959.822 ms oleg325-client.virtnet: 1 18266 0.61 MB/sec execute 95 sec latency 16960.059 ms oleg325-client.virtnet: 1 18463 0.60 MB/sec execute 96 sec latency 16973.783 ms oleg325-client.virtnet: 1 18599 0.60 MB/sec execute 97 sec latency 38.887 ms oleg325-client.virtnet: 1 18743 0.60 MB/sec execute 98 sec latency 46.263 ms oleg325-client.virtnet: 1 18957 0.60 MB/sec execute 99 sec latency 28.036 ms oleg325-client.virtnet: 1 19243 0.63 MB/sec execute 100 sec latency 30.284 ms oleg325-client.virtnet: 1 19396 0.62 MB/sec execute 101 sec latency 49.056 ms oleg325-client.virtnet: 1 19527 0.62 MB/sec execute 102 sec latency 37.169 ms oleg325-client.virtnet: 1 19660 0.61 MB/sec execute 103 sec latency 61.164 ms oleg325-client.virtnet: 1 19858 0.61 MB/sec execute 104 sec latency 34.796 ms oleg325-client.virtnet: 1 20012 0.60 MB/sec execute 105 sec latency 41.717 ms oleg325-client.virtnet: 1 20202 0.61 MB/sec execute 106 sec latency 183.717 ms oleg325-client.virtnet: 1 20202 0.61 MB/sec execute 107 sec latency 1183.885 ms oleg325-client.virtnet: 1 20202 0.60 MB/sec execute 10Failover mds2 to oleg325-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0001 18:28:55 (1746138535) targets are mounted 18:28:55 (1746138535) facet_failover done 8 sec latency 2184.159 ms oleg325-client.virtnet: 1 20202 0.59 MB/sec execute 109 sec latency 3185.671 ms oleg325-client.virtnet: 1 20202 0.59 MB/sec execute 110 sec latency 4185.950 ms oleg325-client.virtnet: 1 20202 0.58 MB/sec execute 111 sec latency 5186.125 ms oleg325-client.virtnet: 1 20202 0.58 MB/sec execute 112 sec latency 6187.672 ms oleg325-client.virtnet: 1 20202 0.57 MB/sec execute 113 sec latency 7187.855 ms oleg325-client.virtnet: 1 20202 0.57 MB/sec execute 114 sec latency 8188.152 ms oleg325-client.virtnet: 1 20202 0.56 MB/sec execute 115 sec latency 9189.055 ms oleg325-client.virtnet: 1 20202 0.56 MB/sec execute 116 sec latency 10190.288 ms oleg325-client.virtnet: 1 20202 0.55 MB/sec execute 117 sec latency 11191.636 ms oleg325-client.virtnet: 1 20202 0.55 MB/sec execute 118 sec latency 12193.257 ms oleg325-client.virtnet: 1 20202 0.54 MB/sec execute 119 sec latency 13193.436 ms oleg325-client.virtnet: 1 20202 0.54 MB/sec execute 120 sec latency 14193.617 ms oleg325-client.virtnet: 1 20202 0.54 MB/sec execute 121 sec latency 15193.795 ms oleg325-client.virtnet: 1 20202 0.53 MB/sec execute 122 sec latency 16194.013 ms oleg325-client.virtnet: 1 20202 0.53 MB/sec execute 123 sec latency 17194.347 ms oleg325-client.virtnet: 1 20202 0.52 MB/sec execute 124 sec latency 18194.603 ms oleg325-client.virtnet: 1 20202 0.52 MB/sec execute 125 sec latency 19194.805 ms oleg325-client.virtnet: 1 20202 0.51 MB/sec execute 126 sec latency 20195.063 ms oleg325-client.virtnet: 1 20202 0.51 MB/sec execute 127 sec latency 21195.420 ms oleg325-client.virtnet: 1 20202 0.51 MB/sec execute 128 sec latency 22195.582 ms oleg325-client.virtnet: 1 20202 0.50 MB/sec execute 129 sec latency 23196.972 ms oleg325-client.virtnet: 1 20202 0.50 MB/sec execute 130 sec latency 24197.143 ms oleg325-client.virtnet: 1 20202 0.49 MB/sec execute 131 sec latency 25197.438 ms oleg325-client.virtnet: 1 20202 0.49 MB/sec execute 132 sec latency 26198.354 ms oleg325-client.virtnet: 1 20202 0.49 MB/sec execute 133 sec latency 27198.689 ms oleg325-client.virtnet: 1 20202 0.48 MB/sec execute 134 sec latency 28199.013 ms oleg325-client.virtnet: 1 20202 0.48 MB/sec execute 135 sec latency 29199.733 ms oleg325-client.virtnet: 1 20202 0.48 MB/sec execute 136 sec latency 30199.916 ms oleg325-client.virtnet: 1 20202 0.47 MB/sec execute 137 sec latency 31200.271 ms oleg325-client.virtnet: 1 20202 0.47 MB/sec execute 138 sec latency 32200.432 ms oleg325-client.virtnet: 1 20202 0.47 MB/sec execute 139 sec latency 33200.692 ms oleg325-client.virtnet: 1 20202 0.46 MB/sec execute 140 sec latency 34200.854 ms oleg325-client.virtnet: 1 20202 0.46 MB/sec execute 141 sec latency 35201.330 ms oleg325-client.virtnet: 1 20202 0.46 MB/sec execute 142 sec latency 36201.593 ms oleg325-client.virtnet: 1 20202 0.45 MB/sec execute 143 sec latency 37201.764 ms oleg325-client.virtnet: 1 20202 0.45 MB/sec execute 144 sec latency 38202.076 ms oleg325-client.virtnet: 1 20202 0.45 MB/sec execute 145 sec latency 39202.275 ms oleg325-client.virtnet: 1 20202 0.44 MB/sec execute 146 sec latency 40202.456 ms oleg325-client.virtnet: 1 20202 0.44 MB/sec execute 147 sec latency 41202.668 ms oleg325-client.virtnet: 1 20202 0.44 MB/sec execute 148 sec latency 42203.370 ms oleg325-client.virtnet: 1 20202 0.44 MB/sec execute 149 sec latency 43203.567 ms oleg325-client.virtnet: 1 20202 0.43 MB/sec execute 150 sec latency 44203.787 ms oleg325-client.virtnet: 1 20202 0.43 MB/sec execute 151 sec latency 45203.963 ms oleg325-client.virtnet: 1 2oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1285668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13012 3591968 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36744 3569968 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49756 7161936 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg325-server Stopping /mnt/lustre-mds1 (opts:) on oleg325-server 0202 0.43 MB/sec execute 152 sec latency 46204.200 ms oleg325-client.virtnet: 1 20202 0.42 MB/sec execute 153 sec latency 47204.451 ms oleg325-client.virtnet: 1 20202 0.42 MB/sec execute 154 sec latency 48204.785 ms oleg325-client.virtnet: 1 20202 0.42 MB/sec execute 155 sec latency 49205.251 ms oleg325-client.virtnet: 1 20202 0.42 MB/sec execute 156 sec latency 50205.426 ms oleg325-client.virtnet: 1 20202 0.41 MB/sec execute 157 sec latency 51205.589 ms oleg325-client.virtnet: 1 20202 0.41 MB/sec execute 158 sec latency 52206.207 ms oleg325-client.virtnet: 1 20202 0.41 MB/sec execute 159 sec latency 53206.387 ms oleg325-client.virtnet: 1 20202 0.41 MB/sec execute 160 sec latency 54206.570 ms oleg325-client.virtnet: 1 20202 0.40 MB/sec execute 161 sec latency 55206.783 ms oleg325-client.virtnet: 1 20202 0.40 MB/sec execute 162 sec latency 56206.972 ms oleg325-client.virtnet: 1 20202 0.40 MB/sec execute 163 sec latency 57207.142 ms oleg325-client.virtnet: 1 20202 0.40 MB/sec execute 164 sec latency 58209.276 ms oleg325-client.virtnet: 1 20202 0.39 MB/sec execute 165 sec latency 59209.638 ms oleg325-client.virtnet: 1 20202 0.39 MB/sec execute 166 sec latency 60209.924 ms oleg325-client.virtnet: 1 20202 0.39 MB/sec execute 167 sec latency 61210.116 ms oleg325-client.virtnet: 1 20202 0.39 MB/sec execute 168 sec latency 62210.321 ms oleg325-client.virtnet: 1 20202 0.38 MB/sec execute 169 sec latency 63210.691 ms oleg325-client.virtnet: 1 20202 0.38 MB/sec execute 170 sec latency 64210.873 ms oleg325-client.virtnet: 1 20202 0.38 MB/sec execute 171 sec latency 65211.079 ms oleg325-client.virtnet: 1 20202 0.38 MB/sec execute 172 sec latency 66211.245 ms oleg325-client.virtnet: 1 20202 0.37 MB/sec execute 173 sec latency 67211.404 ms oleg325-client.virtnet: 1 20202 0.37 MB/sec execute 174 sec latency 68211.563 ms oleg325-client.virtnet: 1 20202 0.37 MB/sec execute 175 sec latency 69212.391 ms oleg325-client.virtnet: 1 20202 0.37 MB/sec execute 176 sec latency 70212.564 ms oleg325-client.virtnet: 1 20202 0.37 MB/sec execute 177 sec latency 71213.752 ms oleg325-client.virtnet: 1 20202 0.36 MB/sec execute 178 sec latency 72214.946 ms oleg325-client.virtnet: 1 20202 0.36 MB/sec execute 179 sec latency 73215.368 ms oleg325-client.virtnet: 1 20202 0.36 MB/sec execute 180 sec latency 74216.301 ms oleg325-client.virtnet: 1 20202 0.36 MB/sec execute 181 sec latency 75216.574 ms oleg325-client.virtnet: 1 20202 0.36 MB/sec execute 182 sec latency 76216.751 ms oleg325-client.virtnet: 1 20202 0.35 MB/sec execute 183 sec latency 77216.958 ms oleg325-client.virtnet: 1 20202 0.35 MB/sec execute 184 sec latency 78217.148 ms oleg325-client.virtnet: 1 20202 0.35 MB/sec execute 185 sec latency 79219.527 ms oleg325-client.virtnet: 1 20238 0.35 MB/sec execute 186 sec latency 80004.903 ms oleg325-client.virtnet: 1 20709 0.38 MB/sec execute 187 sec latency 39.670 ms oleg325-client.virtnet: 1 21271 0.40 MB/sec execute 188 sec latency 29.953 ms oleg325-client.virtnet: 1 21542 0.40 MB/sec execute 189 sec latency 29.881 ms oleg325-client.virtnet: 1 21713 0.40 MB/sec execute 190 sec latency 28.120 ms oleg325-client.virtnet: 1 21864 0.40 MB/sec execute 191 sec latency 46.793 ms oleg325-client.virtnet: 1 22105 0.40 MB/sec execute 192 sec latency 27.522 ms oleg325-client.virtnet: 1 22291 0.40 MB/sec execute 193 sec latency 36.771 ms oleg325-client.virtnet: 1 22533 0.41 MB/sec execute 194 sec latency 24.691 ms oleg325-client.virtnet: 1 22805 0.42 MB/sec execute 195 sec latency 26.853 ms oleg325-client.virtne18:30:09 (1746138609) shut down facet: mds1 facet_host: oleg325-server facet_failover_host: oleg325-server Failover mds1 to oleg325-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0000 18:30:25 (1746138625) targets are mounted 18:30:25 (1746138625) facet_failover done oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1285332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1285652 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13128 3590840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36520 3569912 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7160752 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg325-server Stopping /mnt/lustre-mds2 (opts:) on oleg325-server 18:30:40 (1746138640) shut down facet: mds2 facet_host: oleg325-server facet_failover_host: oleg325-server Failover mds2 to oleg325-server mount facets: mds2 t: 1 23014 0.42 MB/sec execute 196 sec latency 31.884 ms oleg325-client.virtnet: 1 23206 0.42 MB/sec execute 197 sec latency 31.962 ms oleg325-client.virtnet: 1 23491 0.42 MB/sec execute 198 sec latency 26.549 ms oleg325-client.virtnet: 1 23726 0.42 MB/sec execute 199 sec latency 37.234 ms oleg325-client.virtnet: 1 24044 0.44 MB/sec execute 200 sec latency 27.734 ms oleg325-client.virtnet: 1 24522 0.45 MB/sec execute 201 sec latency 34.919 ms oleg325-client.virtnet: 1 24935 0.47 MB/sec execute 202 sec latency 29.487 ms oleg325-client.virtnet: 1 25155 0.47 MB/sec execute 203 sec latency 23.295 ms oleg325-client.virtnet: 1 25329 0.47 MB/sec execute 204 sec latency 23.857 ms oleg325-client.virtnet: 1 25561 0.47 MB/sec execute 205 sec latency 30.957 ms oleg325-client.virtnet: 1 25749 0.47 MB/sec execute 206 sec latency 33.398 ms oleg325-client.virtnet: 1 25913 0.47 MB/sec execute 207 sec latency 51.479 ms oleg325-client.virtnet: 1 26144 0.47 MB/sec execute 208 sec latency 25.789 ms oleg325-client.virtnet: 1 26391 0.48 MB/sec execute 209 sec latency 41.338 ms oleg325-client.virtnet: 1 26537 0.48 MB/sec execute 210 sec latency 31.324 ms oleg325-client.virtnet: 1 26676 0.48 MB/sec execute 211 sec latency 48.392 ms oleg325-client.virtnet: 1 26846 0.48 MB/sec execute 212 sec latency 40.542 ms oleg325-client.virtnet: 1 27016 0.48 MB/sec execute 213 sec latency 38.649 ms oleg325-client.virtnet: 1 27253 0.48 MB/sec execute 214 sec latency 30.286 ms oleg325-client.virtnet: 1 27456 0.48 MB/sec execute 215 sec latency 37.564 ms oleg325-client.virtnet: 1 27813 0.50 MB/sec execute 216 sec latency 40.040 ms oleg325-client.virtnet: 1 28213 0.51 MB/sec execute 217 sec latency 37.235 ms oleg325-client.virtnet: 1 28437 0.52 MB/sec execute 218 sec latency 57.330 ms oleg325-client.virtnet: 1 28557 0.52 MB/sec execute 219 sec latency 64.809 ms oleg325-client.virtnet: 1 28723 0.52 MB/sec execute 220 sec latency 35.713 ms oleg325-client.virtnet: 1 28873 0.52 MB/sec execute 221 sec latency 29.241 ms oleg325-client.virtnet: 1 29036 0.52 MB/sec execute 222 sec latency 48.439 ms oleg325-client.virtnet: 1 29186 0.52 MB/sec execute 223 sec latency 47.941 ms oleg325-client.virtnet: 1 29219 0.52 MB/sec execute 224 sec latency 741.129 ms oleg325-client.virtnet: 1 29366 0.51 MB/sec execute 225 sec latency 66.784 ms oleg325-client.virtnet: 1 29558 0.52 MB/sec execute 226 sec latency 36.116 ms oleg325-client.virtnet: 1 29686 0.52 MB/sec execute 227 sec latency 35.614 ms oleg325-client.virtnet: 1 29709 0.51 MB/sec execute 228 sec latency 871.079 ms oleg325-client.virtnet: 1 29709 0.51 MB/sec execute 229 sec latency 1871.250 ms oleg325-client.virtnet: 1 29709 0.51 MB/sec execute 230 sec latency 2871.419 ms oleg325-client.virtnet: 1 29709 0.51 MB/sec execute 231 sec latency 3871.585 ms oleg325-client.virtnet: 1 29709 0.51 MB/sec execute 232 sec latency 4872.111 ms oleg325-client.virtnet: 1 29709 0.50 MB/sec execute 233 sec latency 5872.272 ms oleg325-client.virtnet: 1 29709 0.50 MB/sec execute 234 sec latency 6872.842 ms oleg325-client.virtnet: 1 29709 0.50 MB/sec execute 235 sec latency 7873.284 ms oleg325-client.virtnet: 1 29709 0.50 MB/sec execute 236 sec latency 8877.450 ms oleg325-client.virtnet: 1 29709 0.49 MB/sec execute 237 sec latency 9877.639 ms oleg325-client.virtnet: 1 29709 0.49 MB/sec execute 238 sec latency 10877.863 ms oleg325-client.virtnet: 1 29709 0.49 MB/sec execute 239 sec latency 11878.046 ms oleg325-client.virtnet: 1 29709 0.49 MB/sec execute 240 sec latency 12880.Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0001 18:30:57 (1746138657) targets are mounted 18:30:57 (1746138657) facet_failover done oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 394 ms oleg325-client.virtnet: 1 29709 0.49 MB/sec execute 241 sec latency 13880.555 ms oleg325-client.virtnet: 1 29709 0.48 MB/sec execute 242 sec latency 14880.732 ms oleg325-client.virtnet: 1 29709 0.48 MB/sec execute 243 sec latency 15880.909 ms oleg325-client.virtnet: 1 29709 0.48 MB/sec execute 244 sec latency 16881.508 ms oleg325-client.virtnet: 1 29709 0.48 MB/sec execute 245 sec latency 17882.580 ms oleg325-client.virtnet: 1 29709 0.48 MB/sec execute 246 sec latency 18882.796 ms oleg325-client.virtnet: 1 29709 0.47 MB/sec execute 247 sec latency 19899.681 ms oleg325-client.virtnet: 1 29709 0.47 MB/sec execute 248 sec latency 20900.054 ms oleg325-client.virtnet: 1 29912 0.48 MB/sec execute 249 sec latency 21066.935 ms oleg325-client.virtnet: 1 30077 0.48 MB/sec execute 250 sec latency 34.853 ms oleg325-client.virtnet: 1 30237 0.48 MB/sec execute 251 sec latency 33.378 ms oleg325-client.virtnet: 1 30434 0.48 MB/sec execute 252 sec latency 30.145 ms oleg325-client.virtnet: 1 30663 0.48 MB/sec execute 253 sec latency 27.553 ms oleg325-client.virtnet: 1 30873 0.48 MB/sec execute 254 sec latency 29.619 ms oleg325-client.virtnet: 1 31322 0.50 MB/sec execute 255 sec latency 24.675 ms oleg325-client.virtnet: 1 31781 0.51 MB/sec execute 256 sec latency 28.355 ms oleg325-client.virtnet: 1 32113 0.52 MB/sec execute 257 sec latency 22.245 ms oleg325-client.virtnet: 1 32309 0.52 MB/sec execute 258 sec latency 50.235 ms oleg325-client.virtnet: 1 32470 0.52 MB/sec execute 259 sec latency 24.721 ms oleg325-client.virtnet: 1 32697 0.52 MB/sec execute 260 sec latency 29.192 ms oleg325-client.virtnet: 1 32887 0.52 MB/sec execute 261 sec latency 27.470 ms oleg325-client.virtnet: 1 33098 0.52 MB/sec execute 262 sec latency 31.677 ms oleg325-client.virtnet: 1 33367 0.53 MB/sec execute 263 sec latency 24.664 ms oleg325-client.virtnet: 1 33550 0.53 MB/sec execute 264 sec latency 31.362 ms oleg325-client.virtnet: 1 33719 0.53 MB/sec execute 265 sec latency 32.797 ms oleg325-client.virtnet: 1 33896 0.52 MB/sec execute 266 sec latency 36.005 ms oleg325-client.virtnet: 1 34119 0.52 MB/sec execute 267 sec latency 26.062 ms oleg325-client.virtnet: 1 34352 0.53 MB/sec execute 268 sec latency 30.539 ms oleg325-client.virtnet: 1 34568 0.53 MB/sec execute 269 sec latency 30.235 ms oleg325-client.virtnet: 1 34974 0.54 MB/sec execute 270 sec latency 32.698 ms oleg325-client.virtnet: 1 35463 0.56 MB/sec execute 271 sec latency 35.601 ms oleg325-client.virtnet: 1 35647 0.56 MB/sec execute 272 sec latency 30.868 ms oleg325-client.virtnet: 1 35807 0.56 MB/sec execute 273 sec latency 32.529 ms oleg325-client.virtnet: 1 35957 0.56 MB/sec execute 274 sec latency 30.000 ms oleg325-client.virtnet: 1 36169 0.56 MB/sec execute 275 sec latency 27.039 ms oleg325-client.virtnet: 1 36365 0.56 MB/sec execute 276 sec latency 29.656 ms oleg325-client.virtnet: 1 36562 0.56 MB/sec execute 277 sec latency 29.164 ms oleg325-client.virtnet: 1 36759 0.56 MB/sec execute 278 sec latency 23.441 ms oleg325-client.virtnet: 1 37028 0.57 MB/sec execute 279 sec latency 44.201 ms oleg325-client.virtnet: 1 37206 0.57 MB/sec execute 280 sec latency 28.658 ms oleg325-client.virtnet: 1 37363 0.56 MB/sec execute 281 sec latency 34.155 ms oleg325-client.virtnet: 1 37594 0.56 MB/sec execute 282 sec latency 31.510 ms oleg325-client.virtnet: 1 37867 0.57 MB/sec execute 283 sec latency 31.586 ms oleg325-client.virtnet: 1 38101 0.57 MB/sec execute 284 sec latency 30.699 ms oleg325-client.virtnet: 1 38516 0.58 MB/sec execute 285 sec latency 24.985 ms oleg325-client.virtnet: 1 39016 0.60 MB/sec execute 286 sec latency 29.000 ms oleg325-client.virtnet: 1 39203 0.60 MB/sec execute 287 sec latency 27.005 ms oleg325-client.virtnet: 1 39395 0.60 MB/sec execute 288 sec latency 27.064 ms oleg325-client.virtnet: 1 39540 0.60 MB/sec execute 289 sec latency 31.257 ms oleg325-client.virtnet: 1 39756 0.60 MB/sec execute 290 sec latency 37.669 ms oleg325-client.virtnet: 1 39882 0.59 MB/sec execute 291 sec latency 45.369 ms oleg325-client.virtnet: 1 40029 0.59 MB/sec execute 292 sec latency 49.185 ms oleg325-client.virtnet: 1 40223 0.60 MB/sec execute 293 sec latency 37.340 ms oleg325-client.virtnet: 1 40473 0.60 MB/sec execute 294 sec latency 27.057 ms oleg325-client.virtnet: 1 40640 0.60 MB/sec execute 295 sec latency 31.097 ms oleg325-client.virtnet: 1 40797 0.60 MB/sec execute 296 sec latency 54.665 ms oleg325-client.virtnet: 1 41002 0.60 MB/sec execute 297 sec latency 27.454 ms oleg325-client.virtnet: 1 41209 0.60 MB/sec execute 298 sec latency 29.500 ms oleg325-client.virtnet: 1 41426 0.60 MB/sec execute 299 sec latency 63.991 ms oleg325-client.virtnet: 1 cleanup 300 sec oleg325-client.virtnet: 0 cleanup 301 sec oleg325-client.virtnet: oleg325-client.virtnet: Operation Count AvgLat MaxLat oleg325-client.virtnet: ---------------------------------------- oleg325-client.virtnet: NTCreateX 6012 22.348 23281.955 oleg325-client.virtnet: Close 4409 3.121 36.003 oleg325-client.virtnet: Rename 249 20.985 54.638 oleg325-client.virtnet: Unlink 1216 4.884 15.651 oleg325-client.virtnet: Qpathinfo 5446 6.666 16973.762 oleg325-client.virtnet: Qfileinfo 938 0.240 7.667 oleg325-client.virtnet: Qfsinfo 969 0.489 19.568 oleg325-client.virtnet: Sfileinfo 499 174.817 80004.889 oleg325-client.virtnet: Find 2073 0.788 36.902 oleg325-client.virtnet: WriteX 2932 2.188 41.326 oleg325-client.virtnet: ReadX 9108 0.029 4.309 oleg325-client.virtnet: LockX 18 2.376 4.428 oleg325-client.virtnet: UnlockX 18 2.751 6.868 oleg325-client.virtnet: Flush 421 18.458 741.114 oleg325-client.virtnet: oleg325-client.virtnet: Throughput 0.599636 MB/sec 1 clients 1 procs max_latency=80004.903 ms oleg325-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg325-client.virtnet at Thu May 1 18:31:53 EDT 2025 with return code 0 oleg325-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg325-client.virtnet oleg325-client.virtnet: /mnt/lustre/d70b.replay-single/oleg325-client.virtnet /mnt/lustre/d70b.replay-single/oleg325-client.virtnet oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg325-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg325-client.virtnet: removed directory: 'clients/client0' oleg325-client.virtnet: removed directory: 'clients' oleg325-client.virtnet: removed 'client.txt' oleg325-client.virtnet: /mnt/lustre/d70b.replay-single/oleg325-client.virtnet oleg325-client.virtnet: dbench successfully finished