== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 16:09:20 (1746130160) Starting client oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre Started clients oleg316-client.virtnet: 192.168.203.116@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg316-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14205 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2152 1285536 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26160 3556140 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7161596 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg316-server Stopping /mnt/lustre-mds1 (opts:) on oleg316-server 16:09:31 (1746130171) shut down facet: mds1 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds1 to oleg316-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 16:09:49 (1746130189) targets are mounted 16:09:49 (1746130189) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg316-client.virtnet: looking for dbench program oleg316-client.virtnet: /usr/bin/dbench oleg316-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg316-client.virtnet oleg316-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg316-client.virtnet' oleg316-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg316-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg316-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg316-client.virtnet at Thu May 1 16:09:22 EDT 2025 oleg316-client.virtnet: waiting for dbench pid 14230 oleg316-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg316-client.virtnet: oleg316-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg316-client.virtnet: failed to create barrier semaphore oleg316-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg316-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg316-client.virtnet: releasing clients oleg316-client.virtnet: 1 137 6.02 MB/sec warmup 1 sec latency 99.685 ms oleg316-client.virtnet: 1 279 5.22 MB/sec warmup 2 sec latency 45.283 ms oleg316-client.virtnet: 1 421 4.95 MB/sec warmup 3 sec latency 48.860 ms oleg316-client.virtnet: 1 640 5.26 MB/sec warmup 4 sec latency 81.372 ms oleg316-client.virtnet: 1 773 4.27 MB/sec warmup 5 sec latency 37.871 ms oleg316-client.virtnet: 1 923 3.63 MB/sec warmup 6 sec latency 34.615 ms oleg316-client.virtnet: 1 1095 3.27 MB/sec warmup 7 sec latency 33.953 ms oleg316-client.virtnet: 1 1244 2.88 MB/sec warmup 8 sec latency 45.083 ms oleg316-client.virtnet: 1 1468 2.89 MB/sec warmup 9 sec latency 54.130 ms oleg316-client.virtnet: 1 1627 2.61 MB/sec warmup 10 sec latency 29.260 ms oleg316-client.virtnet: 1 1832 2.39 MB/sec warmup 11 sec latency 42.824 ms oleg316-client.virtnet: 1 2012 2.21 MB/sec warmup 12 sec latency 40.479 ms oleg316-client.virtnet: 1 2196 2.06 MB/sec warmup 13 sec latency 33.736 ms oleg316-client.virtnet: 1 2395 2.00 MB/sec warmup 14 sec latency 68.196 ms oleg316-client.virtnet: 1 2582 1.93 MB/sec warmup 15 sec latency 38.985 ms oleg316-client.virtnet: 1 2977 2.09 MB/sec warmup 16 sec latency 35.959 ms oleg316-client.virtnet: 1 3352 2.08 MB/sec warmup 17 sec latency 34.324 ms oleg316-client.virtnet: 1 3644 2.14 MB/sec warmup 18 sec latency 36.800 ms oleg316-client.virtnet: 1 3808 2.09 MB/sec warmup 19 sec latency 38.077 ms oleg316-client.virtnet: 1 3907 2.00 MB/sec warmup 20 sec latency 41.774 ms oleg316-client.virtnet: 1 3995 1.91 MB/sec warmup 21 sec latency 49.750 ms oleg316-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 108.459 ms oleg316-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 1115.753 ms oleg316-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 2116.869 ms oleg316-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 3117.049 ms oleg316-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 4118.274 ms oleg316-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 5118.527 ms oleg316-client.virtnet: 1 4083 1.43 MB/sec warmup 28 sec latency 6118.895 ms oleg316-client.virtnet: 1 4083 1.38 MB/sec warmup 29 sec latency 7119.061 ms oleg316-client.virtnet: 1 4212 1.34 MB/sec warmup 30 sec latency 7397.650 ms oleg316-client.virtnet: 1 4299 1.30 MB/sec warmup 31 sec latency 68.457 ms oleg316-client.virtnet: 1 4386 1.27 MB/sec warmup 32 sec latency 51.975 ms oleg316-client.virtnet: 1 4505 1.24 MB/sec warmup 33 sec latency 51.786 ms oleg316-client.virtnet:UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2172 1285516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 27444 3567640 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5572 3593996 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 33016 7161636 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg316-server Stopping /mnt/lustre-mds2 (opts:) on oleg316-server 16:10:05 (1746130205) shut down facet: mds2 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds2 to oleg316-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 16:10:22 (1746130222) targets are mounted 16:10:22 (1746130222) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37440 3566852 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12216 3594496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49656 7161348 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg316-server Stopping /mnt/lustre-mds1 (opts:) on oleg316-server 16:10:38 (1746130238) shut down facet: mds1 facet_host: oleg316-server facet_failover_host: oleg316-server 1 4667 1.24 MB/sec warmup 34 sec latency 48.342 ms oleg316-client.virtnet: 1 4810 1.20 MB/sec warmup 35 sec latency 42.734 ms oleg316-client.virtnet: 1 5034 1.25 MB/sec warmup 36 sec latency 35.288 ms oleg316-client.virtnet: 1 5154 1.22 MB/sec warmup 37 sec latency 53.088 ms oleg316-client.virtnet: 1 5218 1.19 MB/sec warmup 38 sec latency 429.888 ms oleg316-client.virtnet: 1 5350 1.16 MB/sec warmup 39 sec latency 38.309 ms oleg316-client.virtnet: 1 5442 1.14 MB/sec warmup 40 sec latency 44.817 ms oleg316-client.virtnet: 1 5618 1.11 MB/sec warmup 41 sec latency 42.457 ms oleg316-client.virtnet: 1 5633 1.09 MB/sec warmup 42 sec latency 884.365 ms oleg316-client.virtnet: 1 5633 1.06 MB/sec warmup 43 sec latency 1884.756 ms oleg316-client.virtnet: 1 5633 1.04 MB/sec warmup 44 sec latency 2885.153 ms oleg316-client.virtnet: 1 5633 1.02 MB/sec warmup 45 sec latency 3885.413 ms oleg316-client.virtnet: 1 5633 0.99 MB/sec warmup 46 sec latency 4885.650 ms oleg316-client.virtnet: 1 5633 0.97 MB/sec warmup 47 sec latency 5886.531 ms oleg316-client.virtnet: 1 5633 0.95 MB/sec warmup 48 sec latency 6887.017 ms oleg316-client.virtnet: 1 5633 0.93 MB/sec warmup 49 sec latency 7887.290 ms oleg316-client.virtnet: 1 5633 0.91 MB/sec warmup 50 sec latency 8887.545 ms oleg316-client.virtnet: 1 5633 0.90 MB/sec warmup 51 sec latency 9887.982 ms oleg316-client.virtnet: 1 5633 0.88 MB/sec warmup 52 sec latency 10888.195 ms oleg316-client.virtnet: 1 5633 0.86 MB/sec warmup 53 sec latency 11888.564 ms oleg316-client.virtnet: 1 5633 0.85 MB/sec warmup 54 sec latency 12889.145 ms oleg316-client.virtnet: 1 5633 0.83 MB/sec warmup 55 sec latency 13889.468 ms oleg316-client.virtnet: 1 5633 0.82 MB/sec warmup 56 sec latency 14889.616 ms oleg316-client.virtnet: 1 5633 0.80 MB/sec warmup 57 sec latency 15890.280 ms oleg316-client.virtnet: 1 5633 0.79 MB/sec warmup 58 sec latency 16890.438 ms oleg316-client.virtnet: 1 5633 0.77 MB/sec warmup 59 sec latency 17890.614 ms oleg316-client.virtnet: 1 5633 0.00 MB/sec execute 1 sec latency 19892.257 ms oleg316-client.virtnet: 1 5657 0.01 MB/sec execute 2 sec latency 20710.953 ms oleg316-client.virtnet: 1 5820 0.05 MB/sec execute 3 sec latency 35.823 ms oleg316-client.virtnet: 1 5968 0.35 MB/sec execute 4 sec latency 45.183 ms oleg316-client.virtnet: 1 6140 0.47 MB/sec execute 5 sec latency 67.061 ms oleg316-client.virtnet: 1 6480 1.14 MB/sec execute 6 sec latency 37.403 ms oleg316-client.virtnet: 1 6801 1.16 MB/sec execute 7 sec latency 42.136 ms oleg316-client.virtnet: 1 7099 1.47 MB/sec execute 8 sec latency 38.070 ms oleg316-client.virtnet: 1 7240 1.35 MB/sec execute 9 sec latency 32.342 ms oleg316-client.virtnet: 1 7373 1.31 MB/sec execute 10 sec latency 50.642 ms oleg316-client.virtnet: 1 7488 1.22 MB/sec execute 11 sec latency 43.005 ms oleg316-client.virtnet: 1 7603 1.12 MB/sec execute 12 sec latency 43.345 ms oleg316-client.virtnet: 1 7681 1.03 MB/sec execute 13 sec latency 263.859 ms oleg316-client.virtnet: 1 7823 0.98 MB/sec execute 14 sec latency 49.406 ms oleg316-client.virtnet: 1 7921 0.93 MB/sec execute 15 sec latency 56.149 ms oleg316-client.virtnet: 1 8035 0.89 MB/sec execute 16 sec latency 61.215 ms oleg316-client.virtnet: 1 8195 0.91 MB/sec execute 17 sec latency 61.868 ms oleg316-client.virtnet: 1 8339 0.86 MB/sec execute 18 sec latency 39.678 ms oleg316-client.virtnet: 1 8560 0.97 MB/sec execute 19 sec latency 54.409 msFailover mds1 to oleg316-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 16:10:58 (1746130258) targets are mounted 16:10:58 (1746130258) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1285580 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37644 3568672 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11900 3594216 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49544 7162888 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg316-server Stopping /mnt/lustre-mds2 (opts:) on oleg316-server 16:11:14 (1746130274) shut down facet: mds2 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds2 to oleg316-server mount facets: mds2 oleg316-client.virtnet: 1 8678 0.93 MB/sec execute 20 sec latency 40.434 ms oleg316-client.virtnet: 1 8797 0.89 MB/sec execute 21 sec latency 44.533 ms oleg316-client.virtnet: 1 8913 0.86 MB/sec execute 22 sec latency 42.096 ms oleg316-client.virtnet: 1 9074 0.82 MB/sec execute 23 sec latency 37.282 ms oleg316-client.virtnet: 1 9214 0.80 MB/sec execute 24 sec latency 53.636 ms oleg316-client.virtnet: 1 9442 0.81 MB/sec execute 25 sec latency 41.032 ms oleg316-client.virtnet: 1 9551 0.79 MB/sec execute 26 sec latency 57.059 ms oleg316-client.virtnet: 1 9974 0.91 MB/sec execute 27 sec latency 38.580 ms oleg316-client.virtnet: 1 10288 0.95 MB/sec execute 28 sec latency 56.465 ms oleg316-client.virtnet: 1 10742 1.07 MB/sec execute 29 sec latency 24.276 ms oleg316-client.virtnet: 1 10976 1.07 MB/sec execute 30 sec latency 46.769 ms oleg316-client.virtnet: 1 11092 1.05 MB/sec execute 31 sec latency 79.748 ms oleg316-client.virtnet: 1 11166 1.02 MB/sec execute 32 sec latency 51.377 ms oleg316-client.virtnet: 1 11372 0.99 MB/sec execute 33 sec latency 38.054 ms oleg316-client.virtnet: 1 11510 0.98 MB/sec execute 34 sec latency 40.746 ms oleg316-client.virtnet: 1 11627 0.95 MB/sec execute 35 sec latency 46.963 ms oleg316-client.virtnet: 1 11812 0.96 MB/sec execute 36 sec latency 36.318 ms oleg316-client.virtnet: 1 12102 1.02 MB/sec execute 37 sec latency 23.985 ms oleg316-client.virtnet: 1 12268 0.99 MB/sec execute 38 sec latency 26.730 ms oleg316-client.virtnet: 1 12386 0.97 MB/sec execute 39 sec latency 67.663 ms oleg316-client.virtnet: 1 12499 0.95 MB/sec execute 40 sec latency 61.008 ms oleg316-client.virtnet: 1 12660 0.93 MB/sec execute 41 sec latency 40.308 ms oleg316-client.virtnet: 1 12805 0.91 MB/sec execute 42 sec latency 42.452 ms oleg316-client.virtnet: 1 12982 0.92 MB/sec execute 43 sec latency 45.615 ms oleg316-client.virtnet: 1 13083 0.90 MB/sec execute 44 sec latency 42.272 ms oleg316-client.virtnet: 1 13218 0.90 MB/sec execute 45 sec latency 72.278 ms oleg316-client.virtnet: 1 13426 0.95 MB/sec execute 46 sec latency 52.427 ms oleg316-client.virtnet: 1 13572 0.96 MB/sec execute 47 sec latency 571.348 ms oleg316-client.virtnet: 1 13833 0.95 MB/sec execute 48 sec latency 652.582 ms oleg316-client.virtnet: 1 14059 0.97 MB/sec execute 49 sec latency 67.735 ms oleg316-client.virtnet: 1 14227 1.00 MB/sec execute 50 sec latency 53.156 ms oleg316-client.virtnet: 1 14318 0.99 MB/sec execute 51 sec latency 434.791 ms oleg316-client.virtnet: 1 14318 0.97 MB/sec execute 52 sec latency 1435.507 ms oleg316-client.virtnet: 1 14318 0.95 MB/sec execute 53 sec latency 2435.738 ms oleg316-client.virtnet: 1 14318 0.93 MB/sec execute 54 sec latency 3437.488 ms oleg316-client.virtnet: 1 14318 0.91 MB/sec execute 55 sec latency 4438.450 ms oleg316-client.virtnet: 1 14318 0.90 MB/sec execute 56 sec latency 5440.268 ms oleg316-client.virtnet: 1 14318 0.88 MB/sec execute 57 sec latency 6440.752 ms oleg316-client.virtnet: 1 14318 0.87 MB/sec execute 58 sec latency 7441.443 ms oleg316-client.virtnet: 1 14318 0.85 MB/sec execute 59 sec latency 8441.929 ms oleg316-client.virtnet: 1 14318 0.84 MB/sec execute 60 sec latency 9442.482 ms oleg316-client.virtnet: 1 14318 0.82 MB/sec execute 61 sec latency 10442.700 ms oleg316-client.virtnet: 1 14318 0.81 MB/sec execute 62 sec latency 11444.045 ms oleg316-client.virtnet: 1 14318 0.80 MB/sec execute 63 sec latency 12444.299 ms oleg316-client.virtnet: 1 14318 0.79 MB/sec execuStarting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 16:11:31 (1746130291) targets are mounted 16:11:31 (1746130291) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38156 3566248 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12080 3594028 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50236 7160276 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg316-server Stopping /mnt/lustre-mds1 (opts:) on oleg316-server 16:11:46 (1746130306) shut down facet: mds1 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds1 to oleg316-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 16:12:03 (1746130323) targets are mounted 16:12:03 (1746130323) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec te 64 sec latency 13445.062 ms oleg316-client.virtnet: 1 14318 0.77 MB/sec execute 65 sec latency 14445.238 ms oleg316-client.virtnet: 1 14318 0.76 MB/sec execute 66 sec latency 15445.789 ms oleg316-client.virtnet: 1 14318 0.75 MB/sec execute 67 sec latency 16447.519 ms oleg316-client.virtnet: 1 14318 0.74 MB/sec execute 68 sec latency 17448.389 ms oleg316-client.virtnet: 1 14318 0.73 MB/sec execute 69 sec latency 18448.613 ms oleg316-client.virtnet: 1 14318 0.72 MB/sec execute 70 sec latency 19448.820 ms oleg316-client.virtnet: 1 14318 0.71 MB/sec execute 71 sec latency 20449.725 ms oleg316-client.virtnet: 1 14369 0.70 MB/sec execute 72 sec latency 20948.446 ms oleg316-client.virtnet: 1 14502 0.70 MB/sec execute 73 sec latency 46.477 ms oleg316-client.virtnet: 1 14598 0.70 MB/sec execute 74 sec latency 61.714 ms oleg316-client.virtnet: 1 14694 0.69 MB/sec execute 75 sec latency 44.313 ms oleg316-client.virtnet: 1 14846 0.68 MB/sec execute 76 sec latency 32.091 ms oleg316-client.virtnet: 1 14954 0.68 MB/sec execute 77 sec latency 46.829 ms oleg316-client.virtnet: 1 15098 0.67 MB/sec execute 78 sec latency 32.645 ms oleg316-client.virtnet: 1 15200 0.67 MB/sec execute 79 sec latency 35.000 ms oleg316-client.virtnet: 1 15374 0.67 MB/sec execute 80 sec latency 39.401 ms oleg316-client.virtnet: 1 15603 0.70 MB/sec execute 81 sec latency 32.220 ms oleg316-client.virtnet: 1 15701 0.69 MB/sec execute 82 sec latency 77.045 ms oleg316-client.virtnet: 1 15819 0.69 MB/sec execute 83 sec latency 64.713 ms oleg316-client.virtnet: 1 15976 0.68 MB/sec execute 84 sec latency 42.378 ms oleg316-client.virtnet: 1 16127 0.67 MB/sec execute 85 sec latency 37.330 ms oleg316-client.virtnet: 1 16267 0.67 MB/sec execute 86 sec latency 46.156 ms oleg316-client.virtnet: 1 16470 0.67 MB/sec execute 87 sec latency 32.195 ms oleg316-client.virtnet: 1 16649 0.67 MB/sec execute 88 sec latency 46.645 ms oleg316-client.virtnet: 1 16912 0.70 MB/sec execute 89 sec latency 38.919 ms oleg316-client.virtnet: 1 17181 0.71 MB/sec execute 90 sec latency 41.763 ms oleg316-client.virtnet: 1 17559 0.73 MB/sec execute 91 sec latency 35.178 ms oleg316-client.virtnet: 1 17812 0.75 MB/sec execute 92 sec latency 38.086 ms oleg316-client.virtnet: 1 17996 0.76 MB/sec execute 93 sec latency 29.714 ms oleg316-client.virtnet: 1 18127 0.75 MB/sec execute 94 sec latency 40.492 ms oleg316-client.virtnet: 1 18251 0.74 MB/sec execute 95 sec latency 38.124 ms oleg316-client.virtnet: 1 18266 0.74 MB/sec execute 96 sec latency 848.688 ms oleg316-client.virtnet: 1 18266 0.73 MB/sec execute 97 sec latency 1848.904 ms oleg316-client.virtnet: 1 18266 0.72 MB/sec execute 98 sec latency 2849.117 ms oleg316-client.virtnet: 1 18266 0.71 MB/sec execute 99 sec latency 3849.432 ms oleg316-client.virtnet: 1 18266 0.71 MB/sec execute 100 sec latency 4849.636 ms oleg316-client.virtnet: 1 18266 0.70 MB/sec execute 101 sec latency 5850.533 ms oleg316-client.virtnet: 1 18266 0.69 MB/sec execute 102 sec latency 6850.697 ms oleg316-client.virtnet: 1 18266 0.69 MB/sec execute 103 sec latency 7851.406 ms oleg316-client.virtnet: 1 18425 0.68 MB/sec execute 104 sec latency 7877.789 ms oleg316-client.virtnet: 1 18520 0.68 MB/sec execute 105 sec latency 63.118 ms oleg316-client.virtnet: 1 18631 0.67 MB/sec execute 106 sec latency 60.619 ms oleg316-client.virtnet: 1 18748 0.67 MB/sec execute 107 sec latency 41.193 ms oleg316-client.virtnet: 1 18905 0.67 MB/sec execute 108 sec latency 50.940 ms oleg316-client.virUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37588 3569376 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11900 3595120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7164496 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg316-server Stopping /mnt/lustre-mds2 (opts:) on oleg316-server 16:12:20 (1746130340) shut down facet: mds2 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds2 to oleg316-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 16:12:36 (1746130356) targets are mounted 16:12:36 (1746130356) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37732 3568728 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12024 3594196 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49756 7162924 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg316-server Stopping /mnt/lustre-mds1 (opts:) on oleg316-server 16:12:52 (1746130372) shut down facet: mds1 facet_host: oleg316-server facet_failover_host: oleg316-server tnet: 1 19020 0.67 MB/sec execute 109 sec latency 53.533 ms oleg316-client.virtnet: 1 19198 0.69 MB/sec execute 110 sec latency 55.489 ms oleg316-client.virtnet: 1 19304 0.68 MB/sec execute 111 sec latency 50.713 ms oleg316-client.virtnet: 1 19382 0.68 MB/sec execute 112 sec latency 118.310 ms oleg316-client.virtnet: 1 19456 0.67 MB/sec execute 113 sec latency 413.821 ms oleg316-client.virtnet: 1 19554 0.67 MB/sec execute 114 sec latency 41.978 ms oleg316-client.virtnet: 1 19694 0.66 MB/sec execute 115 sec latency 45.637 ms oleg316-client.virtnet: 1 19771 0.66 MB/sec execute 116 sec latency 240.322 ms oleg316-client.virtnet: 1 19771 0.65 MB/sec execute 117 sec latency 1241.363 ms oleg316-client.virtnet: 1 19771 0.65 MB/sec execute 118 sec latency 2242.202 ms oleg316-client.virtnet: 1 19771 0.64 MB/sec execute 119 sec latency 3242.446 ms oleg316-client.virtnet: 1 19771 0.64 MB/sec execute 120 sec latency 4243.124 ms oleg316-client.virtnet: 1 19771 0.63 MB/sec execute 121 sec latency 5244.840 ms oleg316-client.virtnet: 1 19771 0.63 MB/sec execute 122 sec latency 6245.077 ms oleg316-client.virtnet: 1 19771 0.62 MB/sec execute 123 sec latency 7245.823 ms oleg316-client.virtnet: 1 19771 0.62 MB/sec execute 124 sec latency 8249.766 ms oleg316-client.virtnet: 1 19771 0.61 MB/sec execute 125 sec latency 9250.030 ms oleg316-client.virtnet: 1 19771 0.61 MB/sec execute 126 sec latency 10252.340 ms oleg316-client.virtnet: 1 19771 0.60 MB/sec execute 127 sec latency 11252.870 ms oleg316-client.virtnet: 1 19771 0.60 MB/sec execute 128 sec latency 12253.608 ms oleg316-client.virtnet: 1 19771 0.59 MB/sec execute 129 sec latency 13255.540 ms oleg316-client.virtnet: 1 19771 0.59 MB/sec execute 130 sec latency 14255.828 ms oleg316-client.virtnet: 1 19771 0.58 MB/sec execute 131 sec latency 15256.099 ms oleg316-client.virtnet: 1 19771 0.58 MB/sec execute 132 sec latency 16257.402 ms oleg316-client.virtnet: 1 19771 0.57 MB/sec execute 133 sec latency 17257.723 ms oleg316-client.virtnet: 1 19771 0.57 MB/sec execute 134 sec latency 18258.088 ms oleg316-client.virtnet: 1 19771 0.57 MB/sec execute 135 sec latency 19258.285 ms oleg316-client.virtnet: 1 19771 0.56 MB/sec execute 136 sec latency 20259.813 ms oleg316-client.virtnet: 1 19878 0.56 MB/sec execute 137 sec latency 20780.869 ms oleg316-client.virtnet: 1 19988 0.56 MB/sec execute 138 sec latency 37.814 ms oleg316-client.virtnet: 1 20163 0.56 MB/sec execute 139 sec latency 49.455 ms oleg316-client.virtnet: 1 20342 0.56 MB/sec execute 140 sec latency 49.437 ms oleg316-client.virtnet: 1 20657 0.58 MB/sec execute 141 sec latency 44.727 ms oleg316-client.virtnet: 1 20977 0.59 MB/sec execute 142 sec latency 45.794 ms oleg316-client.virtnet: 1 21275 0.62 MB/sec execute 143 sec latency 40.519 ms oleg316-client.virtnet: 1 21434 0.62 MB/sec execute 144 sec latency 52.639 ms oleg316-client.virtnet: 1 21550 0.62 MB/sec execute 145 sec latency 43.895 ms oleg316-client.virtnet: 1 21655 0.61 MB/sec execute 146 sec latency 43.753 ms oleg316-client.virtnet: 1 21784 0.61 MB/sec execute 147 sec latency 32.595 ms oleg316-client.virtnet: 1 21901 0.61 MB/sec execute 148 sec latency 48.078 ms oleg316-client.virtnet: 1 22071 0.61 MB/sec execute 149 sec latency 38.987 ms oleg316-client.virtnet: 1 22220 0.60 MB/sec execute 150 sec latency 34.334 ms oleg316-client.virtnet: 1 22361 0.61 MB/sec execute 151 sec latency 37.998 ms oleg316-client.virtnet: 1 22558 0.61 MB/sec execute 152 sec latency 36.825 ms oleg316-client.virtnet: 1 22761 0.62 MB/seFailover mds1 to oleg316-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 16:13:08 (1746130388) targets are mounted 16:13:08 (1746130388) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37428 3569084 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12136 3594024 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7163108 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg316-server Stopping /mnt/lustre-mds2 (opts:) on oleg316-server 16:13:23 (1746130403) shut down facet: mds2 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds2 to oleg316-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all c execute 153 sec latency 38.798 ms oleg316-client.virtnet: 1 22908 0.62 MB/sec execute 154 sec latency 48.716 ms oleg316-client.virtnet: 1 23026 0.61 MB/sec execute 155 sec latency 52.069 ms oleg316-client.virtnet: 1 23129 0.61 MB/sec execute 156 sec latency 62.800 ms oleg316-client.virtnet: 1 23299 0.61 MB/sec execute 157 sec latency 41.254 ms oleg316-client.virtnet: 1 23425 0.61 MB/sec execute 158 sec latency 38.396 ms oleg316-client.virtnet: 1 23627 0.61 MB/sec execute 159 sec latency 38.809 ms oleg316-client.virtnet: 1 23761 0.61 MB/sec execute 160 sec latency 62.035 ms oleg316-client.virtnet: 1 23932 0.61 MB/sec execute 161 sec latency 38.177 ms oleg316-client.virtnet: 1 24319 0.63 MB/sec execute 162 sec latency 32.611 ms oleg316-client.virtnet: 1 24681 0.64 MB/sec execute 163 sec latency 42.401 ms oleg316-client.virtnet: 1 24909 0.65 MB/sec execute 164 sec latency 37.018 ms oleg316-client.virtnet: 1 25089 0.66 MB/sec execute 165 sec latency 32.654 ms oleg316-client.virtnet: 1 25218 0.66 MB/sec execute 166 sec latency 41.905 ms oleg316-client.virtnet: 1 25355 0.65 MB/sec execute 167 sec latency 34.147 ms oleg316-client.virtnet: 1 25527 0.65 MB/sec execute 168 sec latency 41.374 ms oleg316-client.virtnet: 1 25660 0.65 MB/sec execute 169 sec latency 42.037 ms oleg316-client.virtnet: 1 25798 0.65 MB/sec execute 170 sec latency 38.196 ms oleg316-client.virtnet: 1 25913 0.65 MB/sec execute 171 sec latency 82.227 ms oleg316-client.virtnet: 1 26104 0.65 MB/sec execute 172 sec latency 44.221 ms oleg316-client.virtnet: 1 26341 0.66 MB/sec execute 173 sec latency 31.715 ms oleg316-client.virtnet: 1 26486 0.66 MB/sec execute 174 sec latency 45.235 ms oleg316-client.virtnet: 1 26632 0.65 MB/sec execute 175 sec latency 38.552 ms oleg316-client.virtnet: 1 26765 0.65 MB/sec execute 176 sec latency 58.982 ms oleg316-client.virtnet: 1 26887 0.65 MB/sec execute 177 sec latency 513.861 ms oleg316-client.virtnet: 1 27102 0.65 MB/sec execute 178 sec latency 33.164 ms oleg316-client.virtnet: 1 27276 0.65 MB/sec execute 179 sec latency 42.073 ms oleg316-client.virtnet: 1 27342 0.65 MB/sec execute 180 sec latency 491.554 ms oleg316-client.virtnet: 1 27342 0.64 MB/sec execute 181 sec latency 1492.505 ms oleg316-client.virtnet: 1 27342 0.64 MB/sec execute 182 sec latency 2492.760 ms oleg316-client.virtnet: 1 27342 0.64 MB/sec execute 183 sec latency 3493.005 ms oleg316-client.virtnet: 1 27342 0.63 MB/sec execute 184 sec latency 4493.261 ms oleg316-client.virtnet: 1 27342 0.63 MB/sec execute 185 sec latency 5494.515 ms oleg316-client.virtnet: 1 27342 0.63 MB/sec execute 186 sec latency 6495.261 ms oleg316-client.virtnet: 1 27342 0.62 MB/sec execute 187 sec latency 7497.201 ms oleg316-client.virtnet: 1 27342 0.62 MB/sec execute 188 sec latency 8497.512 ms oleg316-client.virtnet: 1 27342 0.62 MB/sec execute 189 sec latency 9497.815 ms oleg316-client.virtnet: 1 27342 0.61 MB/sec execute 190 sec latency 10499.882 ms oleg316-client.virtnet: 1 27342 0.61 MB/sec execute 191 sec latency 11500.232 ms oleg316-client.virtnet: 1 27342 0.61 MB/sec execute 192 sec latency 12500.844 ms oleg316-client.virtnet: 1 27342 0.60 MB/sec execute 193 sec latency 13502.122 ms oleg316-client.virtnet: 1 27342 0.60 MB/sec execute 194 sec latency 14502.423 ms oleg316-client.virtnet: 1 27342 0.60 MB/sec execute 195 sec latency 15503.226 ms oleg316-client.virtnet: 1 27342 0.60 MB/sec execute 196 sec latency 16504.846 ms oleg316-client.virtnet: 1 27342 0.59 MB/sec execute 197 sec latency 17505.045 ms oleg316-clpdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 16:13:42 (1746130422) targets are mounted 16:13:42 (1746130422) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37544 3568892 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12072 3592204 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7161096 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg316-server Stopping /mnt/lustre-mds1 (opts:) on oleg316-server 16:13:57 (1746130437) shut down facet: mds1 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds1 to oleg316-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 16:14:16 (1746130456) targets are mounted 16:14:16 (1746130456) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec ient.virtnet: 1 27342 0.59 MB/sec execute 198 sec latency 18506.739 ms oleg316-client.virtnet: 1 27342 0.59 MB/sec execute 199 sec latency 19506.976 ms oleg316-client.virtnet: 1 27342 0.58 MB/sec execute 200 sec latency 20507.213 ms oleg316-client.virtnet: 1 27349 0.58 MB/sec execute 201 sec latency 21274.918 ms oleg316-client.virtnet: 1 27479 0.58 MB/sec execute 202 sec latency 66.196 ms oleg316-client.virtnet: 1 27788 0.60 MB/sec execute 203 sec latency 57.891 ms oleg316-client.virtnet: 1 27896 0.60 MB/sec execute 204 sec latency 144.731 ms oleg316-client.virtnet: 1 28236 0.60 MB/sec execute 205 sec latency 43.610 ms oleg316-client.virtnet: 1 28449 0.61 MB/sec execute 206 sec latency 46.862 ms oleg316-client.virtnet: 1 28632 0.62 MB/sec execute 207 sec latency 42.634 ms oleg316-client.virtnet: 1 28730 0.61 MB/sec execute 208 sec latency 65.508 ms oleg316-client.virtnet: 1 28852 0.61 MB/sec execute 209 sec latency 71.853 ms oleg316-client.virtnet: 1 29010 0.61 MB/sec execute 210 sec latency 45.262 ms oleg316-client.virtnet: 1 29175 0.61 MB/sec execute 211 sec latency 48.296 ms oleg316-client.virtnet: 1 29315 0.61 MB/sec execute 212 sec latency 36.486 ms oleg316-client.virtnet: 1 29395 0.61 MB/sec execute 213 sec latency 80.321 ms oleg316-client.virtnet: 1 29502 0.61 MB/sec execute 214 sec latency 64.707 ms oleg316-client.virtnet: 1 29625 0.61 MB/sec execute 215 sec latency 97.885 ms oleg316-client.virtnet: 1 29871 0.62 MB/sec execute 216 sec latency 35.808 ms oleg316-client.virtnet: 1 29991 0.61 MB/sec execute 217 sec latency 58.075 ms oleg316-client.virtnet: 1 30134 0.61 MB/sec execute 218 sec latency 36.135 ms oleg316-client.virtnet: 1 30259 0.61 MB/sec execute 219 sec latency 34.498 ms oleg316-client.virtnet: 1 30436 0.61 MB/sec execute 220 sec latency 44.400 ms oleg316-client.virtnet: 1 30605 0.61 MB/sec execute 221 sec latency 30.245 ms oleg316-client.virtnet: 1 30825 0.61 MB/sec execute 222 sec latency 31.339 ms oleg316-client.virtnet: 1 31020 0.61 MB/sec execute 223 sec latency 42.802 ms oleg316-client.virtnet: 1 31403 0.63 MB/sec execute 224 sec latency 36.552 ms oleg316-client.virtnet: 1 31693 0.63 MB/sec execute 225 sec latency 57.581 ms oleg316-client.virtnet: 1 31934 0.64 MB/sec execute 226 sec latency 49.317 ms oleg316-client.virtnet: 1 32032 0.64 MB/sec execute 227 sec latency 53.973 ms oleg316-client.virtnet: 1 32108 0.64 MB/sec execute 228 sec latency 63.986 ms oleg316-client.virtnet: 1 32226 0.64 MB/sec execute 229 sec latency 89.314 ms oleg316-client.virtnet: 1 32338 0.64 MB/sec execute 230 sec latency 59.184 ms oleg316-client.virtnet: 1 32443 0.64 MB/sec execute 231 sec latency 44.036 ms oleg316-client.virtnet: 1 32489 0.63 MB/sec execute 232 sec latency 506.511 ms oleg316-client.virtnet: 1 32489 0.63 MB/sec execute 233 sec latency 1506.973 ms oleg316-client.virtnet: 1 32489 0.63 MB/sec execute 234 sec latency 2508.414 ms oleg316-client.virtnet: 1 32489 0.63 MB/sec execute 235 sec latency 3509.462 ms oleg316-client.virtnet: 1 32584 0.62 MB/sec execute 236 sec latency 3966.473 ms oleg316-client.virtnet: 1 32694 0.62 MB/sec execute 237 sec latency 54.724 ms oleg316-client.virtnet: 1 32823 0.62 MB/sec execute 238 sec latency 47.566 ms oleg316-client.virtnet: 1 32926 0.62 MB/sec execute 239 sec latency 67.963 ms oleg316-client.virtnet: 1 33104 0.62 MB/sec execute 240 sec latency 39.795 ms oleg316-client.virtnet: 1 33362 0.63 MB/sec execute 241 sec latency 30.558 ms oleg316-client.virtnet: 1 33516 0.63 MB/sec execute 242 sec latency 33.465 ms oleg316-client.virtnet: 1 33654 0.63 MB/sec execute 243 sec latency 36.765 ms oleg316-client.virtnet: 1 33769 0.63 MB/sec execute 244 sec latency 41.224 ms oleg316-client.virtnet: 1 33948 0.62 MB/sec execute 245 sec latency 41.912 ms oleg316-client.virtnet: 1 34116 0.62 MB/sec execute 246 sec latency 45.111 ms oleg316-client.virtnet: 1 34318 0.62 MB/sec execute 247 sec latency 35.231 ms oleg316-client.virtnet: 1 34456 0.63 MB/sec execute 248 sec latency 36.957 ms oleg316-client.virtnet: 1 34842 0.64 MB/sec execute 249 sec latency 36.547 ms oleg316-client.virtnet: 1 35158 0.64 MB/sec execute 250 sec latency 37.644 ms oleg316-client.virtnet: 1 35473 0.66 MB/sec execute 251 sec latency 44.027 ms oleg316-client.virtnet: 1 35645 0.65 MB/sec execute 252 sec latency 32.481 ms oleg316-client.virtnet: 1 35810 0.66 MB/sec execute 253 sec latency 34.702 ms oleg316-client.virtnet: 1 35959 0.65 MB/sec execute 254 sec latency 45.927 ms oleg316-client.virtnet: 1 36184 0.65 MB/sec execute 255 sec latency 25.620 ms oleg316-client.virtnet: 1 36310 0.65 MB/sec execute 256 sec latency 40.958 ms oleg316-client.virtnet: 1 36495 0.65 MB/sec execute 257 sec latency 32.368 ms oleg316-client.virtnet: 1 36696 0.65 MB/sec execute 258 sec latency 29.109 ms oleg316-client.virtnet: 1 36933 0.66 MB/sec execute 259 sec latency 45.484 ms oleg316-client.virtnet: 1 37096 0.66 MB/sec execute 260 sec latency 29.911 ms oleg316-client.virtnet: 1 37245 0.66 MB/sec execute 261 sec latency 30.726 ms oleg316-client.virtnet: 1 37402 0.66 MB/sec execute 262 sec latency 38.220 ms oleg316-client.virtnet: 1 37649 0.65 MB/sec execute 263 sec latency 25.688 ms oleg316-client.virtnet: 1 37893 0.66 MB/sec execute 264 sec latency 23.561 ms oleg316-client.virtnet: 1 38064 0.66 MB/sec execute 265 sec latency 42.406 ms oleg316-client.virtnet: 1 38406 0.67 MB/sec execute 266 sec latency 32.751 ms oleg316-client.virtnet: 1 38736 0.67 MB/sec execute 267 sec latency 36.628 ms oleg316-client.virtnet: 1 39072 0.69 MB/sec execute 268 sec latency 29.549 ms oleg316-client.virtnet: 1 39272 0.69 MB/sec execute 269 sec latency 32.361 ms oleg316-client.virtnet: 1 39410 0.69 MB/sec execute 270 sec latency 28.169 ms oleg316-client.virtnet: 1 39544 0.68 MB/sec execute 271 sec latency 35.742 ms oleg316-client.virtnet: 1 39739 0.68 MB/sec execute 272 sec latency 66.168 ms oleg316-client.virtnet: 1 39871 0.68 MB/sec execute 273 sec latency 31.854 ms oleg316-client.virtnet: 1 40017 0.68 MB/sec execute 274 sec latency 34.927 ms oleg316-client.virtnet: 1 40182 0.68 MB/sec execute 275 sec latency 45.989 ms oleg316-client.virtnet: 1 40313 0.68 MB/sec execute 276 sec latency 36.334 ms oleg316-client.virtnet: 1 40514 0.69 MB/sec execute 277 sec latency 46.697 ms oleg316-client.virtnet: 1 40632 0.69 MB/sec execute 278 sec latency 56.712 ms oleg316-client.virtnet: 1 40771 0.68 MB/sec execute 279 sec latency 39.009 ms oleg316-client.virtnet: 1 40870 0.68 MB/sec execute 280 sec latency 46.570 ms oleg316-client.virtnet: 1 41051 0.68 MB/sec execute 281 sec latency 36.313 ms oleg316-client.virtnet: 1 41219 0.68 MB/sec execute 282 sec latency 35.743 ms oleg316-client.virtnet: 1 41416 0.68 MB/sec execute 283 sec latency 47.893 ms oleg316-client.virtnet: 1 41549 0.68 MB/sec execute 284 sec latency 42.530 ms oleg316-client.virtnet: 1 41934 0.69 MB/sec execute 285 sec latency 34.489 ms oleg316-client.virtnet: 1 42251 0.70 MB/sec execute 286 sec latency 62.662 ms oleg316-client.virtnet: 1 42573 0.71 MB/sec execute 287 sec latency 32.510 ms oleg316-client.virtnet: 1 42731 0.71 MB/sec execute 288 sec latency 40.126 ms oleg316-client.virtnet: 1 42902 0.71 MB/sec execute 289 sec latency 29.560 ms oleg316-client.virtnet: 1 43064 0.71 MB/sec execute 290 sec latency 33.304 ms oleg316-client.virtnet: 1 43286 0.70 MB/sec execute 291 sec latency 30.878 ms oleg316-client.virtnet: 1 43412 0.70 MB/sec execute 292 sec latency 83.764 ms oleg316-client.virtnet: 1 43562 0.70 MB/sec execute 293 sec latency 40.405 ms oleg316-client.virtnet: 1 43743 0.70 MB/sec execute 294 sec latency 40.378 ms oleg316-client.virtnet: 1 43982 0.71 MB/sec execute 295 sec latency 29.168 ms oleg316-client.virtnet: 1 44151 0.71 MB/sec execute 296 sec latency 41.232 ms oleg316-client.virtnet: 1 44316 0.71 MB/sec execute 297 sec latency 31.662 ms oleg316-client.virtnet: 1 44451 0.70 MB/sec execute 298 sec latency 29.829 ms oleg316-client.virtnet: 1 44627 0.70 MB/sec execute 299 sec latency 32.265 ms oleg316-client.virtnet: 1 cleanup 300 sec oleg316-client.virtnet: 0 cleanup 301 sec oleg316-client.virtnet: oleg316-client.virtnet: Operation Count AvgLat MaxLat oleg316-client.virtnet: ---------------------------------------- oleg316-client.virtnet: NTCreateX 6793 24.769 20948.434 oleg316-client.virtnet: Close 5013 3.809 38.155 oleg316-client.virtnet: Rename 286 27.229 83.740 oleg316-client.virtnet: Unlink 1367 5.976 40.967 oleg316-client.virtnet: Qpathinfo 6166 13.160 21274.897 oleg316-client.virtnet: Qfileinfo 1074 0.315 6.818 oleg316-client.virtnet: Qfsinfo 1122 0.656 22.654 oleg316-client.virtnet: Sfileinfo 550 17.310 97.871 oleg316-client.virtnet: Find 2375 1.004 49.391 oleg316-client.virtnet: WriteX 3351 2.674 35.214 oleg316-client.virtnet: ReadX 10626 0.055 51.183 oleg316-client.virtnet: LockX 22 4.372 15.373 oleg316-client.virtnet: UnlockX 22 3.824 10.551 oleg316-client.virtnet: Flush 473 22.889 652.558 oleg316-client.virtnet: oleg316-client.virtnet: Throughput 0.702866 MB/sec 1 clients 1 procs max_latency=21274.918 ms oleg316-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg316-client.virtnet at Thu May 1 16:15:23 EDT 2025 with return code 0 oleg316-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg316-client.virtnet oleg316-client.virtnet: /mnt/lustre/d70b.replay-single/oleg316-client.virtnet /mnt/lustre/d70b.replay-single/oleg316-client.virtnet oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg316-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg316-client.virtnet: removed directory: 'clients/client0' oleg316-client.virtnet: removed directory: 'clients' oleg316-client.virtnet: removed 'client.txt' oleg316-client.virtnet: /mnt/lustre/d70b.replay-single/oleg316-client.virtnet oleg316-client.virtnet: dbench successfully finished