== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 01:02:13 (1746075733) Starting client oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre Started clients oleg216-client.virtnet: 192.168.202.116@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg216-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14006 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2504 1285184 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26152 3580868 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7186316 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg216-server Stopping /mnt/lustre-mds1 (opts:) on oleg216-server 01:02:25 (1746075745) shut down facet: mds1 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds1 to oleg216-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 01:02:43 (1746075763) targets are mounted 01:02:43 (1746075763) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg216-client.virtnet: looking for dbench program oleg216-client.virtnet: /usr/bin/dbench oleg216-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg216-client.virtnet oleg216-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg216-client.virtnet' oleg216-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg216-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg216-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg216-client.virtnet at Thu May 1 01:02:15 EDT 2025 oleg216-client.virtnet: waiting for dbench pid 14031 oleg216-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg216-client.virtnet: oleg216-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg216-client.virtnet: failed to create barrier semaphore oleg216-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg216-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg216-client.virtnet: releasing clients oleg216-client.virtnet: 1 147 6.28 MB/sec warmup 1 sec latency 64.675 ms oleg216-client.virtnet: 1 290 5.35 MB/sec warmup 2 sec latency 44.768 ms oleg216-client.virtnet: 1 413 4.86 MB/sec warmup 3 sec latency 60.004 ms oleg216-client.virtnet: 1 590 5.08 MB/sec warmup 4 sec latency 72.412 ms oleg216-client.virtnet: 1 712 4.24 MB/sec warmup 5 sec latency 51.348 ms oleg216-client.virtnet: 1 762 3.56 MB/sec warmup 6 sec latency 604.643 ms oleg216-client.virtnet: 1 892 3.10 MB/sec warmup 7 sec latency 42.315 ms oleg216-client.virtnet: 1 1025 2.85 MB/sec warmup 8 sec latency 44.381 ms oleg216-client.virtnet: 1 1155 2.55 MB/sec warmup 9 sec latency 43.127 ms oleg216-client.virtnet: 1 1155 2.30 MB/sec warmup 10 sec latency 1008.464 ms oleg216-client.virtnet: 1 1155 2.09 MB/sec warmup 11 sec latency 2008.626 ms oleg216-client.virtnet: 1 1155 1.92 MB/sec warmup 12 sec latency 3009.721 ms oleg216-client.virtnet: 1 1155 1.77 MB/sec warmup 13 sec latency 4012.582 ms oleg216-client.virtnet: 1 1155 1.64 MB/sec warmup 14 sec latency 5012.839 ms oleg216-client.virtnet: 1 1155 1.53 MB/sec warmup 15 sec latency 6014.521 ms oleg216-client.virtnet: 1 1155 1.44 MB/sec warmup 16 sec latency 7014.734 ms oleg216-client.virtnet: 1 1155 1.35 MB/sec warmup 17 sec latency 8019.061 ms oleg216-client.virtnet: 1 1155 1.28 MB/sec warmup 18 sec latency 9020.047 ms oleg216-client.virtnet: 1 1155 1.21 MB/sec warmup 19 sec latency 10020.885 ms oleg216-client.virtnet: 1 1155 1.15 MB/sec warmup 20 sec latency 11022.089 ms oleg216-client.virtnet: 1 1155 1.09 MB/sec warmup 21 sec latency 12024.050 ms oleg216-client.virtnet: 1 1155 1.04 MB/sec warmup 22 sec latency 13027.164 ms oleg216-client.virtnet: 1 1155 1.00 MB/sec warmup 23 sec latency 14027.387 ms oleg216-client.virtnet: 1 1155 0.96 MB/sec warmup 24 sec latency 15028.365 ms oleg216-client.virtnet: 1 1155 0.92 MB/sec warmup 25 sec latency 16028.579 ms oleg216-client.virtnet: 1 1155 0.88 MB/sec warmup 26 sec latency 17030.045 ms oleg216-client.virtnet: 1 1155 0.85 MB/sec warmup 27 sec latency 18030.931 ms oleg216-client.virtnet: 1 1155 0.82 MB/sec warmup 28 sec latency 19031.188 ms oleg216-client.virtnet: 1 1155 0.79 MB/sec warmup 29 sec latency 20031.422 ms oleg216-client.virtnet: 1 1155 0.77 MB/sec warmup 30 sec latency 21032.998 ms oleg216-client.virtnet: 1 1173 0.74 MB/sec warmup 31 sec latency 21914.808 ms oleg216-client.virtnet: 1 1272 0.72 MB/sec warmup 32 sec latency 47.336 ms oleg216-client.virtnet: 1 1481 0.79 MB/sec warmup 33 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1752 3590804 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 27184 3566908 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28936 7157712 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg216-server Stopping /mnt/lustre-mds2 (opts:) on oleg216-server 01:02:59 (1746075779) shut down facet: mds2 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds2 to oleg216-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0001 01:03:17 (1746075797) targets are mounted 01:03:17 (1746075797) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13244 3592904 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36396 3570248 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49640 7163152 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg216-server Stopping /mnt/lustre-mds1 (opts:) on oleg216-server latency 46.203 ms oleg216-client.virtnet: 1 1591 0.77 MB/sec warmup 34 sec latency 46.671 ms oleg216-client.virtnet: 1 1689 0.75 MB/sec warmup 35 sec latency 91.402 ms oleg216-client.virtnet: 1 1888 0.73 MB/sec warmup 36 sec latency 43.901 ms oleg216-client.virtnet: 1 2026 0.72 MB/sec warmup 37 sec latency 48.386 ms oleg216-client.virtnet: 1 2200 0.70 MB/sec warmup 38 sec latency 38.382 ms oleg216-client.virtnet: 1 2397 0.72 MB/sec warmup 39 sec latency 33.315 ms oleg216-client.virtnet: 1 2561 0.72 MB/sec warmup 40 sec latency 50.440 ms oleg216-client.virtnet: 1 2917 0.82 MB/sec warmup 41 sec latency 34.468 ms oleg216-client.virtnet: 1 3025 0.80 MB/sec warmup 42 sec latency 67.896 ms oleg216-client.virtnet: 1 3380 0.82 MB/sec warmup 43 sec latency 33.574 ms oleg216-client.virtnet: 1 3628 0.88 MB/sec warmup 44 sec latency 40.180 ms oleg216-client.virtnet: 1 3780 0.88 MB/sec warmup 45 sec latency 42.057 ms oleg216-client.virtnet: 1 3884 0.86 MB/sec warmup 46 sec latency 41.070 ms oleg216-client.virtnet: 1 3998 0.85 MB/sec warmup 47 sec latency 40.417 ms oleg216-client.virtnet: 1 4083 0.83 MB/sec warmup 48 sec latency 298.281 ms oleg216-client.virtnet: 1 4083 0.82 MB/sec warmup 49 sec latency 1298.868 ms oleg216-client.virtnet: 1 4083 0.80 MB/sec warmup 50 sec latency 2299.067 ms oleg216-client.virtnet: 1 4083 0.79 MB/sec warmup 51 sec latency 3300.524 ms oleg216-client.virtnet: 1 4083 0.77 MB/sec warmup 52 sec latency 4302.997 ms oleg216-client.virtnet: 1 4083 0.76 MB/sec warmup 53 sec latency 5304.615 ms oleg216-client.virtnet: 1 4083 0.74 MB/sec warmup 54 sec latency 6304.791 ms oleg216-client.virtnet: 1 4083 0.73 MB/sec warmup 55 sec latency 7305.039 ms oleg216-client.virtnet: 1 4083 0.72 MB/sec warmup 56 sec latency 8305.547 ms oleg216-client.virtnet: 1 4083 0.70 MB/sec warmup 57 sec latency 9305.787 ms oleg216-client.virtnet: 1 4083 0.69 MB/sec warmup 58 sec latency 10305.963 ms oleg216-client.virtnet: 1 4083 0.68 MB/sec warmup 59 sec latency 11308.531 ms oleg216-client.virtnet: 1 4083 0.00 MB/sec execute 1 sec latency 13310.197 ms oleg216-client.virtnet: 1 4083 0.00 MB/sec execute 2 sec latency 14310.521 ms oleg216-client.virtnet: 1 4083 0.00 MB/sec execute 3 sec latency 15310.770 ms oleg216-client.virtnet: 1 4083 0.00 MB/sec execute 4 sec latency 16310.981 ms oleg216-client.virtnet: 1 4151 0.00 MB/sec execute 5 sec latency 16455.896 ms oleg216-client.virtnet: 1 4277 0.04 MB/sec execute 6 sec latency 46.284 ms oleg216-client.virtnet: 1 4350 0.06 MB/sec execute 7 sec latency 67.835 ms oleg216-client.virtnet: 1 4474 0.09 MB/sec execute 8 sec latency 43.265 ms oleg216-client.virtnet: 1 4628 0.21 MB/sec execute 9 sec latency 45.038 ms oleg216-client.virtnet: 1 4795 0.21 MB/sec execute 10 sec latency 28.166 ms oleg216-client.virtnet: 1 4976 0.46 MB/sec execute 11 sec latency 49.468 ms oleg216-client.virtnet: 1 5066 0.42 MB/sec execute 12 sec latency 62.256 ms oleg216-client.virtnet: 1 5147 0.40 MB/sec execute 13 sec latency 67.821 ms oleg216-client.virtnet: 1 5197 0.37 MB/sec execute 14 sec latency 546.322 ms oleg216-client.virtnet: 1 5280 0.35 MB/sec execute 15 sec latency 645.906 ms oleg216-client.virtnet: 1 5408 0.34 MB/sec execute 16 sec latency 37.960 ms oleg216-client.virtnet: 1 5518 0.32 MB/sec execute 17 sec latency 60.893 ms oleg216-client.virtnet: 1 5618 0.31 MB/sec execute 18 sec latency 54.328 ms oleg216-client.virtnet: 1 5650 0.30 MB/sec ex01:03:35 (1746075815) shut down facet: mds1 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds1 to oleg216-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 01:03:53 (1746075833) targets are mounted 01:03:53 (1746075833) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13272 3592848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36412 3567876 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49684 7160724 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg216-server Stopping /mnt/lustre-mds2 (opts:) on oleg216-server 01:04:08 (1746075848) shut down facet: mds2 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds2 to oleg216-server mount facets: mds2 ecute 19 sec latency 686.716 ms oleg216-client.virtnet: 1 5650 0.28 MB/sec execute 20 sec latency 1686.943 ms oleg216-client.virtnet: 1 5650 0.27 MB/sec execute 21 sec latency 2687.153 ms oleg216-client.virtnet: 1 5650 0.26 MB/sec execute 22 sec latency 3687.950 ms oleg216-client.virtnet: 1 5650 0.25 MB/sec execute 23 sec latency 4688.279 ms oleg216-client.virtnet: 1 5650 0.24 MB/sec execute 24 sec latency 5689.452 ms oleg216-client.virtnet: 1 5650 0.23 MB/sec execute 25 sec latency 6693.441 ms oleg216-client.virtnet: 1 5650 0.22 MB/sec execute 26 sec latency 7694.139 ms oleg216-client.virtnet: 1 5650 0.21 MB/sec execute 27 sec latency 8697.574 ms oleg216-client.virtnet: 1 5650 0.20 MB/sec execute 28 sec latency 9697.972 ms oleg216-client.virtnet: 1 5650 0.20 MB/sec execute 29 sec latency 10698.212 ms oleg216-client.virtnet: 1 5650 0.19 MB/sec execute 30 sec latency 11698.393 ms oleg216-client.virtnet: 1 5650 0.18 MB/sec execute 31 sec latency 12698.690 ms oleg216-client.virtnet: 1 5650 0.18 MB/sec execute 32 sec latency 13699.609 ms oleg216-client.virtnet: 1 5650 0.17 MB/sec execute 33 sec latency 14699.909 ms oleg216-client.virtnet: 1 5650 0.17 MB/sec execute 34 sec latency 15701.026 ms oleg216-client.virtnet: 1 5650 0.16 MB/sec execute 35 sec latency 16701.285 ms oleg216-client.virtnet: 1 5650 0.16 MB/sec execute 36 sec latency 17701.527 ms oleg216-client.virtnet: 1 5650 0.15 MB/sec execute 37 sec latency 18701.759 ms oleg216-client.virtnet: 1 5650 0.15 MB/sec execute 38 sec latency 19702.573 ms oleg216-client.virtnet: 1 5650 0.15 MB/sec execute 39 sec latency 20702.815 ms oleg216-client.virtnet: 1 5650 0.14 MB/sec execute 40 sec latency 21705.915 ms oleg216-client.virtnet: 1 5672 0.14 MB/sec execute 41 sec latency 22551.140 ms oleg216-client.virtnet: 1 5889 0.16 MB/sec execute 42 sec latency 32.749 ms oleg216-client.virtnet: 1 6025 0.18 MB/sec execute 43 sec latency 29.407 ms oleg216-client.virtnet: 1 6381 0.25 MB/sec execute 44 sec latency 29.585 ms oleg216-client.virtnet: 1 6746 0.29 MB/sec execute 45 sec latency 38.257 ms oleg216-client.virtnet: 1 7075 0.38 MB/sec execute 46 sec latency 40.841 ms oleg216-client.virtnet: 1 7292 0.39 MB/sec execute 47 sec latency 27.566 ms oleg216-client.virtnet: 1 7422 0.39 MB/sec execute 48 sec latency 41.247 ms oleg216-client.virtnet: 1 7520 0.39 MB/sec execute 49 sec latency 38.507 ms oleg216-client.virtnet: 1 7650 0.38 MB/sec execute 50 sec latency 38.034 ms oleg216-client.virtnet: 1 7812 0.38 MB/sec execute 51 sec latency 37.330 ms oleg216-client.virtnet: 1 7912 0.38 MB/sec execute 52 sec latency 48.164 ms oleg216-client.virtnet: 1 8014 0.37 MB/sec execute 53 sec latency 67.313 ms oleg216-client.virtnet: 1 8161 0.39 MB/sec execute 54 sec latency 46.902 ms oleg216-client.virtnet: 1 8324 0.39 MB/sec execute 55 sec latency 45.158 ms oleg216-client.virtnet: 1 8545 0.43 MB/sec execute 56 sec latency 35.588 ms oleg216-client.virtnet: 1 8678 0.43 MB/sec execute 57 sec latency 37.514 ms oleg216-client.virtnet: 1 8792 0.42 MB/sec execute 58 sec latency 40.794 ms oleg216-client.virtnet: 1 8909 0.41 MB/sec execute 59 sec latency 49.289 ms oleg216-client.virtnet: 1 9091 0.41 MB/sec execute 60 sec latency 42.347 ms oleg216-client.virtnet: 1 9251 0.41 MB/sec execute 61 sec latency 40.374 ms oleg216-client.virtnet: 1 9454 0.42 MB/sec execute 62 sec latency 32.298 ms oleg216-client.virtnet: 1 9582 0.43 MB/sec execute 63 sec latency 39.175 ms oStarting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0001 01:04:27 (1746075867) targets are mounted 01:04:27 (1746075867) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12932 3593284 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36632 3569996 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7163280 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg216-server Stopping /mnt/lustre-mds1 (opts:) on oleg216-server 01:04:42 (1746075882) shut down facet: mds1 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds1 to oleg216-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 01:04:59 (1746075899) targets are mounted 01:04:59 (1746075899) facet_failover done leg216-client.virtnet: 1 9865 0.47 MB/sec execute 64 sec latency 46.037 ms oleg216-client.virtnet: 1 10089 0.49 MB/sec execute 65 sec latency 44.364 ms oleg216-client.virtnet: 1 10372 0.51 MB/sec execute 66 sec latency 56.093 ms oleg216-client.virtnet: 1 10622 0.54 MB/sec execute 67 sec latency 56.283 ms oleg216-client.virtnet: 1 10744 0.54 MB/sec execute 68 sec latency 48.717 ms oleg216-client.virtnet: 1 10888 0.55 MB/sec execute 69 sec latency 47.209 ms oleg216-client.virtnet: 1 10997 0.55 MB/sec execute 70 sec latency 33.378 ms oleg216-client.virtnet: 1 11055 0.54 MB/sec execute 71 sec latency 531.120 ms oleg216-client.virtnet: 1 11055 0.53 MB/sec execute 72 sec latency 1532.057 ms oleg216-client.virtnet: 1 11055 0.52 MB/sec execute 73 sec latency 2533.104 ms oleg216-client.virtnet: 1 11055 0.52 MB/sec execute 74 sec latency 3533.286 ms oleg216-client.virtnet: 1 11134 0.51 MB/sec execute 75 sec latency 3680.217 ms oleg216-client.virtnet: 1 11277 0.50 MB/sec execute 76 sec latency 61.237 ms oleg216-client.virtnet: 1 11391 0.50 MB/sec execute 77 sec latency 68.482 ms oleg216-client.virtnet: 1 11516 0.50 MB/sec execute 78 sec latency 50.693 ms oleg216-client.virtnet: 1 11627 0.49 MB/sec execute 79 sec latency 43.014 ms oleg216-client.virtnet: 1 11792 0.50 MB/sec execute 80 sec latency 38.710 ms oleg216-client.virtnet: 1 11907 0.50 MB/sec execute 81 sec latency 42.905 ms oleg216-client.virtnet: 1 12103 0.53 MB/sec execute 82 sec latency 49.736 ms oleg216-client.virtnet: 1 12137 0.52 MB/sec execute 83 sec latency 718.134 ms oleg216-client.virtnet: 1 12243 0.52 MB/sec execute 84 sec latency 884.606 ms oleg216-client.virtnet: 1 12372 0.51 MB/sec execute 85 sec latency 60.049 ms oleg216-client.virtnet: 1 12481 0.51 MB/sec execute 86 sec latency 60.724 ms oleg216-client.virtnet: 1 12560 0.50 MB/sec execute 87 sec latency 513.056 ms oleg216-client.virtnet: 1 12560 0.50 MB/sec execute 88 sec latency 1513.613 ms oleg216-client.virtnet: 1 12560 0.49 MB/sec execute 89 sec latency 2517.957 ms oleg216-client.virtnet: 1 12560 0.49 MB/sec execute 90 sec latency 3518.159 ms oleg216-client.virtnet: 1 12560 0.48 MB/sec execute 91 sec latency 4521.383 ms oleg216-client.virtnet: 1 12560 0.47 MB/sec execute 92 sec latency 5521.600 ms oleg216-client.virtnet: 1 12560 0.47 MB/sec execute 93 sec latency 6521.872 ms oleg216-client.virtnet: 1 12560 0.46 MB/sec execute 94 sec latency 7522.039 ms oleg216-client.virtnet: 1 12560 0.46 MB/sec execute 95 sec latency 8523.006 ms oleg216-client.virtnet: 1 12560 0.45 MB/sec execute 96 sec latency 9523.566 ms oleg216-client.virtnet: 1 12560 0.45 MB/sec execute 97 sec latency 10523.800 ms oleg216-client.virtnet: 1 12560 0.45 MB/sec execute 98 sec latency 11525.906 ms oleg216-client.virtnet: 1 12560 0.44 MB/sec execute 99 sec latency 12529.503 ms oleg216-client.virtnet: 1 12560 0.44 MB/sec execute 100 sec latency 13531.056 ms oleg216-client.virtnet: 1 12560 0.43 MB/sec execute 101 sec latency 14531.818 ms oleg216-client.virtnet: 1 12560 0.43 MB/sec execute 102 sec latency 15532.145 ms oleg216-client.virtnet: 1 12560 0.42 MB/sec execute 103 sec latency 16532.634 ms oleg216-client.virtnet: 1 12560 0.42 MB/sec execute 104 sec latency 17532.856 ms oleg216-client.virtnet: 1 12560 0.42 MB/sec execute 105 sec latency 18533.035 ms oleg216-client.virtnet: 1 12560 0.41 MB/sec execute 106 sec latency 19533.294 ms oleg216-client.virtnet: 1 12560 0.41 MB/sec execute 107 sec latency 20533.548 ms oleg216-client.virtnet: 1 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13020 3592988 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36632 3567756 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49652 7160744 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg216-server Stopping /mnt/lustre-mds2 (opts:) on oleg216-server 01:05:15 (1746075915) shut down facet: mds2 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds2 to oleg216-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0001 01:05:33 (1746075933) targets are mounted 01:05:33 (1746075933) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13056 3592908 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36556 3567360 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7160268 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg216-server Stopping /mnt/lustre-mds1 (opts:) on oleg216-server 01:05:47 (1746075947) shut down facet: mds1 facet_host: oleg216-server facet_failover_host: oleg216-server 12638 0.40 MB/sec execute 108 sec latency 21094.149 ms oleg216-client.virtnet: 1 12790 0.40 MB/sec execute 109 sec latency 42.933 ms oleg216-client.virtnet: 1 12980 0.41 MB/sec execute 110 sec latency 54.796 ms oleg216-client.virtnet: 1 13131 0.41 MB/sec execute 111 sec latency 30.109 ms oleg216-client.virtnet: 1 13526 0.44 MB/sec execute 112 sec latency 53.084 ms oleg216-client.virtnet: 1 13822 0.45 MB/sec execute 113 sec latency 40.754 ms oleg216-client.virtnet: 1 14179 0.49 MB/sec execute 114 sec latency 37.084 ms oleg216-client.virtnet: 1 14335 0.49 MB/sec execute 115 sec latency 41.119 ms oleg216-client.virtnet: 1 14492 0.49 MB/sec execute 116 sec latency 34.764 ms oleg216-client.virtnet: 1 14645 0.49 MB/sec execute 117 sec latency 31.608 ms oleg216-client.virtnet: 1 14794 0.49 MB/sec execute 118 sec latency 50.127 ms oleg216-client.virtnet: 1 14928 0.48 MB/sec execute 119 sec latency 43.158 ms oleg216-client.virtnet: 1 15055 0.48 MB/sec execute 120 sec latency 54.996 ms oleg216-client.virtnet: 1 15169 0.48 MB/sec execute 121 sec latency 55.840 ms oleg216-client.virtnet: 1 15330 0.49 MB/sec execute 122 sec latency 41.255 ms oleg216-client.virtnet: 1 15476 0.48 MB/sec execute 123 sec latency 28.287 ms oleg216-client.virtnet: 1 15699 0.50 MB/sec execute 124 sec latency 32.096 ms oleg216-client.virtnet: 1 15833 0.50 MB/sec execute 125 sec latency 40.966 ms oleg216-client.virtnet: 1 15978 0.50 MB/sec execute 126 sec latency 39.480 ms oleg216-client.virtnet: 1 16125 0.49 MB/sec execute 127 sec latency 33.332 ms oleg216-client.virtnet: 1 16296 0.49 MB/sec execute 128 sec latency 45.402 ms oleg216-client.virtnet: 1 16497 0.50 MB/sec execute 129 sec latency 38.679 ms oleg216-client.virtnet: 1 16630 0.50 MB/sec execute 130 sec latency 48.563 ms oleg216-client.virtnet: 1 16789 0.50 MB/sec execute 131 sec latency 38.182 ms oleg216-client.virtnet: 1 17110 0.53 MB/sec execute 132 sec latency 40.586 ms oleg216-client.virtnet: 1 17369 0.53 MB/sec execute 133 sec latency 56.874 ms oleg216-client.virtnet: 1 17564 0.54 MB/sec execute 134 sec latency 81.097 ms oleg216-client.virtnet: 1 17764 0.55 MB/sec execute 135 sec latency 55.487 ms oleg216-client.virtnet: 1 17834 0.55 MB/sec execute 136 sec latency 425.877 ms oleg216-client.virtnet: 1 17834 0.55 MB/sec execute 137 sec latency 1427.053 ms oleg216-client.virtnet: 1 17834 0.54 MB/sec execute 138 sec latency 2427.269 ms oleg216-client.virtnet: 1 17834 0.54 MB/sec execute 139 sec latency 3427.572 ms oleg216-client.virtnet: 1 17834 0.54 MB/sec execute 140 sec latency 4428.466 ms oleg216-client.virtnet: 1 17921 0.54 MB/sec execute 141 sec latency 4709.970 ms oleg216-client.virtnet: 1 18038 0.54 MB/sec execute 142 sec latency 45.486 ms oleg216-client.virtnet: 1 18154 0.53 MB/sec execute 143 sec latency 46.810 ms oleg216-client.virtnet: 1 18323 0.53 MB/sec execute 144 sec latency 31.643 ms oleg216-client.virtnet: 1 18541 0.53 MB/sec execute 145 sec latency 31.092 ms oleg216-client.virtnet: 1 18692 0.53 MB/sec execute 146 sec latency 33.520 ms oleg216-client.virtnet: 1 18890 0.53 MB/sec execute 147 sec latency 28.840 ms oleg216-client.virtnet: 1 19069 0.54 MB/sec execute 148 sec latency 60.189 ms oleg216-client.virtnet: 1 19314 0.55 MB/sec execute 149 sec latency 29.128 ms oleg216-client.virtnet: 1 19505 0.54 MB/sec execute 150 sec latency 25.898 ms oleg216-client.virtnet: 1 19697 0.54 MB/sec execute 151 sec latency 363.818 ms oleg216-client.virtnet: 1 19697 0.54 MB/sec execute 152 sec latency 1364.071 ms oleg216-clientFailover mds1 to oleg216-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 01:06:04 (1746075964) targets are mounted 01:06:04 (1746075964) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12932 3592968 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36588 3569620 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49520 7162588 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg216-server Stopping /mnt/lustre-mds2 (opts:) on oleg216-server 01:06:19 (1746075979) shut down facet: mds2 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds2 to oleg216-server mount facets: mds2 .virtnet: 1 19697 0.54 MB/sec execute 153 sec latency 2364.267 ms oleg216-client.virtnet: 1 19697 0.53 MB/sec execute 154 sec latency 3364.568 ms oleg216-client.virtnet: 1 19697 0.53 MB/sec execute 155 sec latency 4365.116 ms oleg216-client.virtnet: 1 19697 0.52 MB/sec execute 156 sec latency 5366.098 ms oleg216-client.virtnet: 1 19697 0.52 MB/sec execute 157 sec latency 6366.296 ms oleg216-client.virtnet: 1 19697 0.52 MB/sec execute 158 sec latency 7368.194 ms oleg216-client.virtnet: 1 19697 0.52 MB/sec execute 159 sec latency 8368.612 ms oleg216-client.virtnet: 1 19697 0.51 MB/sec execute 160 sec latency 9368.837 ms oleg216-client.virtnet: 1 19697 0.51 MB/sec execute 161 sec latency 10369.094 ms oleg216-client.virtnet: 1 19697 0.51 MB/sec execute 162 sec latency 11369.736 ms oleg216-client.virtnet: 1 19697 0.50 MB/sec execute 163 sec latency 12370.045 ms oleg216-client.virtnet: 1 19697 0.50 MB/sec execute 164 sec latency 13371.018 ms oleg216-client.virtnet: 1 19697 0.50 MB/sec execute 165 sec latency 14371.268 ms oleg216-client.virtnet: 1 19697 0.49 MB/sec execute 166 sec latency 15371.948 ms oleg216-client.virtnet: 1 19697 0.49 MB/sec execute 167 sec latency 16372.408 ms oleg216-client.virtnet: 1 19697 0.49 MB/sec execute 168 sec latency 17372.808 ms oleg216-client.virtnet: 1 19697 0.48 MB/sec execute 169 sec latency 18373.091 ms oleg216-client.virtnet: 1 19697 0.48 MB/sec execute 170 sec latency 19373.680 ms oleg216-client.virtnet: 1 19697 0.48 MB/sec execute 171 sec latency 20374.187 ms oleg216-client.virtnet: 1 19697 0.48 MB/sec execute 172 sec latency 21374.369 ms oleg216-client.virtnet: 1 19775 0.47 MB/sec execute 173 sec latency 21708.395 ms oleg216-client.virtnet: 1 19951 0.47 MB/sec execute 174 sec latency 47.827 ms oleg216-client.virtnet: 1 20173 0.48 MB/sec execute 175 sec latency 42.468 ms oleg216-client.virtnet: 1 20366 0.48 MB/sec execute 176 sec latency 39.603 ms oleg216-client.virtnet: 1 20756 0.50 MB/sec execute 177 sec latency 29.542 ms oleg216-client.virtnet: 1 21122 0.51 MB/sec execute 178 sec latency 50.179 ms oleg216-client.virtnet: 1 21354 0.52 MB/sec execute 179 sec latency 42.353 ms oleg216-client.virtnet: 1 21491 0.52 MB/sec execute 180 sec latency 42.809 ms oleg216-client.virtnet: 1 21631 0.53 MB/sec execute 181 sec latency 38.074 ms oleg216-client.virtnet: 1 21745 0.52 MB/sec execute 182 sec latency 56.418 ms oleg216-client.virtnet: 1 21845 0.52 MB/sec execute 183 sec latency 43.448 ms oleg216-client.virtnet: 1 22044 0.52 MB/sec execute 184 sec latency 34.396 ms oleg216-client.virtnet: 1 22192 0.52 MB/sec execute 185 sec latency 40.806 ms oleg216-client.virtnet: 1 22319 0.52 MB/sec execute 186 sec latency 43.253 ms oleg216-client.virtnet: 1 22495 0.52 MB/sec execute 187 sec latency 32.353 ms oleg216-client.virtnet: 1 22720 0.54 MB/sec execute 188 sec latency 30.055 ms oleg216-client.virtnet: 1 22882 0.53 MB/sec execute 189 sec latency 29.553 ms oleg216-client.virtnet: 1 23021 0.53 MB/sec execute 190 sec latency 36.802 ms oleg216-client.virtnet: 1 23160 0.53 MB/sec execute 191 sec latency 29.942 ms oleg216-client.virtnet: 1 23340 0.53 MB/sec execute 192 sec latency 32.670 ms oleg216-client.virtnet: 1 23517 0.53 MB/sec execute 193 sec latency 28.338 ms oleg216-client.virtnet: 1 23727 0.53 MB/sec execute 194 sec latency 55.757 ms oleg216-client.virtnet: 1 23921 0.53 MB/sec execute 195 sec latency 41.955 ms oleg216-client.virtnet: 1 24305 0.55 MB/sec execute 196 sec latency 32.887 ms oleg216-client.virtnet: 1 24689 0.56 MB/Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0001 01:06:36 (1746075996) targets are mounted 01:06:36 (1746075996) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12968 3592940 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36716 3569752 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49684 7162692 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg216-server Stopping /mnt/lustre-mds1 (opts:) on oleg216-server 01:06:52 (1746076012) shut down facet: mds1 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds1 to oleg216-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 01:07:10 (1746076030) targets are mounted 01:07:10 (1746076030) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec sec execute 197 sec latency 37.540 ms oleg216-client.virtnet: 1 24940 0.57 MB/sec execute 198 sec latency 36.990 ms oleg216-client.virtnet: 1 25095 0.57 MB/sec execute 199 sec latency 35.003 ms oleg216-client.virtnet: 1 25186 0.57 MB/sec execute 200 sec latency 143.001 ms oleg216-client.virtnet: 1 25186 0.57 MB/sec execute 201 sec latency 1143.271 ms oleg216-client.virtnet: 1 25186 0.57 MB/sec execute 202 sec latency 2144.585 ms oleg216-client.virtnet: 1 25186 0.56 MB/sec execute 203 sec latency 3148.012 ms oleg216-client.virtnet: 1 25243 0.56 MB/sec execute 204 sec latency 3741.433 ms oleg216-client.virtnet: 1 25337 0.56 MB/sec execute 205 sec latency 53.354 ms oleg216-client.virtnet: 1 25459 0.56 MB/sec execute 206 sec latency 45.756 ms oleg216-client.virtnet: 1 25622 0.56 MB/sec execute 207 sec latency 37.145 ms oleg216-client.virtnet: 1 25750 0.55 MB/sec execute 208 sec latency 51.915 ms oleg216-client.virtnet: 1 25864 0.55 MB/sec execute 209 sec latency 36.931 ms oleg216-client.virtnet: 1 26027 0.56 MB/sec execute 210 sec latency 35.438 ms oleg216-client.virtnet: 1 26145 0.55 MB/sec execute 211 sec latency 47.226 ms oleg216-client.virtnet: 1 26350 0.56 MB/sec execute 212 sec latency 54.312 ms oleg216-client.virtnet: 1 26477 0.56 MB/sec execute 213 sec latency 42.855 ms oleg216-client.virtnet: 1 26579 0.56 MB/sec execute 214 sec latency 65.430 ms oleg216-client.virtnet: 1 26676 0.56 MB/sec execute 215 sec latency 67.664 ms oleg216-client.virtnet: 1 26715 0.56 MB/sec execute 216 sec latency 611.353 ms oleg216-client.virtnet: 1 26715 0.55 MB/sec execute 217 sec latency 1611.754 ms oleg216-client.virtnet: 1 26715 0.55 MB/sec execute 218 sec latency 2612.096 ms oleg216-client.virtnet: 1 26715 0.55 MB/sec execute 219 sec latency 3612.383 ms oleg216-client.virtnet: 1 26715 0.55 MB/sec execute 220 sec latency 4613.642 ms oleg216-client.virtnet: 1 26715 0.54 MB/sec execute 221 sec latency 5614.053 ms oleg216-client.virtnet: 1 26715 0.54 MB/sec execute 222 sec latency 6616.239 ms oleg216-client.virtnet: 1 26715 0.54 MB/sec execute 223 sec latency 7616.769 ms oleg216-client.virtnet: 1 26715 0.54 MB/sec execute 224 sec latency 8617.096 ms oleg216-client.virtnet: 1 26715 0.53 MB/sec execute 225 sec latency 9619.531 ms oleg216-client.virtnet: 1 26715 0.53 MB/sec execute 226 sec latency 10619.743 ms oleg216-client.virtnet: 1 26715 0.53 MB/sec execute 227 sec latency 11621.405 ms oleg216-client.virtnet: 1 26715 0.53 MB/sec execute 228 sec latency 12622.377 ms oleg216-client.virtnet: 1 26715 0.52 MB/sec execute 229 sec latency 13622.635 ms oleg216-client.virtnet: 1 26715 0.52 MB/sec execute 230 sec latency 14624.085 ms oleg216-client.virtnet: 1 26715 0.52 MB/sec execute 231 sec latency 15625.193 ms oleg216-client.virtnet: 1 26715 0.52 MB/sec execute 232 sec latency 16625.394 ms oleg216-client.virtnet: 1 26715 0.52 MB/sec execute 233 sec latency 17625.546 ms oleg216-client.virtnet: 1 26715 0.51 MB/sec execute 234 sec latency 18626.916 ms oleg216-client.virtnet: 1 26715 0.51 MB/sec execute 235 sec latency 19629.216 ms oleg216-client.virtnet: 1 26715 0.51 MB/sec execute 236 sec latency 20629.521 ms oleg216-client.virtnet: 1 26715 0.51 MB/sec execute 237 sec latency 21630.705 ms oleg216-client.virtnet: 1 26742 0.50 MB/sec execute 238 sec latency 22303.684 ms oleg216-client.virtnet: 1 26877 0.50 MB/sec execute 239 sec latency 40.185 ms oleg216-client.virtnet: 1 26997 0.50 MB/sec execute 240 sec latency 67.979 ms oleg216-client.virtnet: 1 27081 0.50 MB/sec execute 241 sec latency 59.734 ms oleg216-client.virtnet: 1 27249 0.50 MB/sec execute 242 sec latency 46.631 ms oleg216-client.virtnet: 1 27355 0.50 MB/sec execute 243 sec latency 35.999 ms oleg216-client.virtnet: 1 27648 0.52 MB/sec execute 244 sec latency 34.181 ms oleg216-client.virtnet: 1 27895 0.52 MB/sec execute 245 sec latency 34.970 ms oleg216-client.virtnet: 1 28361 0.54 MB/sec execute 246 sec latency 36.135 ms oleg216-client.virtnet: 1 28512 0.54 MB/sec execute 247 sec latency 39.081 ms oleg216-client.virtnet: 1 28664 0.54 MB/sec execute 248 sec latency 47.054 ms oleg216-client.virtnet: 1 28785 0.54 MB/sec execute 249 sec latency 36.599 ms oleg216-client.virtnet: 1 28883 0.54 MB/sec execute 250 sec latency 46.710 ms oleg216-client.virtnet: 1 29019 0.53 MB/sec execute 251 sec latency 65.189 ms oleg216-client.virtnet: 1 29156 0.53 MB/sec execute 252 sec latency 41.385 ms oleg216-client.virtnet: 1 29278 0.53 MB/sec execute 253 sec latency 44.778 ms oleg216-client.virtnet: 1 29393 0.53 MB/sec execute 254 sec latency 34.624 ms oleg216-client.virtnet: 1 29558 0.53 MB/sec execute 255 sec latency 33.612 ms oleg216-client.virtnet: 1 29697 0.53 MB/sec execute 256 sec latency 33.985 ms oleg216-client.virtnet: 1 29910 0.54 MB/sec execute 257 sec latency 52.118 ms oleg216-client.virtnet: 1 30044 0.54 MB/sec execute 258 sec latency 32.741 ms oleg216-client.virtnet: 1 30150 0.54 MB/sec execute 259 sec latency 54.809 ms oleg216-client.virtnet: 1 30271 0.54 MB/sec execute 260 sec latency 43.707 ms oleg216-client.virtnet: 1 30448 0.53 MB/sec execute 261 sec latency 28.592 ms oleg216-client.virtnet: 1 30650 0.53 MB/sec execute 262 sec latency 36.539 ms oleg216-client.virtnet: 1 30842 0.54 MB/sec execute 263 sec latency 32.741 ms oleg216-client.virtnet: 1 31025 0.54 MB/sec execute 264 sec latency 43.029 ms oleg216-client.virtnet: 1 31430 0.55 MB/sec execute 265 sec latency 23.744 ms oleg216-client.virtnet: 1 31921 0.57 MB/sec execute 266 sec latency 35.078 ms oleg216-client.virtnet: 1 32097 0.57 MB/sec execute 267 sec latency 27.452 ms oleg216-client.virtnet: 1 32269 0.57 MB/sec execute 268 sec latency 25.353 ms oleg216-client.virtnet: 1 32416 0.57 MB/sec execute 269 sec latency 30.887 ms oleg216-client.virtnet: 1 32616 0.57 MB/sec execute 270 sec latency 32.833 ms oleg216-client.virtnet: 1 32777 0.57 MB/sec execute 271 sec latency 37.992 ms oleg216-client.virtnet: 1 32960 0.57 MB/sec execute 272 sec latency 28.324 ms oleg216-client.virtnet: 1 33173 0.57 MB/sec execute 273 sec latency 39.063 ms oleg216-client.virtnet: 1 33452 0.58 MB/sec execute 274 sec latency 24.205 ms oleg216-client.virtnet: 1 33630 0.58 MB/sec execute 275 sec latency 33.019 ms oleg216-client.virtnet: 1 33765 0.57 MB/sec execute 276 sec latency 37.688 ms oleg216-client.virtnet: 1 33944 0.57 MB/sec execute 277 sec latency 33.525 ms oleg216-client.virtnet: 1 34108 0.57 MB/sec execute 278 sec latency 36.528 ms oleg216-client.virtnet: 1 34314 0.57 MB/sec execute 279 sec latency 35.068 ms oleg216-client.virtnet: 1 34443 0.57 MB/sec execute 280 sec latency 33.047 ms oleg216-client.virtnet: 1 34744 0.58 MB/sec execute 281 sec latency 35.527 ms oleg216-client.virtnet: 1 35153 0.59 MB/sec execute 282 sec latency 34.221 ms oleg216-client.virtnet: 1 35506 0.60 MB/sec execute 283 sec latency 31.736 ms oleg216-client.virtnet: 1 35711 0.60 MB/sec execute 284 sec latency 27.571 ms oleg216-client.virtnet: 1 35866 0.60 MB/sec execute 285 sec latency 31.775 ms oleg216-client.virtnet: 1 36015 0.60 MB/sec execute 286 sec latency 37.815 ms oleg216-client.virtnet: 1 36225 0.60 MB/sec execute 287 sec latency 47.453 ms oleg216-client.virtnet: 1 36382 0.60 MB/sec execute 288 sec latency 36.488 ms oleg216-client.virtnet: 1 36527 0.60 MB/sec execute 289 sec latency 34.236 ms oleg216-client.virtnet: 1 36710 0.60 MB/sec execute 290 sec latency 43.312 ms oleg216-client.virtnet: 1 36925 0.61 MB/sec execute 291 sec latency 40.767 ms oleg216-client.virtnet: 1 37074 0.61 MB/sec execute 292 sec latency 39.704 ms oleg216-client.virtnet: 1 37212 0.60 MB/sec execute 293 sec latency 38.405 ms oleg216-client.virtnet: 1 37357 0.60 MB/sec execute 294 sec latency 32.386 ms oleg216-client.virtnet: 1 37563 0.60 MB/sec execute 295 sec latency 34.702 ms oleg216-client.virtnet: 1 37711 0.60 MB/sec execute 296 sec latency 44.399 ms oleg216-client.virtnet: 1 37915 0.60 MB/sec execute 297 sec latency 33.792 ms oleg216-client.virtnet: 1 38111 0.60 MB/sec execute 298 sec latency 32.035 ms oleg216-client.virtnet: 1 38497 0.62 MB/sec execute 299 sec latency 30.256 ms oleg216-client.virtnet: 1 cleanup 300 sec oleg216-client.virtnet: 0 cleanup 301 sec oleg216-client.virtnet: oleg216-client.virtnet: Operation Count AvgLat MaxLat oleg216-client.virtnet: ---------------------------------------- oleg216-client.virtnet: NTCreateX 5954 26.324 22303.666 oleg216-client.virtnet: Close 4416 8.850 21708.380 oleg216-client.virtnet: Rename 256 26.394 81.066 oleg216-client.virtnet: Unlink 1195 10.033 4709.947 oleg216-client.virtnet: Qpathinfo 5453 12.235 22551.111 oleg216-client.virtnet: Qfileinfo 964 0.307 17.721 oleg216-client.virtnet: Qfsinfo 992 0.626 31.435 oleg216-client.virtnet: Sfileinfo 500 17.529 50.669 oleg216-client.virtnet: Find 2126 0.923 41.561 oleg216-client.virtnet: WriteX 2993 2.683 43.453 oleg216-client.virtnet: ReadX 9478 0.062 36.162 oleg216-client.virtnet: LockX 20 3.587 9.578 oleg216-client.virtnet: UnlockX 20 3.298 9.659 oleg216-client.virtnet: Flush 430 22.816 884.571 oleg216-client.virtnet: oleg216-client.virtnet: Throughput 0.617205 MB/sec 1 clients 1 procs max_latency=22551.140 ms oleg216-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg216-client.virtnet at Thu May 1 01:08:16 EDT 2025 with return code 0 oleg216-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg216-client.virtnet oleg216-client.virtnet: /mnt/lustre/d70b.replay-single/oleg216-client.virtnet /mnt/lustre/d70b.replay-single/oleg216-client.virtnet oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg216-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg216-client.virtnet: removed directory: 'clients/client0' oleg216-client.virtnet: removed directory: 'clients' oleg216-client.virtnet: removed 'client.txt' oleg216-client.virtnet: /mnt/lustre/d70b.replay-single/oleg216-client.virtnet oleg216-client.virtnet: dbench successfully finished