== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 06:58:16 (1733399896) Starting client oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre Started clients oleg256-client.virtnet: 192.168.202.156@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg256-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=5295 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3840 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3716096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3685376 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 35840 7401472 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 06:58:22 (1733399902) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 06:58:35 (1733399915) targets are mounted 06:58:35 (1733399915) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3840 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3710976 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3674112 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 46080 7385088 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 06:58:46 (1733399926) shut down oleg256-client.virtnet: looking for dbench program oleg256-client.virtnet: /usr/bin/dbench oleg256-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg256-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg256-client.virtnet' oleg256-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg256-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg256-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg256-client.virtnet at Thu Dec 5 06:58:17 EST 2024 oleg256-client.virtnet: waiting for dbench pid 5320 oleg256-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg256-client.virtnet: oleg256-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg256-client.virtnet: failed to create barrier semaphore oleg256-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg256-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg256-client.virtnet: releasing clients oleg256-client.virtnet: 1 661 21.15 MB/sec warmup 1 sec latency 16.687 ms oleg256-client.virtnet: 1 1138 11.50 MB/sec warmup 2 sec latency 26.185 ms oleg256-client.virtnet: 1 1791 8.77 MB/sec warmup 3 sec latency 44.385 ms oleg256-client.virtnet: 1 2428 7.02 MB/sec warmup 4 sec latency 21.343 ms oleg256-client.virtnet: 1 3246 6.95 MB/sec warmup 5 sec latency 201.416 ms oleg256-client.virtnet: 1 3246 5.79 MB/sec warmup 6 sec latency 1201.566 ms oleg256-client.virtnet: 1 3246 4.97 MB/sec warmup 7 sec latency 2201.743 ms oleg256-client.virtnet: 1 3246 4.35 MB/sec warmup 8 sec latency 3201.916 ms oleg256-client.virtnet: 1 3246 3.86 MB/sec warmup 9 sec latency 4202.086 ms oleg256-client.virtnet: 1 3246 3.48 MB/sec warmup 10 sec latency 5202.287 ms oleg256-client.virtnet: 1 3246 3.16 MB/sec warmup 11 sec latency 6202.481 ms oleg256-client.virtnet: 1 3246 2.90 MB/sec warmup 12 sec latency 7202.637 ms oleg256-client.virtnet: 1 3246 2.67 MB/sec warmup 13 sec latency 8202.832 ms oleg256-client.virtnet: 1 3246 2.48 MB/sec warmup 14 sec latency 9203.007 ms oleg256-client.virtnet: 1 3246 2.32 MB/sec warmup 15 sec latency 10203.212 ms oleg256-client.virtnet: 1 3246 2.17 MB/sec warmup 16 sec latency 11203.364 ms oleg256-client.virtnet: 1 3246 2.05 MB/sec warmup 17 sec latency 12203.504 ms oleg256-client.virtnet: 1 3246 1.93 MB/sec warmup 18 sec latency 13203.661 ms oleg256-client.virtnet: 1 3246 1.83 MB/sec warmup 19 sec latency 14203.829 ms oleg256-client.virtnet: 1 3246 1.74 MB/sec warmup 20 sec latency 15203.974 ms oleg256-client.virtnet: 1 3246 1.66 MB/sec warmup 21 sec latency 16204.154 ms oleg256-client.virtnet: 1 3246 1.58 MB/sec warmup 22 sec latency 17204.266 ms oleg256-client.virtnet: 1 3907 1.74 MB/sec warmup 23 sec latency 17346.649 ms oleg256-client.virtnet: 1 4377 1.69 MB/sec warmup 24 sec latency 26.598 ms oleg256-client.virtnet: 1 5023 1.81 MB/sec warmup 25 sec latency 22.105 ms oleg256-client.virtnet: 1 5589 1.76 MB/sec warmup 26 sec latency 22.994 ms oleg256-client.virtnet: 1 6451 1.92 MB/sec warmup 27 sec latency 21.693 ms oleg256-client.virtnet: 1 7345 2.10 MB/sec warmup 28 sec latency 18.689 ms oleg256-client.virtnet: 1 7444 2.04 MB/sec warmup 29 sec latency 785.830 ms oleg256-client.virtnet: 1 7444 1.97 MB/sec warmup 30 sec latency 1786.052 ms oleg256-client.virtnet: 1 7444 1.91 MB/sec warmup 31 sec latency 2786.254 ms oleg256-client.virtnet: 1 7444 1.85 MB/sec warmup 32 sec latency 3786.432Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 06:58:58 (1733399938) targets are mounted 06:58:58 (1733399938) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3719168 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7454720 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 06:59:11 (1733399951) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 06:59:24 (1733399964) targets are mounted 06:59:24 (1733399964) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3739648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3717120 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7456768 1% /mnt/lustre test_70b fail mds1 4 times ms oleg256-client.virtnet: 1 7444 1.79 MB/sec warmup 33 sec latency 4786.720 ms oleg256-client.virtnet: 1 7444 1.74 MB/sec warmup 34 sec latency 5786.966 ms oleg256-client.virtnet: 1 7444 1.69 MB/sec warmup 35 sec latency 6787.139 ms oleg256-client.virtnet: 1 7444 1.64 MB/sec warmup 36 sec latency 7787.303 ms oleg256-client.virtnet: 1 7444 1.60 MB/sec warmup 37 sec latency 8787.482 ms oleg256-client.virtnet: 1 7444 1.56 MB/sec warmup 38 sec latency 9787.758 ms oleg256-client.virtnet: 1 7444 1.52 MB/sec warmup 39 sec latency 10787.919 ms oleg256-client.virtnet: 1 7444 1.48 MB/sec warmup 40 sec latency 11788.048 ms oleg256-client.virtnet: 1 7444 1.44 MB/sec warmup 41 sec latency 12788.194 ms oleg256-client.virtnet: 1 7444 1.41 MB/sec warmup 42 sec latency 13788.334 ms oleg256-client.virtnet: 1 7444 1.38 MB/sec warmup 43 sec latency 14788.499 ms oleg256-client.virtnet: 1 7444 1.34 MB/sec warmup 44 sec latency 15788.688 ms oleg256-client.virtnet: 1 7444 1.31 MB/sec warmup 45 sec latency 16788.876 ms oleg256-client.virtnet: 1 7444 1.29 MB/sec warmup 46 sec latency 17789.048 ms oleg256-client.virtnet: 1 7444 1.26 MB/sec warmup 47 sec latency 18789.199 ms oleg256-client.virtnet: 1 7778 1.24 MB/sec warmup 48 sec latency 19065.485 ms oleg256-client.virtnet: 1 8245 1.25 MB/sec warmup 49 sec latency 31.094 ms oleg256-client.virtnet: 1 8825 1.29 MB/sec warmup 50 sec latency 28.092 ms oleg256-client.virtnet: 1 9497 1.30 MB/sec warmup 51 sec latency 20.650 ms oleg256-client.virtnet: 1 10440 1.42 MB/sec warmup 52 sec latency 20.810 ms oleg256-client.virtnet: 1 11054 1.48 MB/sec warmup 53 sec latency 10.048 ms oleg256-client.virtnet: 1 11335 1.45 MB/sec warmup 54 sec latency 470.605 ms oleg256-client.virtnet: 1 11335 1.43 MB/sec warmup 55 sec latency 1470.852 ms oleg256-client.virtnet: 1 11335 1.40 MB/sec warmup 56 sec latency 2471.036 ms oleg256-client.virtnet: 1 11335 1.38 MB/sec warmup 57 sec latency 3471.217 ms oleg256-client.virtnet: 1 11335 1.35 MB/sec warmup 58 sec latency 4471.452 ms oleg256-client.virtnet: 1 11335 1.33 MB/sec warmup 59 sec latency 5471.616 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 1 sec latency 7471.917 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 2 sec latency 8472.086 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 3 sec latency 9472.295 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 4 sec latency 10472.437 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 5 sec latency 11472.659 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 6 sec latency 12472.853 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 7 sec latency 13473.029 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 8 sec latency 14473.246 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 9 sec latency 15473.455 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 10 sec latency 16473.721 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 11 sec latency 17473.954 ms oleg256-client.virtnet: 1 11335 0.00 MB/sec execute 12 sec latency 18474.158 ms oleg256-client.virtnet: 1 11572 0.05 MB/sec execute 13 sec latency 18823.957 ms oleg256-client.virtnet: 1 12132 0.35 MB/sec execute 14 sec latency 30.480 ms oleg256-client.virtnet: 1 12672 0.36 MB/sec execute 15 sec latency 25.565 ms oleg256-client.virtnet: 1 13416 0.67 MB/sec execute 16 sec latency 23.228 ms oleg256-client.virtnet: 1 14329 1.03 MB/sec execute 17 sec latency 20.715 ms oleg256-client.virtnet: Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 06:59:36 (1733399976) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 06:59:49 (1733399989) targets are mounted 06:59:49 (1733399989) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3716096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3708928 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7425024 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 07:00:01 (1733400001) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 07:00:14 (1733400014) targets are mounted 07:00:14 (1733400014) facet_failover done 1 14816 1.05 MB/sec execute 18 sec latency 8.835 ms oleg256-client.virtnet: 1 15109 1.03 MB/sec execute 19 sec latency 344.924 ms oleg256-client.virtnet: 1 15109 0.98 MB/sec execute 20 sec latency 1345.058 ms oleg256-client.virtnet: 1 15109 0.93 MB/sec execute 21 sec latency 2345.223 ms oleg256-client.virtnet: 1 15109 0.89 MB/sec execute 22 sec latency 3345.396 ms oleg256-client.virtnet: 1 15109 0.85 MB/sec execute 23 sec latency 4345.617 ms oleg256-client.virtnet: 1 15109 0.82 MB/sec execute 24 sec latency 5345.814 ms oleg256-client.virtnet: 1 15109 0.79 MB/sec execute 25 sec latency 6346.017 ms oleg256-client.virtnet: 1 15109 0.75 MB/sec execute 26 sec latency 7346.191 ms oleg256-client.virtnet: 1 15109 0.73 MB/sec execute 27 sec latency 8346.421 ms oleg256-client.virtnet: 1 15109 0.70 MB/sec execute 28 sec latency 9346.636 ms oleg256-client.virtnet: 1 15109 0.68 MB/sec execute 29 sec latency 10346.871 ms oleg256-client.virtnet: 1 15109 0.65 MB/sec execute 30 sec latency 11346.997 ms oleg256-client.virtnet: 1 15109 0.63 MB/sec execute 31 sec latency 12347.148 ms oleg256-client.virtnet: 1 15109 0.61 MB/sec execute 32 sec latency 13347.333 ms oleg256-client.virtnet: 1 15109 0.59 MB/sec execute 33 sec latency 14347.501 ms oleg256-client.virtnet: 1 15109 0.58 MB/sec execute 34 sec latency 15347.706 ms oleg256-client.virtnet: 1 15109 0.56 MB/sec execute 35 sec latency 16347.899 ms oleg256-client.virtnet: 1 15109 0.55 MB/sec execute 36 sec latency 17348.051 ms oleg256-client.virtnet: 1 15109 0.53 MB/sec execute 37 sec latency 18348.246 ms oleg256-client.virtnet: 1 15427 0.55 MB/sec execute 38 sec latency 18790.669 ms oleg256-client.virtnet: 1 16026 0.62 MB/sec execute 39 sec latency 25.849 ms oleg256-client.virtnet: 1 16635 0.65 MB/sec execute 40 sec latency 18.376 ms oleg256-client.virtnet: 1 17489 0.81 MB/sec execute 41 sec latency 28.558 ms oleg256-client.virtnet: 1 18146 0.90 MB/sec execute 42 sec latency 8.928 ms oleg256-client.virtnet: 1 18647 0.90 MB/sec execute 43 sec latency 26.594 ms oleg256-client.virtnet: 1 18964 0.91 MB/sec execute 44 sec latency 460.335 ms oleg256-client.virtnet: 1 18964 0.89 MB/sec execute 45 sec latency 1460.502 ms oleg256-client.virtnet: 1 18964 0.87 MB/sec execute 46 sec latency 2460.794 ms oleg256-client.virtnet: 1 18964 0.85 MB/sec execute 47 sec latency 3461.008 ms oleg256-client.virtnet: 1 18964 0.84 MB/sec execute 48 sec latency 4461.181 ms oleg256-client.virtnet: 1 18964 0.82 MB/sec execute 49 sec latency 5461.434 ms oleg256-client.virtnet: 1 18964 0.80 MB/sec execute 50 sec latency 6461.679 ms oleg256-client.virtnet: 1 18964 0.79 MB/sec execute 51 sec latency 7461.876 ms oleg256-client.virtnet: 1 18964 0.77 MB/sec execute 52 sec latency 8462.070 ms oleg256-client.virtnet: 1 18964 0.76 MB/sec execute 53 sec latency 9462.297 ms oleg256-client.virtnet: 1 18964 0.74 MB/sec execute 54 sec latency 10462.617 ms oleg256-client.virtnet: 1 18964 0.73 MB/sec execute 55 sec latency 11462.777 ms oleg256-client.virtnet: 1 18964 0.72 MB/sec execute 56 sec latency 12462.886 ms oleg256-client.virtnet: 1 18964 0.70 MB/sec execute 57 sec latency 13463.083 ms oleg256-client.virtnet: 1 18964 0.69 MB/sec execute 58 sec latency 14463.309 ms oleg256-client.virtnet: 1 18964 0.68 MB/sec execute 59 sec latency 15463.517 ms oleg256-client.virtnet: 1 18964 0.67 MB/sec execute 60 sec latency 16463.783 ms oleg256-client.virtnet: 1 18964 0.66 MB/sec execute 61 sec latency 17464.028 ms oleg256-client.virtnet: 1oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3730432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3700736 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7431168 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 07:00:26 (1733400026) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 07:00:39 (1733400039) targets are mounted 07:00:39 (1733400039) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3727360 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3704832 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7432192 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 07:00:52 (1733400052) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 18964 0.65 MB/sec execute 62 sec latency 18464.240 ms oleg256-client.virtnet: 1 19338 0.69 MB/sec execute 63 sec latency 18857.264 ms oleg256-client.virtnet: 1 19934 0.68 MB/sec execute 64 sec latency 28.342 ms oleg256-client.virtnet: 1 20719 0.78 MB/sec execute 65 sec latency 25.143 ms oleg256-client.virtnet: 1 21590 0.86 MB/sec execute 66 sec latency 18.156 ms oleg256-client.virtnet: 1 22069 0.86 MB/sec execute 67 sec latency 24.644 ms oleg256-client.virtnet: 1 22573 0.87 MB/sec execute 68 sec latency 18.924 ms oleg256-client.virtnet: 1 22933 0.90 MB/sec execute 69 sec latency 469.761 ms oleg256-client.virtnet: 1 22933 0.89 MB/sec execute 70 sec latency 1469.934 ms oleg256-client.virtnet: 1 22933 0.88 MB/sec execute 71 sec latency 2470.104 ms oleg256-client.virtnet: 1 22933 0.87 MB/sec execute 72 sec latency 3470.289 ms oleg256-client.virtnet: 1 22933 0.85 MB/sec execute 73 sec latency 4470.488 ms oleg256-client.virtnet: 1 22933 0.84 MB/sec execute 74 sec latency 5470.644 ms oleg256-client.virtnet: 1 22933 0.83 MB/sec execute 75 sec latency 6470.815 ms oleg256-client.virtnet: 1 22933 0.82 MB/sec execute 76 sec latency 7470.999 ms oleg256-client.virtnet: 1 22933 0.81 MB/sec execute 77 sec latency 8471.138 ms oleg256-client.virtnet: 1 22933 0.80 MB/sec execute 78 sec latency 9471.302 ms oleg256-client.virtnet: 1 22933 0.79 MB/sec execute 79 sec latency 10471.503 ms oleg256-client.virtnet: 1 22933 0.78 MB/sec execute 80 sec latency 11471.629 ms oleg256-client.virtnet: 1 22933 0.77 MB/sec execute 81 sec latency 12471.730 ms oleg256-client.virtnet: 1 22933 0.76 MB/sec execute 82 sec latency 13471.890 ms oleg256-client.virtnet: 1 22933 0.75 MB/sec execute 83 sec latency 14472.029 ms oleg256-client.virtnet: 1 22933 0.74 MB/sec execute 84 sec latency 15472.176 ms oleg256-client.virtnet: 1 22933 0.73 MB/sec execute 85 sec latency 16472.291 ms oleg256-client.virtnet: 1 22933 0.73 MB/sec execute 86 sec latency 17472.503 ms oleg256-client.virtnet: 1 22933 0.72 MB/sec execute 87 sec latency 18472.733 ms oleg256-client.virtnet: 1 23299 0.71 MB/sec execute 88 sec latency 18847.707 ms oleg256-client.virtnet: 1 23891 0.73 MB/sec execute 89 sec latency 21.452 ms oleg256-client.virtnet: 1 24860 0.83 MB/sec execute 90 sec latency 33.543 ms oleg256-client.virtnet: 1 25285 0.84 MB/sec execute 91 sec latency 12.954 ms oleg256-client.virtnet: 1 25666 0.83 MB/sec execute 92 sec latency 42.370 ms oleg256-client.virtnet: 1 26081 0.84 MB/sec execute 93 sec latency 29.948 ms oleg256-client.virtnet: 1 26551 0.87 MB/sec execute 94 sec latency 172.981 ms oleg256-client.virtnet: 1 26551 0.86 MB/sec execute 95 sec latency 1173.131 ms oleg256-client.virtnet: 1 26551 0.85 MB/sec execute 96 sec latency 2173.311 ms oleg256-client.virtnet: 1 26551 0.84 MB/sec execute 97 sec latency 3173.513 ms oleg256-client.virtnet: 1 26551 0.83 MB/sec execute 98 sec latency 4173.689 ms oleg256-client.virtnet: 1 26551 0.82 MB/sec execute 99 sec latency 5173.910 ms oleg256-client.virtnet: 1 26551 0.82 MB/sec execute 100 sec latency 6174.061 ms oleg256-client.virtnet: 1 26551 0.81 MB/sec execute 101 sec latency 7174.268 ms oleg256-client.virtnet: 1 26551 0.80 MB/sec execute 102 sec latency 8174.478 ms oleg256-client.virtnet: 1 26551 0.79 MB/sec execute 103 sec latency 9174.652 ms oleg256-client.virtnet: 1 26551 0.78 MB/sec execute 104 sec latency 10174.831 ms oleg256-client.virtnet: 1 26551 0.78 MB/sec execute 105 sec latency 11174.976 ms oleg256-client.virtnet: 1 26551 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 07:01:04 (1733400064) targets are mounted 07:01:04 (1733400064) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3713024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3715072 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7428096 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 07:01:16 (1733400076) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 07:01:29 (1733400089) targets are mounted 07:01:29 (1733400089) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3721216 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3712000 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7433216 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 07:01:43 (1733400103) shut down 0.77 MB/sec execute 106 sec latency 12175.117 ms oleg256-client.virtnet: 1 26551 0.76 MB/sec execute 107 sec latency 13175.370 ms oleg256-client.virtnet: 1 26551 0.76 MB/sec execute 108 sec latency 14175.517 ms oleg256-client.virtnet: 1 26551 0.75 MB/sec execute 109 sec latency 15175.703 ms oleg256-client.virtnet: 1 26551 0.74 MB/sec execute 110 sec latency 16175.924 ms oleg256-client.virtnet: 1 26551 0.73 MB/sec execute 111 sec latency 17176.104 ms oleg256-client.virtnet: 1 26551 0.73 MB/sec execute 112 sec latency 18176.354 ms oleg256-client.virtnet: 1 26912 0.73 MB/sec execute 113 sec latency 18509.634 ms oleg256-client.virtnet: 1 27645 0.77 MB/sec execute 114 sec latency 25.132 ms oleg256-client.virtnet: 1 28561 0.82 MB/sec execute 115 sec latency 20.078 ms oleg256-client.virtnet: 1 29093 0.82 MB/sec execute 116 sec latency 8.793 ms oleg256-client.virtnet: 1 29558 0.83 MB/sec execute 117 sec latency 25.286 ms oleg256-client.virtnet: 1 30146 0.85 MB/sec execute 118 sec latency 26.785 ms oleg256-client.virtnet: 1 30311 0.85 MB/sec execute 119 sec latency 627.133 ms oleg256-client.virtnet: 1 30311 0.84 MB/sec execute 120 sec latency 1627.275 ms oleg256-client.virtnet: 1 30311 0.83 MB/sec execute 121 sec latency 2627.443 ms oleg256-client.virtnet: 1 30311 0.83 MB/sec execute 122 sec latency 3627.631 ms oleg256-client.virtnet: 1 30311 0.82 MB/sec execute 123 sec latency 4627.844 ms oleg256-client.virtnet: 1 30311 0.81 MB/sec execute 124 sec latency 5628.039 ms oleg256-client.virtnet: 1 30311 0.81 MB/sec execute 125 sec latency 6628.194 ms oleg256-client.virtnet: 1 30311 0.80 MB/sec execute 126 sec latency 7628.364 ms oleg256-client.virtnet: 1 30311 0.79 MB/sec execute 127 sec latency 8628.591 ms oleg256-client.virtnet: 1 30311 0.79 MB/sec execute 128 sec latency 9628.774 ms oleg256-client.virtnet: 1 30311 0.78 MB/sec execute 129 sec latency 10628.916 ms oleg256-client.virtnet: 1 30311 0.78 MB/sec execute 130 sec latency 11629.088 ms oleg256-client.virtnet: 1 30311 0.77 MB/sec execute 131 sec latency 12629.307 ms oleg256-client.virtnet: 1 30311 0.76 MB/sec execute 132 sec latency 13629.562 ms oleg256-client.virtnet: 1 30311 0.76 MB/sec execute 133 sec latency 14629.846 ms oleg256-client.virtnet: 1 30311 0.75 MB/sec execute 134 sec latency 15630.078 ms oleg256-client.virtnet: 1 30311 0.75 MB/sec execute 135 sec latency 16630.330 ms oleg256-client.virtnet: 1 30311 0.74 MB/sec execute 136 sec latency 17630.556 ms oleg256-client.virtnet: 1 30311 0.74 MB/sec execute 137 sec latency 18630.771 ms oleg256-client.virtnet: 1 30516 0.73 MB/sec execute 138 sec latency 19153.929 ms oleg256-client.virtnet: 1 30887 0.74 MB/sec execute 139 sec latency 41.128 ms oleg256-client.virtnet: 1 31427 0.77 MB/sec execute 140 sec latency 39.530 ms oleg256-client.virtnet: 1 32072 0.80 MB/sec execute 141 sec latency 35.506 ms oleg256-client.virtnet: 1 32424 0.81 MB/sec execute 142 sec latency 11.537 ms oleg256-client.virtnet: 1 32823 0.81 MB/sec execute 143 sec latency 25.745 ms oleg256-client.virtnet: 1 33194 0.81 MB/sec execute 144 sec latency 34.265 ms oleg256-client.virtnet: 1 33550 0.83 MB/sec execute 145 sec latency 325.331 ms oleg256-client.virtnet: 1 33550 0.82 MB/sec execute 146 sec latency 1325.495 ms oleg256-client.virtnet: 1 33550 0.81 MB/sec execute 147 sec latency 2325.644 ms oleg256-client.virtnet: 1 33550 0.81 MB/sec execute 148 sec latency 3325.767 ms oleg256-client.virtnet: 1 33550 0.80 MB/sec execute 149 sec latency 4325.931 ms oleg256-client.virtnet: 1 33550 0.80 MB/Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 07:01:55 (1733400115) targets are mounted 07:01:55 (1733400115) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3731456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3704832 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7436288 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 07:02:07 (1733400127) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 07:02:21 (1733400141) targets are mounted 07:02:21 (1733400141) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3740672 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3719168 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7459840 1% /mnt/lustre test_70b fail mds1 11 times sec execute 150 sec latency 5326.091 ms oleg256-client.virtnet: 1 33550 0.79 MB/sec execute 151 sec latency 6326.253 ms oleg256-client.virtnet: 1 33550 0.79 MB/sec execute 152 sec latency 7326.529 ms oleg256-client.virtnet: 1 33550 0.78 MB/sec execute 153 sec latency 8326.782 ms oleg256-client.virtnet: 1 33550 0.78 MB/sec execute 154 sec latency 9327.006 ms oleg256-client.virtnet: 1 33550 0.77 MB/sec execute 155 sec latency 10327.243 ms oleg256-client.virtnet: 1 33550 0.77 MB/sec execute 156 sec latency 11327.532 ms oleg256-client.virtnet: 1 33550 0.76 MB/sec execute 157 sec latency 12327.664 ms oleg256-client.virtnet: 1 33550 0.76 MB/sec execute 158 sec latency 13327.766 ms oleg256-client.virtnet: 1 33550 0.75 MB/sec execute 159 sec latency 14327.934 ms oleg256-client.virtnet: 1 33550 0.75 MB/sec execute 160 sec latency 15328.104 ms oleg256-client.virtnet: 1 33550 0.74 MB/sec execute 161 sec latency 16328.282 ms oleg256-client.virtnet: 1 33550 0.74 MB/sec execute 162 sec latency 17328.500 ms oleg256-client.virtnet: 1 33747 0.74 MB/sec execute 163 sec latency 17855.845 ms oleg256-client.virtnet: 1 34192 0.73 MB/sec execute 164 sec latency 32.605 ms oleg256-client.virtnet: 1 34738 0.76 MB/sec execute 165 sec latency 37.433 ms oleg256-client.virtnet: 1 35393 0.78 MB/sec execute 166 sec latency 34.142 ms oleg256-client.virtnet: 1 35787 0.80 MB/sec execute 167 sec latency 15.977 ms oleg256-client.virtnet: 1 36056 0.80 MB/sec execute 168 sec latency 14.291 ms oleg256-client.virtnet: 1 36330 0.79 MB/sec execute 169 sec latency 47.731 ms oleg256-client.virtnet: 1 36487 0.79 MB/sec execute 170 sec latency 483.455 ms oleg256-client.virtnet: 1 36487 0.79 MB/sec execute 171 sec latency 1483.685 ms oleg256-client.virtnet: 1 36487 0.78 MB/sec execute 172 sec latency 2483.982 ms oleg256-client.virtnet: 1 36487 0.78 MB/sec execute 173 sec latency 3484.317 ms oleg256-client.virtnet: 1 36487 0.77 MB/sec execute 174 sec latency 4484.579 ms oleg256-client.virtnet: 1 36487 0.77 MB/sec execute 175 sec latency 5484.886 ms oleg256-client.virtnet: 1 36487 0.76 MB/sec execute 176 sec latency 6485.144 ms oleg256-client.virtnet: 1 36487 0.76 MB/sec execute 177 sec latency 7485.352 ms oleg256-client.virtnet: 1 36487 0.76 MB/sec execute 178 sec latency 8485.588 ms oleg256-client.virtnet: 1 36487 0.75 MB/sec execute 179 sec latency 9485.817 ms oleg256-client.virtnet: 1 36487 0.75 MB/sec execute 180 sec latency 10486.092 ms oleg256-client.virtnet: 1 36487 0.74 MB/sec execute 181 sec latency 11486.341 ms oleg256-client.virtnet: 1 36487 0.74 MB/sec execute 182 sec latency 12486.617 ms oleg256-client.virtnet: 1 36487 0.74 MB/sec execute 183 sec latency 13486.882 ms oleg256-client.virtnet: 1 36487 0.73 MB/sec execute 184 sec latency 14487.107 ms oleg256-client.virtnet: 1 36487 0.73 MB/sec execute 185 sec latency 15487.332 ms oleg256-client.virtnet: 1 36487 0.72 MB/sec execute 186 sec latency 16487.528 ms oleg256-client.virtnet: 1 36487 0.72 MB/sec execute 187 sec latency 17488.835 ms oleg256-client.virtnet: 1 36619 0.72 MB/sec execute 188 sec latency 18116.709 ms oleg256-client.virtnet: 1 37094 0.73 MB/sec execute 189 sec latency 38.455 ms oleg256-client.virtnet: 1 37419 0.73 MB/sec execute 190 sec latency 35.726 ms oleg256-client.virtnet: 1 37868 0.74 MB/sec execute 191 sec latency 41.452 ms oleg256-client.virtnet: 1 38288 0.75 MB/sec execute 192 sec latency 43.348 ms oleg256-client.virtnet: 1 39003 0.78 MB/sec execute 193 sec latency 47.147 ms oleg256-client.virtnet: 1 39360 0.79 MB/sec execFailing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 07:02:33 (1733400153) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 07:02:46 (1733400166) targets are mounted 07:02:46 (1733400166) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3724288 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7475200 1% /mnt/lustre test_70b fail mds1 12 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 07:02:57 (1733400177) shut down Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 07:03:10 (1733400190) targets are mounted 07:03:10 (1733400190) facet_failover done ute 194 sec latency 14.332 ms oleg256-client.virtnet: 1 39661 0.78 MB/sec execute 195 sec latency 188.427 ms oleg256-client.virtnet: 1 39661 0.78 MB/sec execute 196 sec latency 1188.531 ms oleg256-client.virtnet: 1 39661 0.78 MB/sec execute 197 sec latency 2188.629 ms oleg256-client.virtnet: 1 39661 0.77 MB/sec execute 198 sec latency 3188.810 ms oleg256-client.virtnet: 1 39661 0.77 MB/sec execute 199 sec latency 4188.940 ms oleg256-client.virtnet: 1 39661 0.76 MB/sec execute 200 sec latency 5189.075 ms oleg256-client.virtnet: 1 39661 0.76 MB/sec execute 201 sec latency 6189.201 ms oleg256-client.virtnet: 1 39661 0.76 MB/sec execute 202 sec latency 7189.322 ms oleg256-client.virtnet: 1 39661 0.75 MB/sec execute 203 sec latency 8189.452 ms oleg256-client.virtnet: 1 39661 0.75 MB/sec execute 204 sec latency 9189.581 ms oleg256-client.virtnet: 1 39661 0.75 MB/sec execute 205 sec latency 10189.716 ms oleg256-client.virtnet: 1 39661 0.74 MB/sec execute 206 sec latency 11189.823 ms oleg256-client.virtnet: 1 39661 0.74 MB/sec execute 207 sec latency 12189.914 ms oleg256-client.virtnet: 1 39661 0.73 MB/sec execute 208 sec latency 13190.030 ms oleg256-client.virtnet: 1 39661 0.73 MB/sec execute 209 sec latency 14190.142 ms oleg256-client.virtnet: 1 39661 0.73 MB/sec execute 210 sec latency 15190.224 ms oleg256-client.virtnet: 1 39661 0.72 MB/sec execute 211 sec latency 16190.359 ms oleg256-client.virtnet: 1 39661 0.72 MB/sec execute 212 sec latency 17190.595 ms oleg256-client.virtnet: 1 39744 0.72 MB/sec execute 213 sec latency 17938.203 ms oleg256-client.virtnet: 1 40045 0.72 MB/sec execute 214 sec latency 55.224 ms oleg256-client.virtnet: 1 40516 0.73 MB/sec execute 215 sec latency 28.984 ms oleg256-client.virtnet: 1 40861 0.73 MB/sec execute 216 sec latency 35.587 ms oleg256-client.virtnet: 1 41331 0.73 MB/sec execute 217 sec latency 38.429 ms oleg256-client.virtnet: 1 41950 0.76 MB/sec execute 218 sec latency 32.684 ms oleg256-client.virtnet: 1 42598 0.78 MB/sec execute 219 sec latency 33.661 ms oleg256-client.virtnet: 1 42843 0.78 MB/sec execute 220 sec latency 573.919 ms oleg256-client.virtnet: 1 42843 0.78 MB/sec execute 221 sec latency 1574.106 ms oleg256-client.virtnet: 1 42843 0.77 MB/sec execute 222 sec latency 2574.287 ms oleg256-client.virtnet: 1 42843 0.77 MB/sec execute 223 sec latency 3574.425 ms oleg256-client.virtnet: 1 42843 0.77 MB/sec execute 224 sec latency 4574.576 ms oleg256-client.virtnet: 1 42843 0.76 MB/sec execute 225 sec latency 5574.781 ms oleg256-client.virtnet: 1 42843 0.76 MB/sec execute 226 sec latency 6575.078 ms oleg256-client.virtnet: 1 42843 0.76 MB/sec execute 227 sec latency 7575.220 ms oleg256-client.virtnet: 1 42843 0.75 MB/sec execute 228 sec latency 8575.393 ms oleg256-client.virtnet: 1 42843 0.75 MB/sec execute 229 sec latency 9575.578 ms oleg256-client.virtnet: 1 42843 0.75 MB/sec execute 230 sec latency 10575.764 ms oleg256-client.virtnet: 1 42843 0.74 MB/sec execute 231 sec latency 11575.965 ms oleg256-client.virtnet: 1 42843 0.74 MB/sec execute 232 sec latency 12576.145 ms oleg256-client.virtnet: 1 42843 0.74 MB/sec execute 233 sec latency 13576.392 ms oleg256-client.virtnet: 1 42843 0.73 MB/sec execute 234 sec latency 14576.531 ms oleg256-client.virtnet: 1 42843 0.73 MB/sec execute 235 sec latency 15576.701 ms oleg256-client.virtnet: 1 42843 0.73 MB/sec execute 236 sec latency 16576.863 ms oleg256-client.virtnet: 1 42843 0.72 MB/sec execute 237 sec latency 17577.038 ms oleg256-client.virtnet: 1 42945 0.72 MB/sec executeoleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 238 sec latency 18276.112 ms oleg256-client.virtnet: 1 43286 0.72 MB/sec execute 239 sec latency 49.123 ms oleg256-client.virtnet: 1 43587 0.72 MB/sec execute 240 sec latency 32.515 ms oleg256-client.virtnet: 1 44032 0.73 MB/sec execute 241 sec latency 39.187 ms oleg256-client.virtnet: 1 44316 0.73 MB/sec execute 242 sec latency 32.710 ms oleg256-client.virtnet: 1 44634 0.73 MB/sec execute 243 sec latency 33.438 ms oleg256-client.virtnet: 1 45025 0.73 MB/sec execute 244 sec latency 34.415 ms oleg256-client.virtnet: 1 45534 0.75 MB/sec execute 245 sec latency 35.954 ms oleg256-client.virtnet: 1 46135 0.77 MB/sec execute 246 sec latency 42.749 ms oleg256-client.virtnet: 1 46440 0.77 MB/sec execute 247 sec latency 15.373 ms oleg256-client.virtnet: 1 46705 0.77 MB/sec execute 248 sec latency 15.000 ms oleg256-client.virtnet: 1 46969 0.77 MB/sec execute 249 sec latency 61.197 ms oleg256-client.virtnet: 1 47273 0.77 MB/sec execute 250 sec latency 39.531 ms oleg256-client.virtnet: 1 47657 0.78 MB/sec execute 251 sec latency 34.297 ms oleg256-client.virtnet: 1 48001 0.78 MB/sec execute 252 sec latency 33.786 ms oleg256-client.virtnet: 1 48443 0.78 MB/sec execute 253 sec latency 22.528 ms oleg256-client.virtnet: 1 48920 0.80 MB/sec execute 254 sec latency 33.036 ms oleg256-client.virtnet: 1 49672 0.82 MB/sec execute 255 sec latency 29.922 ms oleg256-client.virtnet: 1 50028 0.82 MB/sec execute 256 sec latency 14.025 ms oleg256-client.virtnet: 1 50295 0.82 MB/sec execute 257 sec latency 15.531 ms oleg256-client.virtnet: 1 50562 0.82 MB/sec execute 258 sec latency 38.427 ms oleg256-client.virtnet: 1 50844 0.82 MB/sec execute 259 sec latency 39.638 ms oleg256-client.virtnet: 1 51301 0.83 MB/sec execute 260 sec latency 27.540 ms oleg256-client.virtnet: 1 51740 0.83 MB/sec execute 261 sec latency 32.868 ms oleg256-client.virtnet: 1 52182 0.83 MB/sec execute 262 sec latency 30.056 ms oleg256-client.virtnet: 1 52884 0.85 MB/sec execute 263 sec latency 32.183 ms oleg256-client.virtnet: 1 53374 0.86 MB/sec execute 264 sec latency 19.991 ms oleg256-client.virtnet: 1 53684 0.87 MB/sec execute 265 sec latency 13.837 ms oleg256-client.virtnet: 1 54026 0.86 MB/sec execute 266 sec latency 37.609 ms oleg256-client.virtnet: 1 54340 0.87 MB/sec execute 267 sec latency 33.680 ms oleg256-client.virtnet: 1 54785 0.87 MB/sec execute 268 sec latency 36.197 ms oleg256-client.virtnet: 1 55090 0.87 MB/sec execute 269 sec latency 36.729 ms oleg256-client.virtnet: 1 55502 0.87 MB/sec execute 270 sec latency 27.225 ms oleg256-client.virtnet: 1 55845 0.88 MB/sec execute 271 sec latency 36.209 ms oleg256-client.virtnet: 1 56599 0.90 MB/sec execute 272 sec latency 33.458 ms oleg256-client.virtnet: 1 57064 0.91 MB/sec execute 273 sec latency 12.878 ms oleg256-client.virtnet: 1 57442 0.91 MB/sec execute 274 sec latency 11.980 ms oleg256-client.virtnet: 1 57726 0.91 MB/sec execute 275 sec latency 54.396 ms oleg256-client.virtnet: 1 58268 0.92 MB/sec execute 276 sec latency 26.777 ms oleg256-client.virtnet: 1 58651 0.92 MB/sec execute 277 sec latency 33.607 ms oleg256-client.virtnet: 1 59154 0.92 MB/sec execute 278 sec latency 30.126 ms oleg256-client.virtnet: 1 59755 0.94 MB/sec execute 279 sec latency 30.535 ms oleg256-client.virtnet: 1 60483 0.95 MB/sec execute 280 sec latency 20.914 ms oleg256-client.virtnet: 1 60847 0.95 MB/sec execute 281 sec latency 12.694 ms oleg256-client.virtnet: 1 61213 0.95 MB/sec execute 282 sec latency 35.151 ms oleg256-client.virtnet: 1 61621 0.95 MB/sec execute 283 sec latency 26.610 ms oleg256-client.virtnet: 1 62105 0.96 MB/sec execute 284 sec latency 25.920 ms oleg256-client.virtnet: 1 62495 0.96 MB/sec execute 285 sec latency 28.516 ms oleg256-client.virtnet: 1 62883 0.96 MB/sec execute 286 sec latency 41.565 ms oleg256-client.virtnet: 1 63516 0.98 MB/sec execute 287 sec latency 38.940 ms oleg256-client.virtnet: 1 64022 0.99 MB/sec execute 288 sec latency 31.125 ms oleg256-client.virtnet: 1 64320 0.99 MB/sec execute 289 sec latency 15.405 ms oleg256-client.virtnet: 1 64614 0.99 MB/sec execute 290 sec latency 48.559 ms oleg256-client.virtnet: 1 64862 0.99 MB/sec execute 291 sec latency 38.245 ms oleg256-client.virtnet: 1 65373 1.00 MB/sec execute 292 sec latency 37.626 ms oleg256-client.virtnet: 1 65684 1.00 MB/sec execute 293 sec latency 31.443 ms oleg256-client.virtnet: 1 66105 0.99 MB/sec execute 294 sec latency 15.087 ms oleg256-client.virtnet: 1 66650 1.01 MB/sec execute 295 sec latency 33.281 ms oleg256-client.virtnet: 1 67399 1.03 MB/sec execute 296 sec latency 32.292 ms oleg256-client.virtnet: 1 67741 1.03 MB/sec execute 297 sec latency 12.864 ms oleg256-client.virtnet: 1 68075 1.03 MB/sec execute 298 sec latency 14.214 ms oleg256-client.virtnet: 1 68333 1.02 MB/sec execute 299 sec latency 44.297 ms oleg256-client.virtnet: 1 cleanup 300 sec oleg256-client.virtnet: 0 cleanup 300 sec oleg256-client.virtnet: oleg256-client.virtnet: Operation Count AvgLat MaxLat oleg256-client.virtnet: ---------------------------------------- oleg256-client.virtnet: NTCreateX 9916 11.782 18790.656 oleg256-client.virtnet: Close 7301 3.812 18509.617 oleg256-client.virtnet: Rename 425 52.628 18116.685 oleg256-client.virtnet: Unlink 1997 11.314 17938.182 oleg256-client.virtnet: Qpathinfo 8978 5.977 18857.245 oleg256-client.virtnet: Qfileinfo 1565 0.147 2.289 oleg256-client.virtnet: Qfsinfo 1654 0.117 2.524 oleg256-client.virtnet: Sfileinfo 828 28.352 18823.943 oleg256-client.virtnet: Find 3471 0.468 11.794 oleg256-client.virtnet: WriteX 4919 0.858 7.515 oleg256-client.virtnet: ReadX 15511 0.026 6.039 oleg256-client.virtnet: LockX 32 1.172 1.970 oleg256-client.virtnet: UnlockX 32 1.274 1.995 oleg256-client.virtnet: Flush 707 46.073 19153.897 oleg256-client.virtnet: oleg256-client.virtnet: Throughput 1.02491 MB/sec 1 clients 1 procs max_latency=19153.929 ms oleg256-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg256-client.virtnet at Thu Dec 5 07:04:17 EST 2024 with return code 0 oleg256-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: /mnt/lustre/d70b.replay-single/oleg256-client.virtnet /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg256-client.virtnet: removed directory: 'clients/client0' oleg256-client.virtnet: removed directory: 'clients' oleg256-client.virtnet: removed 'client.txt' oleg256-client.virtnet: /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: dbench successfully finished