== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 19:07:33 (1746140853) Starting client oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre Started clients oleg337-client.virtnet: 192.168.203.137@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg337-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=6387 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 28672 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3724288 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 34816 7459840 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:07:43 (1746140863) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:07:58 (1746140878) targets are mounted 19:07:58 (1746140878) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210048 3840 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 32768 3686400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 11264 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7424000 1% /mnt/lustre oleg337-client.virtnet: looking for dbench program oleg337-client.virtnet: /usr/bin/dbench oleg337-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg337-client.virtnet oleg337-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg337-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg337-client.virtnet' oleg337-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg337-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg337-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg337-client.virtnet at Thu May 1 19:07:35 EDT 2025 oleg337-client.virtnet: waiting for dbench pid 6412 oleg337-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg337-client.virtnet: oleg337-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg337-client.virtnet: failed to create barrier semaphore oleg337-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg337-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg337-client.virtnet: releasing clients oleg337-client.virtnet: 1 149 6.27 MB/sec warmup 1 sec latency 78.222 ms oleg337-client.virtnet: 1 313 5.72 MB/sec warmup 2 sec latency 46.262 ms oleg337-client.virtnet: 1 429 5.03 MB/sec warmup 3 sec latency 73.425 ms oleg337-client.virtnet: 1 645 5.25 MB/sec warmup 4 sec latency 53.440 ms oleg337-client.virtnet: 1 730 4.25 MB/sec warmup 5 sec latency 159.040 ms oleg337-client.virtnet: 1 850 3.59 MB/sec warmup 6 sec latency 75.312 ms oleg337-client.virtnet: 1 923 3.10 MB/sec warmup 7 sec latency 144.892 ms oleg337-client.virtnet: 1 959 2.72 MB/sec warmup 8 sec latency 724.280 ms oleg337-client.virtnet: 1 959 2.42 MB/sec warmup 9 sec latency 1724.524 ms oleg337-client.virtnet: 1 959 2.18 MB/sec warmup 10 sec latency 2724.804 ms oleg337-client.virtnet: 1 959 1.98 MB/sec warmup 11 sec latency 3726.874 ms oleg337-client.virtnet: 1 959 1.82 MB/sec warmup 12 sec latency 4727.090 ms oleg337-client.virtnet: 1 959 1.68 MB/sec warmup 13 sec latency 5727.952 ms oleg337-client.virtnet: 1 959 1.56 MB/sec warmup 14 sec latency 6729.150 ms oleg337-client.virtnet: 1 959 1.45 MB/sec warmup 15 sec latency 7729.927 ms oleg337-client.virtnet: 1 959 1.36 MB/sec warmup 16 sec latency 8730.909 ms oleg337-client.virtnet: 1 959 1.28 MB/sec warmup 17 sec latency 9731.057 ms oleg337-client.virtnet: 1 959 1.21 MB/sec warmup 18 sec latency 10732.237 ms oleg337-client.virtnet: 1 959 1.15 MB/sec warmup 19 sec latency 11733.746 ms oleg337-client.virtnet: 1 959 1.09 MB/sec warmup 20 sec latency 12733.911 ms oleg337-client.virtnet: 1 959 1.04 MB/sec warmup 21 sec latency 13734.118 ms oleg337-client.virtnet: 1 959 0.99 MB/sec warmup 22 sec latency 14734.427 ms oleg337-client.virtnet: 1 959 0.95 MB/sec warmup 23 sec latency 15734.598 ms oleg337-client.virtnet: 1 959 0.91 MB/sec warmup 24 sec latency 16735.535 ms oleg337-client.virtnet: 1 959 0.87 MB/sec warmup 25 sec latency 17735.692 ms oleg337-client.virtnet: 1 959 0.84 MB/sec warmup 26 sec latency 18736.578 ms oleg337-client.virtnet: 1 1029 0.85 MB/sec warmup 27 sec latency 19092.239 ms oleg337-client.virtnet: 1 1185 0.82 MB/sec warmup 28 sec latency 39.693 ms oleg337-client.virtnet: 1 1421 0.90 MB/sec warmup 29 sec latency 52.132 ms oleg337-client.virtnet: 1 1557 0.87 MB/sec warmup 30 sec latency 98.542 ms oleg337-client.virtnet: 1 1696 0.85 MB/sec warmup 31 sec latency 106.119 ms oleg337-client.virtnet: 1 1857 0.82 MB/sec warmup 32 sec latency 51.117 test_70b fail mds1 2 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:08:12 (1746140892) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:08:27 (1746140907) targets are mounted 19:08:27 (1746140907) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3719168 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3722240 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7441408 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:08:42 (1746140922) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 ms oleg337-client.virtnet: 1 1995 0.80 MB/sec warmup 33 sec latency 119.378 ms oleg337-client.virtnet: 1 2166 0.79 MB/sec warmup 34 sec latency 43.152 ms oleg337-client.virtnet: 1 2356 0.80 MB/sec warmup 35 sec latency 104.126 ms oleg337-client.virtnet: 1 2455 0.78 MB/sec warmup 36 sec latency 97.213 ms oleg337-client.virtnet: 1 2477 0.78 MB/sec warmup 37 sec latency 919.813 ms oleg337-client.virtnet: 1 2477 0.75 MB/sec warmup 38 sec latency 1920.376 ms oleg337-client.virtnet: 1 2477 0.74 MB/sec warmup 39 sec latency 2920.546 ms oleg337-client.virtnet: 1 2477 0.72 MB/sec warmup 40 sec latency 3921.275 ms oleg337-client.virtnet: 1 2477 0.70 MB/sec warmup 41 sec latency 4921.613 ms oleg337-client.virtnet: 1 2477 0.68 MB/sec warmup 42 sec latency 5921.832 ms oleg337-client.virtnet: 1 2477 0.67 MB/sec warmup 43 sec latency 6921.991 ms oleg337-client.virtnet: 1 2477 0.65 MB/sec warmup 44 sec latency 7922.150 ms oleg337-client.virtnet: 1 2477 0.64 MB/sec warmup 45 sec latency 8923.672 ms oleg337-client.virtnet: 1 2477 0.62 MB/sec warmup 46 sec latency 9924.000 ms oleg337-client.virtnet: 1 2477 0.61 MB/sec warmup 47 sec latency 10924.142 ms oleg337-client.virtnet: 1 2477 0.60 MB/sec warmup 48 sec latency 11924.378 ms oleg337-client.virtnet: 1 2477 0.59 MB/sec warmup 49 sec latency 12924.745 ms oleg337-client.virtnet: 1 2477 0.57 MB/sec warmup 50 sec latency 13925.448 ms oleg337-client.virtnet: 1 2477 0.56 MB/sec warmup 51 sec latency 14926.515 ms oleg337-client.virtnet: 1 2477 0.55 MB/sec warmup 52 sec latency 15926.830 ms oleg337-client.virtnet: 1 2477 0.54 MB/sec warmup 53 sec latency 16928.708 ms oleg337-client.virtnet: 1 2477 0.53 MB/sec warmup 54 sec latency 17928.943 ms oleg337-client.virtnet: 1 2477 0.52 MB/sec warmup 55 sec latency 18929.122 ms oleg337-client.virtnet: 1 2477 0.51 MB/sec warmup 56 sec latency 19930.822 ms oleg337-client.virtnet: 1 2572 0.51 MB/sec warmup 57 sec latency 20403.844 ms oleg337-client.virtnet: 1 2897 0.57 MB/sec warmup 58 sec latency 97.508 ms oleg337-client.virtnet: 1 3189 0.58 MB/sec warmup 59 sec latency 117.196 ms oleg337-client.virtnet: 1 3781 1.37 MB/sec execute 1 sec latency 67.459 ms oleg337-client.virtnet: 1 3951 0.84 MB/sec execute 2 sec latency 27.189 ms oleg337-client.virtnet: 1 4041 0.56 MB/sec execute 3 sec latency 48.936 ms oleg337-client.virtnet: 1 4236 0.47 MB/sec execute 4 sec latency 33.306 ms oleg337-client.virtnet: 1 4343 0.42 MB/sec execute 5 sec latency 119.922 ms oleg337-client.virtnet: 1 4444 0.40 MB/sec execute 6 sec latency 296.848 ms oleg337-client.virtnet: 1 4444 0.34 MB/sec execute 7 sec latency 1296.993 ms oleg337-client.virtnet: 1 4444 0.30 MB/sec execute 8 sec latency 2297.152 ms oleg337-client.virtnet: 1 4444 0.27 MB/sec execute 9 sec latency 3298.855 ms oleg337-client.virtnet: 1 4444 0.24 MB/sec execute 10 sec latency 4299.016 ms oleg337-client.virtnet: 1 4444 0.22 MB/sec execute 11 sec latency 5299.414 ms oleg337-client.virtnet: 1 4444 0.20 MB/sec execute 12 sec latency 6304.131 ms oleg337-client.virtnet: 1 4444 0.18 MB/sec execute 13 sec latency 7304.297 ms oleg337-client.virtnet: 1 4444 0.17 MB/sec execute 14 sec latency 8305.207 ms oleg337-client.virtnet: 1 4444 0.16 MB/sec execute 15 sec latency 9305.447 ms oleg337-client.virtnet: 1 4444 0.15 MB/sec execute 16 sec latency 10306.057 ms oleg337-client.virtnet: 1 4444 0.14 MB/sec execute 17 sec latency 11306.216 ms oleg337-client.virtnet: 1 4Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:08:56 (1746140936) targets are mounted 19:08:56 (1746140936) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3721216 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3740672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7461888 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:09:08 (1746140948) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:09:22 (1746140962) targets are mounted 19:09:22 (1746140962) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3709952 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3729408 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7439360 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:09:37 (1746140977) shut down 444 0.13 MB/sec execute 18 sec latency 12306.387 ms oleg337-client.virtnet: 1 4444 0.13 MB/sec execute 19 sec latency 13306.589 ms oleg337-client.virtnet: 1 4444 0.12 MB/sec execute 20 sec latency 14308.311 ms oleg337-client.virtnet: 1 4444 0.11 MB/sec execute 21 sec latency 15308.450 ms oleg337-client.virtnet: 1 4484 0.11 MB/sec execute 22 sec latency 15907.992 ms oleg337-client.virtnet: 1 4610 0.16 MB/sec execute 23 sec latency 136.246 ms oleg337-client.virtnet: 1 4787 0.16 MB/sec execute 24 sec latency 32.402 ms oleg337-client.virtnet: 1 5012 0.27 MB/sec execute 25 sec latency 41.123 ms oleg337-client.virtnet: 1 5170 0.26 MB/sec execute 26 sec latency 63.834 ms oleg337-client.virtnet: 1 5317 0.26 MB/sec execute 27 sec latency 58.451 ms oleg337-client.virtnet: 1 5449 0.25 MB/sec execute 28 sec latency 103.354 ms oleg337-client.virtnet: 1 5596 0.25 MB/sec execute 29 sec latency 111.505 ms oleg337-client.virtnet: 1 5759 0.25 MB/sec execute 30 sec latency 38.948 ms oleg337-client.virtnet: 1 5960 0.28 MB/sec execute 31 sec latency 92.927 ms oleg337-client.virtnet: 1 6135 0.30 MB/sec execute 32 sec latency 92.027 ms oleg337-client.virtnet: 1 6140 0.29 MB/sec execute 33 sec latency 994.432 ms oleg337-client.virtnet: 1 6140 0.28 MB/sec execute 34 sec latency 1995.216 ms oleg337-client.virtnet: 1 6140 0.28 MB/sec execute 35 sec latency 2996.027 ms oleg337-client.virtnet: 1 6140 0.27 MB/sec execute 36 sec latency 3996.221 ms oleg337-client.virtnet: 1 6140 0.26 MB/sec execute 37 sec latency 4996.470 ms oleg337-client.virtnet: 1 6140 0.25 MB/sec execute 38 sec latency 5997.503 ms oleg337-client.virtnet: 1 6140 0.25 MB/sec execute 39 sec latency 6997.834 ms oleg337-client.virtnet: 1 6140 0.24 MB/sec execute 40 sec latency 7998.215 ms oleg337-client.virtnet: 1 6140 0.24 MB/sec execute 41 sec latency 8998.590 ms oleg337-client.virtnet: 1 6140 0.23 MB/sec execute 42 sec latency 9998.841 ms oleg337-client.virtnet: 1 6140 0.23 MB/sec execute 43 sec latency 10999.576 ms oleg337-client.virtnet: 1 6140 0.22 MB/sec execute 44 sec latency 12000.614 ms oleg337-client.virtnet: 1 6140 0.22 MB/sec execute 45 sec latency 13001.968 ms oleg337-client.virtnet: 1 6140 0.21 MB/sec execute 46 sec latency 14002.246 ms oleg337-client.virtnet: 1 6140 0.21 MB/sec execute 47 sec latency 15002.362 ms oleg337-client.virtnet: 1 6140 0.20 MB/sec execute 48 sec latency 16002.771 ms oleg337-client.virtnet: 1 6140 0.20 MB/sec execute 49 sec latency 17003.829 ms oleg337-client.virtnet: 1 6140 0.19 MB/sec execute 50 sec latency 18003.993 ms oleg337-client.virtnet: 1 6140 0.19 MB/sec execute 51 sec latency 19004.190 ms oleg337-client.virtnet: 1 6213 0.21 MB/sec execute 52 sec latency 19547.246 ms oleg337-client.virtnet: 1 6485 0.27 MB/sec execute 53 sec latency 156.963 ms oleg337-client.virtnet: 1 6575 0.26 MB/sec execute 54 sec latency 182.022 ms oleg337-client.virtnet: 1 6906 0.29 MB/sec execute 55 sec latency 128.038 ms oleg337-client.virtnet: 1 7209 0.35 MB/sec execute 56 sec latency 28.722 ms oleg337-client.virtnet: 1 7396 0.36 MB/sec execute 57 sec latency 27.212 ms oleg337-client.virtnet: 1 7549 0.36 MB/sec execute 58 sec latency 28.525 ms oleg337-client.virtnet: 1 7753 0.35 MB/sec execute 59 sec latency 29.811 ms oleg337-client.virtnet: 1 7844 0.35 MB/sec execute 60 sec latency 87.179 ms oleg337-client.virtnet: 1 7915 0.35 MB/sec execute 61 sec latency 175.773 ms oleg337-client.virtnet: 1 7915 0.34 MB/sec facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:09:52 (1746140992) targets are mounted 19:09:52 (1746140992) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3712000 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7458816 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:10:10 (1746141010) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 execute 62 sec latency 1176.524 ms oleg337-client.virtnet: 1 7915 0.34 MB/sec execute 63 sec latency 2176.750 ms oleg337-client.virtnet: 1 7915 0.33 MB/sec execute 64 sec latency 3177.285 ms oleg337-client.virtnet: 1 7915 0.33 MB/sec execute 65 sec latency 4178.542 ms oleg337-client.virtnet: 1 7915 0.32 MB/sec execute 66 sec latency 5179.148 ms oleg337-client.virtnet: 1 7915 0.32 MB/sec execute 67 sec latency 6179.319 ms oleg337-client.virtnet: 1 7915 0.31 MB/sec execute 68 sec latency 7179.719 ms oleg337-client.virtnet: 1 7915 0.31 MB/sec execute 69 sec latency 8179.917 ms oleg337-client.virtnet: 1 7915 0.30 MB/sec execute 70 sec latency 9180.088 ms oleg337-client.virtnet: 1 7915 0.30 MB/sec execute 71 sec latency 10180.246 ms oleg337-client.virtnet: 1 7915 0.30 MB/sec execute 72 sec latency 11180.615 ms oleg337-client.virtnet: 1 7915 0.29 MB/sec execute 73 sec latency 12180.815 ms oleg337-client.virtnet: 1 7915 0.29 MB/sec execute 74 sec latency 13183.770 ms oleg337-client.virtnet: 1 7915 0.28 MB/sec execute 75 sec latency 14184.008 ms oleg337-client.virtnet: 1 7915 0.28 MB/sec execute 76 sec latency 15185.010 ms oleg337-client.virtnet: 1 7927 0.28 MB/sec execute 77 sec latency 16015.366 ms oleg337-client.virtnet: 1 8047 0.28 MB/sec execute 78 sec latency 105.786 ms oleg337-client.virtnet: 1 8231 0.29 MB/sec execute 79 sec latency 83.745 ms oleg337-client.virtnet: 1 8385 0.29 MB/sec execute 80 sec latency 26.797 ms oleg337-client.virtnet: 1 8597 0.32 MB/sec execute 81 sec latency 113.020 ms oleg337-client.virtnet: 1 8742 0.32 MB/sec execute 82 sec latency 67.797 ms oleg337-client.virtnet: 1 8912 0.32 MB/sec execute 83 sec latency 44.340 ms oleg337-client.virtnet: 1 9095 0.31 MB/sec execute 84 sec latency 66.603 ms oleg337-client.virtnet: 1 9245 0.31 MB/sec execute 85 sec latency 40.116 ms oleg337-client.virtnet: 1 9449 0.32 MB/sec execute 86 sec latency 33.098 ms oleg337-client.virtnet: 1 9533 0.32 MB/sec execute 87 sec latency 108.441 ms oleg337-client.virtnet: 1 9684 0.33 MB/sec execute 88 sec latency 95.424 ms oleg337-client.virtnet: 1 9724 0.32 MB/sec execute 89 sec latency 673.970 ms oleg337-client.virtnet: 1 9724 0.32 MB/sec execute 90 sec latency 1675.959 ms oleg337-client.virtnet: 1 9724 0.32 MB/sec execute 91 sec latency 2676.159 ms oleg337-client.virtnet: 1 9724 0.31 MB/sec execute 92 sec latency 3677.222 ms oleg337-client.virtnet: 1 9724 0.31 MB/sec execute 93 sec latency 4680.306 ms oleg337-client.virtnet: 1 9724 0.31 MB/sec execute 94 sec latency 5682.020 ms oleg337-client.virtnet: 1 9724 0.30 MB/sec execute 95 sec latency 6683.214 ms oleg337-client.virtnet: 1 9724 0.30 MB/sec execute 96 sec latency 7683.563 ms oleg337-client.virtnet: 1 9724 0.30 MB/sec execute 97 sec latency 8683.738 ms oleg337-client.virtnet: 1 9724 0.29 MB/sec execute 98 sec latency 9687.253 ms oleg337-client.virtnet: 1 9724 0.29 MB/sec execute 99 sec latency 10687.525 ms oleg337-client.virtnet: 1 9724 0.29 MB/sec execute 100 sec latency 11688.040 ms oleg337-client.virtnet: 1 9724 0.29 MB/sec execute 101 sec latency 12689.739 ms oleg337-client.virtnet: 1 9724 0.28 MB/sec execute 102 sec latency 13691.007 ms oleg337-client.virtnet: 1 9724 0.28 MB/sec execute 103 sec latency 14691.329 ms oleg337-client.virtnet: 1 9724 0.28 MB/sec execute 104 sec latency 15691.607 ms oleg337-client.virtnet: 1 9724 0.28 MB/sec execute 105 sec latency 16691.762 ms oleg337-client.virtnet: 1 9724 0.27 MB/sec execute 10Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:10:25 (1746141025) targets are mounted 19:10:25 (1746141025) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3723264 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7459840 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:10:48 (1746141048) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:11:03 (1746141063) targets are mounted 19:11:03 (1746141063) facet_failover done 6 sec latency 17693.646 ms oleg337-client.virtnet: 1 9724 0.27 MB/sec execute 107 sec latency 18693.848 ms oleg337-client.virtnet: 1 9724 0.27 MB/sec execute 108 sec latency 19694.145 ms oleg337-client.virtnet: 1 9724 0.27 MB/sec execute 109 sec latency 20694.399 ms oleg337-client.virtnet: 1 9724 0.26 MB/sec execute 110 sec latency 21696.392 ms oleg337-client.virtnet: 1 9724 0.26 MB/sec execute 111 sec latency 22696.511 ms oleg337-client.virtnet: 1 9724 0.26 MB/sec execute 112 sec latency 23697.446 ms oleg337-client.virtnet: 1 9724 0.26 MB/sec execute 113 sec latency 24697.686 ms oleg337-client.virtnet: 1 9724 0.25 MB/sec execute 114 sec latency 25698.810 ms oleg337-client.virtnet: 1 9724 0.25 MB/sec execute 115 sec latency 26698.988 ms oleg337-client.virtnet: 1 9724 0.25 MB/sec execute 116 sec latency 27699.711 ms oleg337-client.virtnet: 1 9876 0.27 MB/sec execute 117 sec latency 28269.352 ms oleg337-client.virtnet: 1 10099 0.28 MB/sec execute 118 sec latency 131.120 ms oleg337-client.virtnet: 1 10336 0.29 MB/sec execute 119 sec latency 139.237 ms oleg337-client.virtnet: 1 10621 0.32 MB/sec execute 120 sec latency 58.533 ms oleg337-client.virtnet: 1 10785 0.32 MB/sec execute 121 sec latency 32.125 ms oleg337-client.virtnet: 1 10937 0.32 MB/sec execute 122 sec latency 41.613 ms oleg337-client.virtnet: 1 11078 0.32 MB/sec execute 123 sec latency 38.620 ms oleg337-client.virtnet: 1 11241 0.32 MB/sec execute 124 sec latency 33.518 ms oleg337-client.virtnet: 1 11381 0.32 MB/sec execute 125 sec latency 95.508 ms oleg337-client.virtnet: 1 11461 0.32 MB/sec execute 126 sec latency 73.624 ms oleg337-client.virtnet: 1 11463 0.32 MB/sec execute 127 sec latency 967.696 ms oleg337-client.virtnet: 1 11463 0.32 MB/sec execute 128 sec latency 1967.877 ms oleg337-client.virtnet: 1 11463 0.31 MB/sec execute 129 sec latency 2968.050 ms oleg337-client.virtnet: 1 11463 0.31 MB/sec execute 130 sec latency 3969.072 ms oleg337-client.virtnet: 1 11463 0.31 MB/sec execute 131 sec latency 4969.874 ms oleg337-client.virtnet: 1 11463 0.31 MB/sec execute 132 sec latency 5970.218 ms oleg337-client.virtnet: 1 11463 0.30 MB/sec execute 133 sec latency 6970.351 ms oleg337-client.virtnet: 1 11463 0.30 MB/sec execute 134 sec latency 7970.532 ms oleg337-client.virtnet: 1 11463 0.30 MB/sec execute 135 sec latency 8970.695 ms oleg337-client.virtnet: 1 11463 0.30 MB/sec execute 136 sec latency 9970.899 ms oleg337-client.virtnet: 1 11463 0.30 MB/sec execute 137 sec latency 10971.082 ms oleg337-client.virtnet: 1 11463 0.29 MB/sec execute 138 sec latency 11972.348 ms oleg337-client.virtnet: 1 11463 0.29 MB/sec execute 139 sec latency 12973.863 ms oleg337-client.virtnet: 1 11463 0.29 MB/sec execute 140 sec latency 13974.036 ms oleg337-client.virtnet: 1 11463 0.29 MB/sec execute 141 sec latency 14974.906 ms oleg337-client.virtnet: 1 11463 0.28 MB/sec execute 142 sec latency 15975.180 ms oleg337-client.virtnet: 1 11463 0.28 MB/sec execute 143 sec latency 16975.662 ms oleg337-client.virtnet: 1 11463 0.28 MB/sec execute 144 sec latency 17976.344 ms oleg337-client.virtnet: 1 11463 0.28 MB/sec execute 145 sec latency 18976.708 ms oleg337-client.virtnet: 1 11463 0.28 MB/sec execute 146 sec latency 19979.242 ms oleg337-client.virtnet: 1 11463 0.27 MB/sec execute 147 sec latency 20979.468 ms oleg337-client.virtnet: 1 11463 0.27 MB/sec execute 148 sec latency 21980.754 ms oleg337-client.virtnet: 1 11463 0.27 MB/sec execute 149 sec latency 22980.942 ms oleg337-client.virtnet: 1 11463 0.27 MB/sec executoleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3728384 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7481344 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:11:24 (1746141084) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:11:39 (1746141099) targets are mounted 19:11:39 (1746141099) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3723264 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7436288 1% /mnt/lustre e 150 sec latency 23981.159 ms oleg337-client.virtnet: 1 11463 0.27 MB/sec execute 151 sec latency 24981.352 ms oleg337-client.virtnet: 1 11463 0.27 MB/sec execute 152 sec latency 25981.913 ms oleg337-client.virtnet: 1 11463 0.26 MB/sec execute 153 sec latency 26983.189 ms oleg337-client.virtnet: 1 11463 0.26 MB/sec execute 154 sec latency 27983.459 ms oleg337-client.virtnet: 1 11463 0.26 MB/sec execute 155 sec latency 28984.043 ms oleg337-client.virtnet: 1 11463 0.26 MB/sec execute 156 sec latency 29984.546 ms oleg337-client.virtnet: 1 11471 0.26 MB/sec execute 157 sec latency 30746.996 ms oleg337-client.virtnet: 1 11572 0.26 MB/sec execute 158 sec latency 151.781 ms oleg337-client.virtnet: 1 11641 0.26 MB/sec execute 159 sec latency 133.642 ms oleg337-client.virtnet: 1 11789 0.26 MB/sec execute 160 sec latency 87.913 ms oleg337-client.virtnet: 1 11924 0.26 MB/sec execute 161 sec latency 44.867 ms oleg337-client.virtnet: 1 12144 0.28 MB/sec execute 162 sec latency 122.607 ms oleg337-client.virtnet: 1 12274 0.28 MB/sec execute 163 sec latency 59.296 ms oleg337-client.virtnet: 1 12378 0.28 MB/sec execute 164 sec latency 111.361 ms oleg337-client.virtnet: 1 12517 0.27 MB/sec execute 165 sec latency 64.186 ms oleg337-client.virtnet: 1 12662 0.27 MB/sec execute 166 sec latency 108.305 ms oleg337-client.virtnet: 1 12813 0.27 MB/sec execute 167 sec latency 51.734 ms oleg337-client.virtnet: 1 12913 0.27 MB/sec execute 168 sec latency 384.676 ms oleg337-client.virtnet: 1 12913 0.27 MB/sec execute 169 sec latency 1385.215 ms oleg337-client.virtnet: 1 12913 0.27 MB/sec execute 170 sec latency 2387.192 ms oleg337-client.virtnet: 1 12913 0.27 MB/sec execute 171 sec latency 3388.932 ms oleg337-client.virtnet: 1 12913 0.27 MB/sec execute 172 sec latency 4389.792 ms oleg337-client.virtnet: 1 12913 0.26 MB/sec execute 173 sec latency 5397.089 ms oleg337-client.virtnet: 1 12913 0.26 MB/sec execute 174 sec latency 6398.170 ms oleg337-client.virtnet: 1 12913 0.26 MB/sec execute 175 sec latency 7398.424 ms oleg337-client.virtnet: 1 12913 0.26 MB/sec execute 176 sec latency 8398.651 ms oleg337-client.virtnet: 1 12913 0.26 MB/sec execute 177 sec latency 9400.017 ms oleg337-client.virtnet: 1 12913 0.26 MB/sec execute 178 sec latency 10400.299 ms oleg337-client.virtnet: 1 12913 0.26 MB/sec execute 179 sec latency 11400.675 ms oleg337-client.virtnet: 1 12913 0.25 MB/sec execute 180 sec latency 12400.991 ms oleg337-client.virtnet: 1 12913 0.25 MB/sec execute 181 sec latency 13402.583 ms oleg337-client.virtnet: 1 12913 0.25 MB/sec execute 182 sec latency 14404.197 ms oleg337-client.virtnet: 1 12913 0.25 MB/sec execute 183 sec latency 15404.495 ms oleg337-client.virtnet: 1 12913 0.25 MB/sec execute 184 sec latency 16405.332 ms oleg337-client.virtnet: 1 12913 0.25 MB/sec execute 185 sec latency 17406.697 ms oleg337-client.virtnet: 1 12913 0.25 MB/sec execute 186 sec latency 18408.732 ms oleg337-client.virtnet: 1 12999 0.25 MB/sec execute 187 sec latency 18923.992 ms oleg337-client.virtnet: 1 13072 0.25 MB/sec execute 188 sec latency 170.196 ms oleg337-client.virtnet: 1 13235 0.25 MB/sec execute 189 sec latency 90.537 ms oleg337-client.virtnet: 1 13552 0.27 MB/sec execute 190 sec latency 91.460 ms oleg337-client.virtnet: 1 13674 0.27 MB/sec execute 191 sec latency 138.480 ms oleg337-client.virtnet: 1 14014 0.28 MB/sec execute 192 sec latency 78.067 ms oleg337-client.virtnet: 1 14262 0.30 MB/sec execute 193 sec latency 38.507 ms oleg337-client.virtnet: 1 14395 0.30 MB/sec execute 194 sec ltest_70b fail mds1 9 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:11:54 (1746141114) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:12:09 (1746141129) targets are mounted 19:12:09 (1746141129) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3718144 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7463936 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 19:12:23 (1746141143) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server atency 57.021 ms oleg337-client.virtnet: 1 14523 0.30 MB/sec execute 195 sec latency 39.650 ms oleg337-client.virtnet: 1 14610 0.30 MB/sec execute 196 sec latency 80.869 ms oleg337-client.virtnet: 1 14694 0.30 MB/sec execute 197 sec latency 67.545 ms oleg337-client.virtnet: 1 14704 0.30 MB/sec execute 198 sec latency 916.401 ms oleg337-client.virtnet: 1 14704 0.30 MB/sec execute 199 sec latency 1916.611 ms oleg337-client.virtnet: 1 14704 0.30 MB/sec execute 200 sec latency 2916.775 ms oleg337-client.virtnet: 1 14704 0.29 MB/sec execute 201 sec latency 3916.947 ms oleg337-client.virtnet: 1 14704 0.29 MB/sec execute 202 sec latency 4920.656 ms oleg337-client.virtnet: 1 14704 0.29 MB/sec execute 203 sec latency 5920.838 ms oleg337-client.virtnet: 1 14704 0.29 MB/sec execute 204 sec latency 6921.567 ms oleg337-client.virtnet: 1 14704 0.29 MB/sec execute 205 sec latency 7921.690 ms oleg337-client.virtnet: 1 14704 0.29 MB/sec execute 206 sec latency 8922.138 ms oleg337-client.virtnet: 1 14704 0.29 MB/sec execute 207 sec latency 9922.311 ms oleg337-client.virtnet: 1 14704 0.28 MB/sec execute 208 sec latency 10922.544 ms oleg337-client.virtnet: 1 14704 0.28 MB/sec execute 209 sec latency 11922.666 ms oleg337-client.virtnet: 1 14704 0.28 MB/sec execute 210 sec latency 12923.045 ms oleg337-client.virtnet: 1 14704 0.28 MB/sec execute 211 sec latency 13923.741 ms oleg337-client.virtnet: 1 14704 0.28 MB/sec execute 212 sec latency 14923.902 ms oleg337-client.virtnet: 1 14704 0.28 MB/sec execute 213 sec latency 15924.127 ms oleg337-client.virtnet: 1 14704 0.28 MB/sec execute 214 sec latency 16925.832 ms oleg337-client.virtnet: 1 14704 0.27 MB/sec execute 215 sec latency 17926.008 ms oleg337-client.virtnet: 1 14704 0.27 MB/sec execute 216 sec latency 18927.180 ms oleg337-client.virtnet: 1 14704 0.27 MB/sec execute 217 sec latency 19927.775 ms oleg337-client.virtnet: 1 14801 0.27 MB/sec execute 218 sec latency 19984.905 ms oleg337-client.virtnet: 1 14904 0.27 MB/sec execute 219 sec latency 142.439 ms oleg337-client.virtnet: 1 14996 0.27 MB/sec execute 220 sec latency 85.843 ms oleg337-client.virtnet: 1 15109 0.27 MB/sec execute 221 sec latency 90.766 ms oleg337-client.virtnet: 1 15246 0.27 MB/sec execute 222 sec latency 72.612 ms oleg337-client.virtnet: 1 15426 0.27 MB/sec execute 223 sec latency 40.780 ms oleg337-client.virtnet: 1 15621 0.29 MB/sec execute 224 sec latency 71.405 ms oleg337-client.virtnet: 1 15761 0.29 MB/sec execute 225 sec latency 97.284 ms oleg337-client.virtnet: 1 15870 0.28 MB/sec execute 226 sec latency 99.985 ms oleg337-client.virtnet: 1 15970 0.28 MB/sec execute 227 sec latency 210.109 ms oleg337-client.virtnet: 1 15970 0.28 MB/sec execute 228 sec latency 1212.687 ms oleg337-client.virtnet: 1 15970 0.28 MB/sec execute 229 sec latency 2213.222 ms oleg337-client.virtnet: 1 15970 0.28 MB/sec execute 230 sec latency 3213.388 ms oleg337-client.virtnet: 1 15970 0.28 MB/sec execute 231 sec latency 4215.040 ms oleg337-client.virtnet: 1 15970 0.28 MB/sec execute 232 sec latency 5215.222 ms oleg337-client.virtnet: 1 15970 0.28 MB/sec execute 233 sec latency 6215.373 ms oleg337-client.virtnet: 1 15970 0.27 MB/sec execute 234 sec latency 7215.930 ms oleg337-client.virtnet: 1 15970 0.27 MB/sec execute 235 sec latency 8216.240 ms oleg337-client.virtnet: 1 15970 0.27 MB/sec execute 236 sec latency 9217.940 ms oleg337-client.virtnet: 1 15970 0.27 MB/sec execute 237 sec latency 10218.117 ms oleg337-client.virtnet: 1 15970 0.27 MB/sec execute 238 sec latency 11218.320 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 19:12:38 (1746141158) targets are mounted 19:12:38 (1746141158) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec ms oleg337-client.virtnet: 1 15970 0.27 MB/sec execute 239 sec latency 12218.556 ms oleg337-client.virtnet: 1 15970 0.27 MB/sec execute 240 sec latency 13218.711 ms oleg337-client.virtnet: 1 15970 0.27 MB/sec execute 241 sec latency 14219.258 ms oleg337-client.virtnet: 1 15970 0.27 MB/sec execute 242 sec latency 15220.740 ms oleg337-client.virtnet: 1 15970 0.26 MB/sec execute 243 sec latency 16220.971 ms oleg337-client.virtnet: 1 15970 0.26 MB/sec execute 244 sec latency 17223.953 ms oleg337-client.virtnet: 1 15970 0.26 MB/sec execute 245 sec latency 18224.215 ms oleg337-client.virtnet: 1 15970 0.26 MB/sec execute 246 sec latency 19224.790 ms oleg337-client.virtnet: 1 16000 0.26 MB/sec execute 247 sec latency 19749.413 ms oleg337-client.virtnet: 1 16134 0.26 MB/sec execute 248 sec latency 112.999 ms oleg337-client.virtnet: 1 16256 0.26 MB/sec execute 249 sec latency 47.485 ms oleg337-client.virtnet: 1 16459 0.26 MB/sec execute 250 sec latency 43.677 ms oleg337-client.virtnet: 1 16612 0.26 MB/sec execute 251 sec latency 111.347 ms oleg337-client.virtnet: 1 16803 0.27 MB/sec execute 252 sec latency 140.484 ms oleg337-client.virtnet: 1 17176 0.28 MB/sec execute 253 sec latency 112.340 ms oleg337-client.virtnet: 1 17518 0.29 MB/sec execute 254 sec latency 88.978 ms oleg337-client.virtnet: 1 17812 0.30 MB/sec execute 255 sec latency 44.767 ms oleg337-client.virtnet: 1 17989 0.30 MB/sec execute 256 sec latency 32.627 ms oleg337-client.virtnet: 1 18131 0.30 MB/sec execute 257 sec latency 32.378 ms oleg337-client.virtnet: 1 18278 0.30 MB/sec execute 258 sec latency 28.416 ms oleg337-client.virtnet: 1 18461 0.30 MB/sec execute 259 sec latency 111.129 ms oleg337-client.virtnet: 1 18551 0.30 MB/sec execute 260 sec latency 89.393 ms oleg337-client.virtnet: 1 18667 0.30 MB/sec execute 261 sec latency 89.883 ms oleg337-client.virtnet: 1 18836 0.31 MB/sec execute 262 sec latency 94.509 ms oleg337-client.virtnet: 1 19023 0.31 MB/sec execute 263 sec latency 29.108 ms oleg337-client.virtnet: 1 19263 0.32 MB/sec execute 264 sec latency 77.324 ms oleg337-client.virtnet: 1 19430 0.31 MB/sec execute 265 sec latency 73.726 ms oleg337-client.virtnet: 1 19566 0.31 MB/sec execute 266 sec latency 32.993 ms oleg337-client.virtnet: 1 19751 0.31 MB/sec execute 267 sec latency 77.073 ms oleg337-client.virtnet: 1 19921 0.31 MB/sec execute 268 sec latency 33.598 ms oleg337-client.virtnet: 1 20130 0.32 MB/sec execute 269 sec latency 57.918 ms oleg337-client.virtnet: 1 20234 0.32 MB/sec execute 270 sec latency 110.051 ms oleg337-client.virtnet: 1 20416 0.32 MB/sec execute 271 sec latency 102.635 ms oleg337-client.virtnet: 1 20762 0.33 MB/sec execute 272 sec latency 99.385 ms oleg337-client.virtnet: 1 21084 0.34 MB/sec execute 273 sec latency 91.638 ms oleg337-client.virtnet: 1 21356 0.35 MB/sec execute 274 sec latency 43.654 ms oleg337-client.virtnet: 1 21549 0.35 MB/sec execute 275 sec latency 29.295 ms oleg337-client.virtnet: 1 21684 0.35 MB/sec execute 276 sec latency 57.803 ms oleg337-client.virtnet: 1 21809 0.35 MB/sec execute 277 sec latency 32.431 ms oleg337-client.virtnet: 1 22001 0.35 MB/sec execute 278 sec latency 31.534 ms oleg337-client.virtnet: 1 22066 0.35 MB/sec execute 279 sec latency 118.565 ms oleg337-client.virtnet: 1 22159 0.35 MB/sec execute 280 sec latency 95.569 ms oleg337-client.virtnet: 1 22268 0.35 MB/sec execute 281 sec latency 100.442 ms oleg337-client.virtnet: 1 22410 0.35 MB/sec execute 282 sec latency 79.348 ms oleg337-client.virtnet: 1 22573 0.35 MB/sec execute 283 sec latency 34.796 ms oleg337-client.virtnet: 1 22812 0.36 MB/sec execute 284 sec latency 36.612 ms oleg337-client.virtnet: 1 22955 0.36 MB/sec execute 285 sec latency 119.429 ms oleg337-client.virtnet: 1 23101 0.36 MB/sec execute 286 sec latency 103.689 ms oleg337-client.virtnet: 1 23218 0.36 MB/sec execute 287 sec latency 138.383 ms oleg337-client.virtnet: 1 23431 0.36 MB/sec execute 288 sec latency 62.619 ms oleg337-client.virtnet: 1 23627 0.36 MB/sec execute 289 sec latency 36.615 ms oleg337-client.virtnet: 1 23754 0.36 MB/sec execute 290 sec latency 97.527 ms oleg337-client.virtnet: 1 23932 0.36 MB/sec execute 291 sec latency 89.957 ms oleg337-client.virtnet: 1 24255 0.38 MB/sec execute 292 sec latency 107.833 ms oleg337-client.virtnet: 1 24530 0.38 MB/sec execute 293 sec latency 88.244 ms oleg337-client.virtnet: 1 24839 0.39 MB/sec execute 294 sec latency 53.369 ms oleg337-client.virtnet: 1 25000 0.39 MB/sec execute 295 sec latency 30.975 ms oleg337-client.virtnet: 1 25160 0.39 MB/sec execute 296 sec latency 25.408 ms oleg337-client.virtnet: 1 25309 0.39 MB/sec execute 297 sec latency 36.944 ms oleg337-client.virtnet: 1 25456 0.39 MB/sec execute 298 sec latency 27.480 ms oleg337-client.virtnet: 1 25607 0.39 MB/sec execute 299 sec latency 82.812 ms oleg337-client.virtnet: 1 cleanup 300 sec oleg337-client.virtnet: 0 cleanup 301 sec oleg337-client.virtnet: oleg337-client.virtnet: Operation Count AvgLat MaxLat oleg337-client.virtnet: ---------------------------------------- oleg337-client.virtnet: NTCreateX 3905 41.980 30746.935 oleg337-client.virtnet: Close 2831 9.378 16015.335 oleg337-client.virtnet: Rename 162 24.965 62.846 oleg337-client.virtnet: Unlink 817 29.984 19749.376 oleg337-client.virtnet: Qpathinfo 3500 9.981 19546.958 oleg337-client.virtnet: Qfileinfo 593 0.269 9.158 oleg337-client.virtnet: Qfsinfo 645 0.475 17.382 oleg337-client.virtnet: Sfileinfo 309 68.039 15907.972 oleg337-client.virtnet: Find 1333 1.043 76.658 oleg337-client.virtnet: WriteX 1883 2.325 33.028 oleg337-client.virtnet: ReadX 5890 0.049 16.312 oleg337-client.virtnet: LockX 12 4.672 9.045 oleg337-client.virtnet: UnlockX 12 4.215 10.492 oleg337-client.virtnet: Flush 276 65.349 182.010 oleg337-client.virtnet: oleg337-client.virtnet: Throughput 0.390371 MB/sec 1 clients 1 procs max_latency=30746.996 ms oleg337-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg337-client.virtnet at Thu May 1 19:13:36 EDT 2025 with return code 0 oleg337-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg337-client.virtnet oleg337-client.virtnet: /mnt/lustre/d70b.replay-single/oleg337-client.virtnet /mnt/lustre/d70b.replay-single/oleg337-client.virtnet oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg337-client.virtnet: removed directory: 'clients/client0' oleg337-client.virtnet: removed directory: 'clients' oleg337-client.virtnet: removed 'client.txt' oleg337-client.virtnet: /mnt/lustre/d70b.replay-single/oleg337-client.virtnet oleg337-client.virtnet: dbench successfully finished