== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 18:53:32 (1746140012) Starting client oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre Started clients oleg309-client.virtnet: 192.168.203.109@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg309-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13962 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2512 1285176 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3580864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1568 3605452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7186316 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg309-server Stopping /mnt/lustre-mds1 (opts:) on oleg309-server 18:53:44 (1746140024) shut down facet: mds1 facet_host: oleg309-server facet_failover_host: oleg309-server Failover mds1 to oleg309-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 18:54:02 (1746140042) targets are mounted 18:54:02 (1746140042) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg309-client.virtnet: looking for dbench program oleg309-client.virtnet: /usr/bin/dbench oleg309-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg309-client.virtnet oleg309-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg309-client.virtnet' oleg309-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg309-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg309-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg309-client.virtnet at Thu May 1 18:53:34 EDT 2025 oleg309-client.virtnet: waiting for dbench pid 13990 oleg309-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg309-client.virtnet: oleg309-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg309-client.virtnet: failed to create barrier semaphore oleg309-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg309-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg309-client.virtnet: releasing clients oleg309-client.virtnet: 1 147 6.28 MB/sec warmup 1 sec latency 53.579 ms oleg309-client.virtnet: 1 339 6.13 MB/sec warmup 2 sec latency 67.289 ms oleg309-client.virtnet: 1 504 5.82 MB/sec warmup 3 sec latency 36.791 ms oleg309-client.virtnet: 1 744 5.33 MB/sec warmup 4 sec latency 38.424 ms oleg309-client.virtnet: 1 874 4.33 MB/sec warmup 5 sec latency 343.189 ms oleg309-client.virtnet: 1 1050 3.81 MB/sec warmup 6 sec latency 40.707 ms oleg309-client.virtnet: 1 1164 3.29 MB/sec warmup 7 sec latency 117.699 ms oleg309-client.virtnet: 1 1385 3.25 MB/sec warmup 8 sec latency 95.214 ms oleg309-client.virtnet: 1 1385 2.89 MB/sec warmup 9 sec latency 1095.366 ms oleg309-client.virtnet: 1 1385 2.60 MB/sec warmup 10 sec latency 2096.907 ms oleg309-client.virtnet: 1 1385 2.36 MB/sec warmup 11 sec latency 3097.258 ms oleg309-client.virtnet: 1 1385 2.17 MB/sec warmup 12 sec latency 4098.049 ms oleg309-client.virtnet: 1 1385 2.00 MB/sec warmup 13 sec latency 5098.207 ms oleg309-client.virtnet: 1 1385 1.86 MB/sec warmup 14 sec latency 6099.848 ms oleg309-client.virtnet: 1 1385 1.73 MB/sec warmup 15 sec latency 7099.960 ms oleg309-client.virtnet: 1 1385 1.62 MB/sec warmup 16 sec latency 8100.119 ms oleg309-client.virtnet: 1 1385 1.53 MB/sec warmup 17 sec latency 9101.034 ms oleg309-client.virtnet: 1 1385 1.44 MB/sec warmup 18 sec latency 10101.205 ms oleg309-client.virtnet: 1 1385 1.37 MB/sec warmup 19 sec latency 11102.315 ms oleg309-client.virtnet: 1 1385 1.30 MB/sec warmup 20 sec latency 12102.492 ms oleg309-client.virtnet: 1 1385 1.24 MB/sec warmup 21 sec latency 13102.646 ms oleg309-client.virtnet: 1 1385 1.18 MB/sec warmup 22 sec latency 14104.374 ms oleg309-client.virtnet: 1 1385 1.13 MB/sec warmup 23 sec latency 15105.008 ms oleg309-client.virtnet: 1 1385 1.08 MB/sec warmup 24 sec latency 16105.195 ms oleg309-client.virtnet: 1 1385 1.04 MB/sec warmup 25 sec latency 17105.654 ms oleg309-client.virtnet: 1 1385 1.00 MB/sec warmup 26 sec latency 18107.106 ms oleg309-client.virtnet: 1 1385 0.96 MB/sec warmup 27 sec latency 19107.476 ms oleg309-client.virtnet: 1 1385 0.93 MB/sec warmup 28 sec latency 20108.222 ms oleg309-client.virtnet: 1 1385 0.90 MB/sec warmup 29 sec latency 21108.395 ms oleg309-client.virtnet: 1 1385 0.87 MB/sec warmup 30 sec latency 22109.463 ms oleg309-client.virtnet: 1 1509 0.84 MB/sec warmup 31 sec latency 22142.116 ms oleg309-client.virtnet: 1 1650 0.82 MB/sec warmup 32 sec latency 42.987 ms oleg309-client.virtnet: 1 1805 0.80 MB/sec warmup 33 sUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3566560 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12064 3590900 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7157460 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg309-server Stopping /mnt/lustre-mds2 (opts:) on oleg309-server 18:54:18 (1746140058) shut down facet: mds2 facet_host: oleg309-server facet_failover_host: oleg309-server Failover mds2 to oleg309-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0001 18:54:34 (1746140074) targets are mounted 18:54:34 (1746140074) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3568752 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12072 3594336 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7163088 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg309-server Stopping /mnt/lustre-mds1 (opts:) on oleg309-server 18:54:51 (1746140091) shut down facet: mds1 facet_host: oleg309-server facet_failover_host: oleg309-server ec latency 40.755 ms oleg309-client.virtnet: 1 1980 0.78 MB/sec warmup 34 sec latency 73.427 ms oleg309-client.virtnet: 1 2152 0.76 MB/sec warmup 35 sec latency 53.567 ms oleg309-client.virtnet: 1 2373 0.77 MB/sec warmup 36 sec latency 38.819 ms oleg309-client.virtnet: 1 2473 0.77 MB/sec warmup 37 sec latency 55.096 ms oleg309-client.virtnet: 1 2719 0.84 MB/sec warmup 38 sec latency 63.398 ms oleg309-client.virtnet: 1 2987 0.86 MB/sec warmup 39 sec latency 39.885 ms oleg309-client.virtnet: 1 3357 0.89 MB/sec warmup 40 sec latency 36.773 ms oleg309-client.virtnet: 1 3592 0.94 MB/sec warmup 41 sec latency 59.009 ms oleg309-client.virtnet: 1 3760 0.95 MB/sec warmup 42 sec latency 44.213 ms oleg309-client.virtnet: 1 3844 0.92 MB/sec warmup 43 sec latency 75.684 ms oleg309-client.virtnet: 1 3942 0.91 MB/sec warmup 44 sec latency 55.693 ms oleg309-client.virtnet: 1 4045 0.89 MB/sec warmup 45 sec latency 75.475 ms oleg309-client.virtnet: 1 4083 0.87 MB/sec warmup 46 sec latency 745.239 ms oleg309-client.virtnet: 1 4083 0.85 MB/sec warmup 47 sec latency 1746.502 ms oleg309-client.virtnet: 1 4083 0.83 MB/sec warmup 48 sec latency 2746.661 ms oleg309-client.virtnet: 1 4083 0.82 MB/sec warmup 49 sec latency 3747.374 ms oleg309-client.virtnet: 1 4083 0.80 MB/sec warmup 50 sec latency 4747.547 ms oleg309-client.virtnet: 1 4083 0.79 MB/sec warmup 51 sec latency 5751.181 ms oleg309-client.virtnet: 1 4083 0.77 MB/sec warmup 52 sec latency 6751.628 ms oleg309-client.virtnet: 1 4083 0.76 MB/sec warmup 53 sec latency 7752.404 ms oleg309-client.virtnet: 1 4083 0.74 MB/sec warmup 54 sec latency 8753.667 ms oleg309-client.virtnet: 1 4083 0.73 MB/sec warmup 55 sec latency 9755.092 ms oleg309-client.virtnet: 1 4083 0.72 MB/sec warmup 56 sec latency 10755.251 ms oleg309-client.virtnet: 1 4083 0.70 MB/sec warmup 57 sec latency 11755.425 ms oleg309-client.virtnet: 1 4083 0.69 MB/sec warmup 58 sec latency 12757.085 ms oleg309-client.virtnet: 1 4083 0.68 MB/sec warmup 59 sec latency 13757.231 ms oleg309-client.virtnet: 1 4083 0.00 MB/sec execute 1 sec latency 15757.725 ms oleg309-client.virtnet: 1 4083 0.00 MB/sec execute 2 sec latency 16763.112 ms oleg309-client.virtnet: 1 4174 0.00 MB/sec execute 3 sec latency 17339.294 ms oleg309-client.virtnet: 1 4304 0.09 MB/sec execute 4 sec latency 53.560 ms oleg309-client.virtnet: 1 4433 0.12 MB/sec execute 5 sec latency 49.417 ms oleg309-client.virtnet: 1 4557 0.16 MB/sec execute 6 sec latency 66.866 ms oleg309-client.virtnet: 1 4738 0.29 MB/sec execute 7 sec latency 108.627 ms oleg309-client.virtnet: 1 4969 0.63 MB/sec execute 8 sec latency 46.842 ms oleg309-client.virtnet: 1 5089 0.57 MB/sec execute 9 sec latency 48.904 ms oleg309-client.virtnet: 1 5225 0.52 MB/sec execute 10 sec latency 46.594 ms oleg309-client.virtnet: 1 5336 0.48 MB/sec execute 11 sec latency 56.787 ms oleg309-client.virtnet: 1 5418 0.45 MB/sec execute 12 sec latency 51.749 ms oleg309-client.virtnet: 1 5567 0.42 MB/sec execute 13 sec latency 62.887 ms oleg309-client.virtnet: 1 5703 0.41 MB/sec execute 14 sec latency 62.905 ms oleg309-client.virtnet: 1 5786 0.39 MB/sec execute 15 sec latency 343.982 ms oleg309-client.virtnet: 1 5786 0.36 MB/sec execute 16 sec latency 1344.940 ms oleg309-client.virtnet: 1 5786 0.34 MB/sec execute 17 sec latency 2345.236 ms oleg309-client.virtnet: 1 5786 0.32 MB/sec execute 18 sec latency 3345.394 ms oleg309-client.virtnet: 1 5786 0.31 Failover mds1 to oleg309-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 18:55:20 (1746140120) targets are mounted 18:55:20 (1746140120) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37440 3566976 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12252 3593944 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49692 7160920 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg309-server Stopping /mnt/lustre-mds2 (opts:) on oleg309-server 18:55:36 (1746140136) shut down facet: mds2 facet_host: oleg309-server facet_failover_host: oleg309-server MB/sec execute 19 sec latency 4345.674 ms oleg309-client.virtnet: 1 5786 0.29 MB/sec execute 20 sec latency 5345.895 ms oleg309-client.virtnet: 1 5786 0.28 MB/sec execute 21 sec latency 6346.422 ms oleg309-client.virtnet: 1 5786 0.26 MB/sec execute 22 sec latency 7346.573 ms oleg309-client.virtnet: 1 5786 0.25 MB/sec execute 23 sec latency 8346.880 ms oleg309-client.virtnet: 1 5786 0.24 MB/sec execute 24 sec latency 9347.314 ms oleg309-client.virtnet: 1 5786 0.23 MB/sec execute 25 sec latency 10350.233 ms oleg309-client.virtnet: 1 5786 0.22 MB/sec execute 26 sec latency 11350.387 ms oleg309-client.virtnet: 1 5786 0.22 MB/sec execute 27 sec latency 12350.661 ms oleg309-client.virtnet: 1 5786 0.21 MB/sec execute 28 sec latency 13353.502 ms oleg309-client.virtnet: 1 5786 0.20 MB/sec execute 29 sec latency 14356.139 ms oleg309-client.virtnet: 1 5786 0.19 MB/sec execute 30 sec latency 15356.400 ms oleg309-client.virtnet: 1 5786 0.19 MB/sec execute 31 sec latency 16356.557 ms oleg309-client.virtnet: 1 5786 0.18 MB/sec execute 32 sec latency 17357.364 ms oleg309-client.virtnet: 1 5786 0.18 MB/sec execute 33 sec latency 18361.467 ms oleg309-client.virtnet: 1 5786 0.17 MB/sec execute 34 sec latency 19361.859 ms oleg309-client.virtnet: 1 5786 0.17 MB/sec execute 35 sec latency 20362.665 ms oleg309-client.virtnet: 1 5786 0.16 MB/sec execute 36 sec latency 21363.069 ms oleg309-client.virtnet: 1 5786 0.16 MB/sec execute 37 sec latency 22363.284 ms oleg309-client.virtnet: 1 5786 0.15 MB/sec execute 38 sec latency 23364.188 ms oleg309-client.virtnet: 1 5786 0.15 MB/sec execute 39 sec latency 24365.541 ms oleg309-client.virtnet: 1 5786 0.15 MB/sec execute 40 sec latency 25365.764 ms oleg309-client.virtnet: 1 5786 0.14 MB/sec execute 41 sec latency 26366.353 ms oleg309-client.virtnet: 1 5786 0.14 MB/sec execute 42 sec latency 27366.730 ms oleg309-client.virtnet: 1 5786 0.14 MB/sec execute 43 sec latency 28366.895 ms oleg309-client.virtnet: 1 5786 0.13 MB/sec execute 44 sec latency 29367.131 ms oleg309-client.virtnet: 1 5786 0.13 MB/sec execute 45 sec latency 30369.675 ms oleg309-client.virtnet: 1 5786 0.13 MB/sec execute 46 sec latency 31369.914 ms oleg309-client.virtnet: 1 5787 0.12 MB/sec execute 47 sec latency 32367.909 ms oleg309-client.virtnet: 1 5954 0.15 MB/sec execute 48 sec latency 56.532 ms oleg309-client.virtnet: 1 6058 0.16 MB/sec execute 49 sec latency 64.694 ms oleg309-client.virtnet: 1 6335 0.22 MB/sec execute 50 sec latency 61.902 ms oleg309-client.virtnet: 1 6732 0.26 MB/sec execute 51 sec latency 45.691 ms oleg309-client.virtnet: 1 7077 0.33 MB/sec execute 52 sec latency 40.830 ms oleg309-client.virtnet: 1 7199 0.33 MB/sec execute 53 sec latency 47.644 ms oleg309-client.virtnet: 1 7332 0.35 MB/sec execute 54 sec latency 53.229 ms oleg309-client.virtnet: 1 7428 0.34 MB/sec execute 55 sec latency 56.775 ms oleg309-client.virtnet: 1 7520 0.34 MB/sec execute 56 sec latency 67.426 ms oleg309-client.virtnet: 1 7606 0.33 MB/sec execute 57 sec latency 48.292 ms oleg309-client.virtnet: 1 7778 0.33 MB/sec execute 58 sec latency 63.507 ms oleg309-client.virtnet: 1 7889 0.33 MB/sec execute 59 sec latency 46.630 ms oleg309-client.virtnet: 1 7993 0.33 MB/sec execute 60 sec latency 67.350 ms oleg309-client.virtnet: 1 8106 0.33 MB/sec execute 61 sec latency 64.291 ms oleg309-client.virtnet: 1 8289 0.34 MB/sec execute 62 sec latency 43.051 ms oleg309-client.virtnet: 1 8556 0.38 MB/sec exeFailover mds2 to oleg309-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0001 18:55:55 (1746140155) targets are mounted 18:55:55 (1746140155) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37660 3568884 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11904 3594288 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7163172 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg309-server Stopping /mnt/lustre-mds1 (opts:) on oleg309-server 18:56:11 (1746140171) shut down facet: mds1 facet_host: oleg309-server facet_failover_host: oleg309-server Failover mds1 to oleg309-server mount facets: mds1 cute 63 sec latency 32.664 ms oleg309-client.virtnet: 1 8724 0.38 MB/sec execute 64 sec latency 35.625 ms oleg309-client.virtnet: 1 8891 0.38 MB/sec execute 65 sec latency 29.813 ms oleg309-client.virtnet: 1 9087 0.37 MB/sec execute 66 sec latency 43.717 ms oleg309-client.virtnet: 1 9270 0.37 MB/sec execute 67 sec latency 36.554 ms oleg309-client.virtnet: 1 9516 0.38 MB/sec execute 68 sec latency 28.500 ms oleg309-client.virtnet: 1 9702 0.39 MB/sec execute 69 sec latency 46.316 ms oleg309-client.virtnet: 1 10111 0.45 MB/sec execute 70 sec latency 30.811 ms oleg309-client.virtnet: 1 10622 0.51 MB/sec execute 71 sec latency 33.595 ms oleg309-client.virtnet: 1 10845 0.53 MB/sec execute 72 sec latency 35.276 ms oleg309-client.virtnet: 1 10950 0.52 MB/sec execute 73 sec latency 51.294 ms oleg309-client.virtnet: 1 11061 0.52 MB/sec execute 74 sec latency 46.846 ms oleg309-client.virtnet: 1 11169 0.51 MB/sec execute 75 sec latency 51.798 ms oleg309-client.virtnet: 1 11346 0.51 MB/sec execute 76 sec latency 45.911 ms oleg309-client.virtnet: 1 11438 0.50 MB/sec execute 77 sec latency 53.864 ms oleg309-client.virtnet: 1 11459 0.50 MB/sec execute 78 sec latency 588.815 ms oleg309-client.virtnet: 1 11459 0.49 MB/sec execute 79 sec latency 1590.219 ms oleg309-client.virtnet: 1 11459 0.48 MB/sec execute 80 sec latency 2591.660 ms oleg309-client.virtnet: 1 11459 0.48 MB/sec execute 81 sec latency 3592.058 ms oleg309-client.virtnet: 1 11459 0.47 MB/sec execute 82 sec latency 4592.318 ms oleg309-client.virtnet: 1 11544 0.47 MB/sec execute 83 sec latency 4861.295 ms oleg309-client.virtnet: 1 11638 0.47 MB/sec execute 84 sec latency 49.139 ms oleg309-client.virtnet: 1 11803 0.47 MB/sec execute 85 sec latency 70.422 ms oleg309-client.virtnet: 1 12047 0.50 MB/sec execute 86 sec latency 44.663 ms oleg309-client.virtnet: 1 12172 0.50 MB/sec execute 87 sec latency 78.095 ms oleg309-client.virtnet: 1 12298 0.49 MB/sec execute 88 sec latency 59.908 ms oleg309-client.virtnet: 1 12386 0.49 MB/sec execute 89 sec latency 62.685 ms oleg309-client.virtnet: 1 12486 0.48 MB/sec execute 90 sec latency 54.902 ms oleg309-client.virtnet: 1 12542 0.48 MB/sec execute 91 sec latency 502.145 ms oleg309-client.virtnet: 1 12670 0.48 MB/sec execute 92 sec latency 760.256 ms oleg309-client.virtnet: 1 12819 0.47 MB/sec execute 93 sec latency 48.258 ms oleg309-client.virtnet: 1 12944 0.48 MB/sec execute 94 sec latency 86.394 ms oleg309-client.virtnet: 1 13043 0.48 MB/sec execute 95 sec latency 75.926 ms oleg309-client.virtnet: 1 13054 0.47 MB/sec execute 96 sec latency 906.627 ms oleg309-client.virtnet: 1 13054 0.47 MB/sec execute 97 sec latency 1906.774 ms oleg309-client.virtnet: 1 13054 0.46 MB/sec execute 98 sec latency 2906.969 ms oleg309-client.virtnet: 1 13054 0.46 MB/sec execute 99 sec latency 3907.848 ms oleg309-client.virtnet: 1 13054 0.45 MB/sec execute 100 sec latency 4908.029 ms oleg309-client.virtnet: 1 13054 0.45 MB/sec execute 101 sec latency 5909.623 ms oleg309-client.virtnet: 1 13054 0.44 MB/sec execute 102 sec latency 6912.062 ms oleg309-client.virtnet: 1 13054 0.44 MB/sec execute 103 sec latency 7912.403 ms oleg309-client.virtnet: 1 13054 0.43 MB/sec execute 104 sec latency 8912.562 ms oleg309-client.virtnet: 1 13054 0.43 MB/sec execute 105 sec latency 9912.936 ms oleg309-client.virtnet: 1 13054 0.43 MB/sec execute 106 sec latency 10913.197 ms oleg309-client.virtnet: 1 13054 0.42 MB/sec execute 107 sec latency 11913.365 ms oleg309-client.virtnetStarting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 18:56:29 (1746140189) targets are mounted 18:56:29 (1746140189) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37660 3566788 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12032 3594852 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49692 7161640 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg309-server Stopping /mnt/lustre-mds2 (opts:) on oleg309-server 18:56:46 (1746140206) shut down facet: mds2 facet_host: oleg309-server facet_failover_host: oleg309-server Failover mds2 to oleg309-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0001 18:57:04 (1746140224) targets are mounted 18:57:04 (1746140224) facet_failover done : 1 13054 0.42 MB/sec execute 108 sec latency 12913.514 ms oleg309-client.virtnet: 1 13054 0.41 MB/sec execute 109 sec latency 13913.692 ms oleg309-client.virtnet: 1 13054 0.41 MB/sec execute 110 sec latency 14920.594 ms oleg309-client.virtnet: 1 13054 0.41 MB/sec execute 111 sec latency 15920.717 ms oleg309-client.virtnet: 1 13054 0.40 MB/sec execute 112 sec latency 16921.891 ms oleg309-client.virtnet: 1 13054 0.40 MB/sec execute 113 sec latency 17922.098 ms oleg309-client.virtnet: 1 13054 0.40 MB/sec execute 114 sec latency 18924.221 ms oleg309-client.virtnet: 1 13054 0.39 MB/sec execute 115 sec latency 19924.400 ms oleg309-client.virtnet: 1 13054 0.39 MB/sec execute 116 sec latency 20924.996 ms oleg309-client.virtnet: 1 13054 0.39 MB/sec execute 117 sec latency 21925.178 ms oleg309-client.virtnet: 1 13139 0.39 MB/sec execute 118 sec latency 22211.010 ms oleg309-client.virtnet: 1 13285 0.39 MB/sec execute 119 sec latency 60.423 ms oleg309-client.virtnet: 1 13625 0.42 MB/sec execute 120 sec latency 50.670 ms oleg309-client.virtnet: 1 14010 0.44 MB/sec execute 121 sec latency 30.802 ms oleg309-client.virtnet: 1 14229 0.46 MB/sec execute 122 sec latency 48.727 ms oleg309-client.virtnet: 1 14378 0.46 MB/sec execute 123 sec latency 49.215 ms oleg309-client.virtnet: 1 14487 0.46 MB/sec execute 124 sec latency 61.023 ms oleg309-client.virtnet: 1 14593 0.46 MB/sec execute 125 sec latency 47.002 ms oleg309-client.virtnet: 1 14678 0.45 MB/sec execute 126 sec latency 44.335 ms oleg309-client.virtnet: 1 14808 0.45 MB/sec execute 127 sec latency 57.714 ms oleg309-client.virtnet: 1 14940 0.45 MB/sec execute 128 sec latency 63.304 ms oleg309-client.virtnet: 1 15023 0.45 MB/sec execute 129 sec latency 85.124 ms oleg309-client.virtnet: 1 15127 0.45 MB/sec execute 130 sec latency 84.911 ms oleg309-client.virtnet: 1 15265 0.45 MB/sec execute 131 sec latency 55.108 ms oleg309-client.virtnet: 1 15469 0.45 MB/sec execute 132 sec latency 20.572 ms oleg309-client.virtnet: 1 15727 0.47 MB/sec execute 133 sec latency 34.342 ms oleg309-client.virtnet: 1 15887 0.47 MB/sec execute 134 sec latency 47.214 ms oleg309-client.virtnet: 1 16046 0.46 MB/sec execute 135 sec latency 29.999 ms oleg309-client.virtnet: 1 16266 0.46 MB/sec execute 136 sec latency 25.919 ms oleg309-client.virtnet: 1 16543 0.47 MB/sec execute 137 sec latency 31.732 ms oleg309-client.virtnet: 1 16777 0.47 MB/sec execute 138 sec latency 28.683 ms oleg309-client.virtnet: 1 17188 0.50 MB/sec execute 139 sec latency 23.586 ms oleg309-client.virtnet: 1 17590 0.51 MB/sec execute 140 sec latency 31.585 ms oleg309-client.virtnet: 1 17850 0.53 MB/sec execute 141 sec latency 53.729 ms oleg309-client.virtnet: 1 17990 0.54 MB/sec execute 142 sec latency 40.204 ms oleg309-client.virtnet: 1 18103 0.53 MB/sec execute 143 sec latency 57.402 ms oleg309-client.virtnet: 1 18189 0.53 MB/sec execute 144 sec latency 44.759 ms oleg309-client.virtnet: 1 18266 0.53 MB/sec execute 145 sec latency 126.985 ms oleg309-client.virtnet: 1 18266 0.52 MB/sec execute 146 sec latency 1127.519 ms oleg309-client.virtnet: 1 18266 0.52 MB/sec execute 147 sec latency 2127.688 ms oleg309-client.virtnet: 1 18266 0.52 MB/sec execute 148 sec latency 3128.461 ms oleg309-client.virtnet: 1 18266 0.51 MB/sec execute 149 sec latency 4129.509 ms oleg309-client.virtnet: 1 18266 0.51 MB/sec execute 150 sec latency 5129.827 ms oleg309-client.virtnet: 1 18266 0.51 MB/sec execute 151 sec latency 6131.822 ms oleg309-client.virtnet: 1 18392 0.50 MB/sec execute 152 oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37660 3569184 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11904 3593724 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7162908 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg309-server Stopping /mnt/lustre-mds1 (opts:) on oleg309-server 18:57:26 (1746140246) shut down facet: mds1 facet_host: oleg309-server facet_failover_host: oleg309-server Failover mds1 to oleg309-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 18:57:46 (1746140266) targets are mounted 18:57:46 (1746140266) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec sec latency 6573.257 ms oleg309-client.virtnet: 1 18495 0.50 MB/sec execute 153 sec latency 40.680 ms oleg309-client.virtnet: 1 18602 0.50 MB/sec execute 154 sec latency 76.888 ms oleg309-client.virtnet: 1 18707 0.50 MB/sec execute 155 sec latency 62.645 ms oleg309-client.virtnet: 1 18863 0.50 MB/sec execute 156 sec latency 47.489 ms oleg309-client.virtnet: 1 18994 0.50 MB/sec execute 157 sec latency 41.224 ms oleg309-client.virtnet: 1 19179 0.52 MB/sec execute 158 sec latency 47.830 ms oleg309-client.virtnet: 1 19289 0.51 MB/sec execute 159 sec latency 61.676 ms oleg309-client.virtnet: 1 19382 0.51 MB/sec execute 160 sec latency 127.268 ms oleg309-client.virtnet: 1 19403 0.51 MB/sec execute 161 sec latency 843.614 ms oleg309-client.virtnet: 1 19531 0.50 MB/sec execute 162 sec latency 42.709 ms oleg309-client.virtnet: 1 19692 0.50 MB/sec execute 163 sec latency 37.971 ms oleg309-client.virtnet: 1 19831 0.50 MB/sec execute 164 sec latency 62.102 ms oleg309-client.virtnet: 1 19846 0.50 MB/sec execute 165 sec latency 932.855 ms oleg309-client.virtnet: 1 19846 0.49 MB/sec execute 166 sec latency 1933.373 ms oleg309-client.virtnet: 1 19846 0.49 MB/sec execute 167 sec latency 2934.562 ms oleg309-client.virtnet: 1 19846 0.49 MB/sec execute 168 sec latency 3936.159 ms oleg309-client.virtnet: 1 19846 0.49 MB/sec execute 169 sec latency 4936.372 ms oleg309-client.virtnet: 1 19846 0.48 MB/sec execute 170 sec latency 5936.506 ms oleg309-client.virtnet: 1 19846 0.48 MB/sec execute 171 sec latency 6936.653 ms oleg309-client.virtnet: 1 19846 0.48 MB/sec execute 172 sec latency 7937.111 ms oleg309-client.virtnet: 1 19846 0.47 MB/sec execute 173 sec latency 8937.349 ms oleg309-client.virtnet: 1 19846 0.47 MB/sec execute 174 sec latency 9938.438 ms oleg309-client.virtnet: 1 19846 0.47 MB/sec execute 175 sec latency 10938.716 ms oleg309-client.virtnet: 1 19846 0.47 MB/sec execute 176 sec latency 11940.149 ms oleg309-client.virtnet: 1 19846 0.46 MB/sec execute 177 sec latency 12941.339 ms oleg309-client.virtnet: 1 19846 0.46 MB/sec execute 178 sec latency 13941.741 ms oleg309-client.virtnet: 1 19846 0.46 MB/sec execute 179 sec latency 14943.371 ms oleg309-client.virtnet: 1 19846 0.46 MB/sec execute 180 sec latency 15944.025 ms oleg309-client.virtnet: 1 19846 0.45 MB/sec execute 181 sec latency 16945.319 ms oleg309-client.virtnet: 1 19846 0.45 MB/sec execute 182 sec latency 17945.491 ms oleg309-client.virtnet: 1 19846 0.45 MB/sec execute 183 sec latency 18945.632 ms oleg309-client.virtnet: 1 19846 0.45 MB/sec execute 184 sec latency 19946.932 ms oleg309-client.virtnet: 1 19846 0.44 MB/sec execute 185 sec latency 20947.223 ms oleg309-client.virtnet: 1 19846 0.44 MB/sec execute 186 sec latency 21948.119 ms oleg309-client.virtnet: 1 19846 0.44 MB/sec execute 187 sec latency 22948.413 ms oleg309-client.virtnet: 1 19846 0.44 MB/sec execute 188 sec latency 23949.701 ms oleg309-client.virtnet: 1 19846 0.43 MB/sec execute 189 sec latency 24949.874 ms oleg309-client.virtnet: 1 19846 0.43 MB/sec execute 190 sec latency 25950.568 ms oleg309-client.virtnet: 1 19846 0.43 MB/sec execute 191 sec latency 26952.114 ms oleg309-client.virtnet: 1 19846 0.43 MB/sec execute 192 sec latency 27952.347 ms oleg309-client.virtnet: 1 19902 0.43 MB/sec execute 193 sec latency 28389.798 ms oleg309-client.virtnet: 1 19978 0.42 MB/sec execute 194 sec latency 80.674 ms oleg309-client.virtnet: 1 20132 0.43 MB/sec execute 195 sec latency 61.878 ms oleg309-client.virtnet: 1 20239 0.43 MB/sec execute 196 sec latencyUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37716 3568244 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11996 3594260 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49712 7162504 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg309-server Stopping /mnt/lustre-mds2 (opts:) on oleg309-server 18:58:01 (1746140281) shut down facet: mds2 facet_host: oleg309-server facet_failover_host: oleg309-server Failover mds2 to oleg309-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0001 18:58:19 (1746140299) targets are mounted 18:58:19 (1746140299) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37584 3569040 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11904 3594312 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163352 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg309-server Stopping /mnt/lustre-mds1 (opts:) on oleg309-server 18:58:35 (1746140315) shut down facet: mds1 facet_host: oleg309-server facet_failover_host: oleg309-server 73.813 ms oleg309-client.virtnet: 1 20662 0.45 MB/sec execute 197 sec latency 28.438 ms oleg309-client.virtnet: 1 20990 0.45 MB/sec execute 198 sec latency 34.051 ms oleg309-client.virtnet: 1 21270 0.47 MB/sec execute 199 sec latency 48.873 ms oleg309-client.virtnet: 1 21409 0.47 MB/sec execute 200 sec latency 36.091 ms oleg309-client.virtnet: 1 21554 0.47 MB/sec execute 201 sec latency 51.233 ms oleg309-client.virtnet: 1 21654 0.47 MB/sec execute 202 sec latency 52.542 ms oleg309-client.virtnet: 1 21808 0.47 MB/sec execute 203 sec latency 58.471 ms oleg309-client.virtnet: 1 21945 0.47 MB/sec execute 204 sec latency 40.562 ms oleg309-client.virtnet: 1 22049 0.47 MB/sec execute 205 sec latency 55.301 ms oleg309-client.virtnet: 1 22156 0.47 MB/sec execute 206 sec latency 44.697 ms oleg309-client.virtnet: 1 22273 0.47 MB/sec execute 207 sec latency 53.488 ms oleg309-client.virtnet: 1 22477 0.47 MB/sec execute 208 sec latency 40.701 ms oleg309-client.virtnet: 1 22734 0.48 MB/sec execute 209 sec latency 29.348 ms oleg309-client.virtnet: 1 22921 0.48 MB/sec execute 210 sec latency 33.877 ms oleg309-client.virtnet: 1 23077 0.48 MB/sec execute 211 sec latency 37.395 ms oleg309-client.virtnet: 1 23273 0.48 MB/sec execute 212 sec latency 31.361 ms oleg309-client.virtnet: 1 23477 0.48 MB/sec execute 213 sec latency 33.856 ms oleg309-client.virtnet: 1 23704 0.48 MB/sec execute 214 sec latency 34.720 ms oleg309-client.virtnet: 1 23921 0.48 MB/sec execute 215 sec latency 32.969 ms oleg309-client.virtnet: 1 24310 0.50 MB/sec execute 216 sec latency 29.151 ms oleg309-client.virtnet: 1 24672 0.51 MB/sec execute 217 sec latency 43.625 ms oleg309-client.virtnet: 1 24896 0.52 MB/sec execute 218 sec latency 70.143 ms oleg309-client.virtnet: 1 25011 0.52 MB/sec execute 219 sec latency 45.486 ms oleg309-client.virtnet: 1 25155 0.52 MB/sec execute 220 sec latency 75.710 ms oleg309-client.virtnet: 1 25257 0.52 MB/sec execute 221 sec latency 65.241 ms oleg309-client.virtnet: 1 25360 0.52 MB/sec execute 222 sec latency 82.097 ms oleg309-client.virtnet: 1 25394 0.51 MB/sec execute 223 sec latency 606.218 ms oleg309-client.virtnet: 1 25394 0.51 MB/sec execute 224 sec latency 1606.427 ms oleg309-client.virtnet: 1 25394 0.51 MB/sec execute 225 sec latency 2606.575 ms oleg309-client.virtnet: 1 25394 0.51 MB/sec execute 226 sec latency 3606.721 ms oleg309-client.virtnet: 1 25425 0.51 MB/sec execute 227 sec latency 4462.719 ms oleg309-client.virtnet: 1 25588 0.50 MB/sec execute 228 sec latency 39.699 ms oleg309-client.virtnet: 1 25685 0.50 MB/sec execute 229 sec latency 54.415 ms oleg309-client.virtnet: 1 25833 0.50 MB/sec execute 230 sec latency 34.311 ms oleg309-client.virtnet: 1 26029 0.51 MB/sec execute 231 sec latency 31.392 ms oleg309-client.virtnet: 1 26280 0.52 MB/sec execute 232 sec latency 41.892 ms oleg309-client.virtnet: 1 26406 0.51 MB/sec execute 233 sec latency 42.704 ms oleg309-client.virtnet: 1 26549 0.51 MB/sec execute 234 sec latency 34.315 ms oleg309-client.virtnet: 1 26676 0.51 MB/sec execute 235 sec latency 50.132 ms oleg309-client.virtnet: 1 26867 0.51 MB/sec execute 236 sec latency 35.532 ms oleg309-client.virtnet: 1 27040 0.51 MB/sec execute 237 sec latency 44.889 ms oleg309-client.virtnet: 1 27218 0.51 MB/sec execute 238 sec latency 47.680 ms oleg309-client.virtnet: 1 27249 0.51 MB/sec execute 239 sec latency 659.034 ms oleg309-client.virtnet: 1 27249 0.51 MB/sec execute 240 sec latency 1659.880 ms oleg309-client.virtnet: 1 27249 0.50 MB/sec execute 241 sFailover mds1 to oleg309-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 18:58:53 (1746140333) targets are mounted 18:58:53 (1746140333) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec ec latency 2660.326 ms oleg309-client.virtnet: 1 27249 0.50 MB/sec execute 242 sec latency 3661.930 ms oleg309-client.virtnet: 1 27249 0.50 MB/sec execute 243 sec latency 4662.998 ms oleg309-client.virtnet: 1 27249 0.50 MB/sec execute 244 sec latency 5667.287 ms oleg309-client.virtnet: 1 27249 0.50 MB/sec execute 245 sec latency 6667.895 ms oleg309-client.virtnet: 1 27249 0.49 MB/sec execute 246 sec latency 7671.094 ms oleg309-client.virtnet: 1 27249 0.49 MB/sec execute 247 sec latency 8671.295 ms oleg309-client.virtnet: 1 27249 0.49 MB/sec execute 248 sec latency 9672.172 ms oleg309-client.virtnet: 1 27249 0.49 MB/sec execute 249 sec latency 10672.646 ms oleg309-client.virtnet: 1 27249 0.49 MB/sec execute 250 sec latency 11673.850 ms oleg309-client.virtnet: 1 27249 0.48 MB/sec execute 251 sec latency 12674.207 ms oleg309-client.virtnet: 1 27249 0.48 MB/sec execute 252 sec latency 13675.171 ms oleg309-client.virtnet: 1 27249 0.48 MB/sec execute 253 sec latency 14675.328 ms oleg309-client.virtnet: 1 27249 0.48 MB/sec execute 254 sec latency 15675.468 ms oleg309-client.virtnet: 1 27249 0.48 MB/sec execute 255 sec latency 16675.908 ms oleg309-client.virtnet: 1 27249 0.47 MB/sec execute 256 sec latency 17677.808 ms oleg309-client.virtnet: 1 27249 0.47 MB/sec execute 257 sec latency 18678.242 ms oleg309-client.virtnet: 1 27249 0.47 MB/sec execute 258 sec latency 19679.427 ms oleg309-client.virtnet: 1 27249 0.47 MB/sec execute 259 sec latency 20679.833 ms oleg309-client.virtnet: 1 27249 0.47 MB/sec execute 260 sec latency 21681.466 ms oleg309-client.virtnet: 1 27288 0.47 MB/sec execute 261 sec latency 22254.958 ms oleg309-client.virtnet: 1 27458 0.47 MB/sec execute 262 sec latency 43.741 ms oleg309-client.virtnet: 1 27768 0.48 MB/sec execute 263 sec latency 57.251 ms oleg309-client.virtnet: 1 28083 0.48 MB/sec execute 264 sec latency 56.256 ms oleg309-client.virtnet: 1 28429 0.50 MB/sec execute 265 sec latency 28.421 ms oleg309-client.virtnet: 1 28632 0.50 MB/sec execute 266 sec latency 43.768 ms oleg309-client.virtnet: 1 28782 0.50 MB/sec execute 267 sec latency 29.145 ms oleg309-client.virtnet: 1 28938 0.50 MB/sec execute 268 sec latency 30.908 ms oleg309-client.virtnet: 1 29156 0.50 MB/sec execute 269 sec latency 57.315 ms oleg309-client.virtnet: 1 29303 0.50 MB/sec execute 270 sec latency 39.360 ms oleg309-client.virtnet: 1 29476 0.50 MB/sec execute 271 sec latency 39.511 ms oleg309-client.virtnet: 1 29650 0.50 MB/sec execute 272 sec latency 31.300 ms oleg309-client.virtnet: 1 29889 0.51 MB/sec execute 273 sec latency 33.334 ms oleg309-client.virtnet: 1 30042 0.51 MB/sec execute 274 sec latency 31.115 ms oleg309-client.virtnet: 1 30198 0.51 MB/sec execute 275 sec latency 39.137 ms oleg309-client.virtnet: 1 30366 0.50 MB/sec execute 276 sec latency 39.780 ms oleg309-client.virtnet: 1 30579 0.50 MB/sec execute 277 sec latency 31.321 ms oleg309-client.virtnet: 1 30795 0.51 MB/sec execute 278 sec latency 33.907 ms oleg309-client.virtnet: 1 30921 0.51 MB/sec execute 279 sec latency 52.948 ms oleg309-client.virtnet: 1 31294 0.52 MB/sec execute 280 sec latency 39.466 ms oleg309-client.virtnet: 1 31591 0.52 MB/sec execute 281 sec latency 53.397 ms oleg309-client.virtnet: 1 31942 0.54 MB/sec execute 282 sec latency 36.304 ms oleg309-client.virtnet: 1 32120 0.54 MB/sec execute 283 sec latency 23.056 ms oleg309-client.virtnet: 1 32277 0.54 MB/sec execute 284 sec latency 34.730 ms oleg309-client.virtnet: 1 32428 0.54 MB/sec execute 285 sec latency 38.513 ms oleg309-client.virtnet: 1 32631 0.53 MB/sec execute 286 sec latency 37.611 ms oleg309-client.virtnet: 1 32765 0.53 MB/sec execute 287 sec latency 54.901 ms oleg309-client.virtnet: 1 32923 0.53 MB/sec execute 288 sec latency 44.402 ms oleg309-client.virtnet: 1 33119 0.54 MB/sec execute 289 sec latency 45.531 ms oleg309-client.virtnet: 1 33360 0.54 MB/sec execute 290 sec latency 43.917 ms oleg309-client.virtnet: 1 33530 0.54 MB/sec execute 291 sec latency 37.295 ms oleg309-client.virtnet: 1 33695 0.54 MB/sec execute 292 sec latency 33.631 ms oleg309-client.virtnet: 1 33875 0.54 MB/sec execute 293 sec latency 30.496 ms oleg309-client.virtnet: 1 34121 0.54 MB/sec execute 294 sec latency 23.388 ms oleg309-client.virtnet: 1 34352 0.54 MB/sec execute 295 sec latency 32.084 ms oleg309-client.virtnet: 1 34550 0.54 MB/sec execute 296 sec latency 40.884 ms oleg309-client.virtnet: 1 34937 0.56 MB/sec execute 297 sec latency 34.337 ms oleg309-client.virtnet: 1 35309 0.56 MB/sec execute 298 sec latency 45.327 ms oleg309-client.virtnet: 1 35571 0.57 MB/sec execute 299 sec latency 31.839 ms oleg309-client.virtnet: 1 cleanup 300 sec oleg309-client.virtnet: 0 cleanup 301 sec oleg309-client.virtnet: oleg309-client.virtnet: Operation Count AvgLat MaxLat oleg309-client.virtnet: ---------------------------------------- oleg309-client.virtnet: NTCreateX 5408 28.620 28388.874 oleg309-client.virtnet: Close 4021 11.893 32367.883 oleg309-client.virtnet: Rename 234 25.350 74.271 oleg309-client.virtnet: Unlink 1086 6.047 34.645 oleg309-client.virtnet: Qpathinfo 4954 9.040 17339.272 oleg309-client.virtnet: Qfileinfo 872 0.258 9.320 oleg309-client.virtnet: Qfsinfo 910 0.603 33.306 oleg309-client.virtnet: Sfileinfo 450 77.368 22254.946 oleg309-client.virtnet: Find 1923 0.909 33.266 oleg309-client.virtnet: WriteX 2722 2.639 36.765 oleg309-client.virtnet: ReadX 8663 0.048 43.756 oleg309-client.virtnet: LockX 18 3.625 9.180 oleg309-client.virtnet: UnlockX 18 3.521 5.955 oleg309-client.virtnet: Flush 387 24.880 843.600 oleg309-client.virtnet: oleg309-client.virtnet: Throughput 0.57022 MB/sec 1 clients 1 procs max_latency=32367.909 ms oleg309-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg309-client.virtnet at Thu May 1 18:59:36 EDT 2025 with return code 0 oleg309-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg309-client.virtnet oleg309-client.virtnet: /mnt/lustre/d70b.replay-single/oleg309-client.virtnet /mnt/lustre/d70b.replay-single/oleg309-client.virtnet oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg309-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg309-client.virtnet: removed directory: 'clients/client0' oleg309-client.virtnet: removed directory: 'clients' oleg309-client.virtnet: removed 'client.txt' oleg309-client.virtnet: /mnt/lustre/d70b.replay-single/oleg309-client.virtnet oleg309-client.virtnet: dbench successfully finished