== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 00:34:35 (1746074075) Starting client oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre Started clients oleg403-client.virtnet: 192.168.204.103@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg403-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=6352 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 28672 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7479296 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server 00:34:43 (1746074083) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:34:57 (1746074097) targets are mounted 00:34:57 (1746074097) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3732480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3681280 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7413760 1% /mnt/lustre oleg403-client.virtnet: looking for dbench program oleg403-client.virtnet: /usr/bin/dbench oleg403-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg403-client.virtnet oleg403-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg403-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg403-client.virtnet' oleg403-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg403-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg403-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg403-client.virtnet at Thu May 1 00:34:36 EDT 2025 oleg403-client.virtnet: waiting for dbench pid 6376 oleg403-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg403-client.virtnet: oleg403-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg403-client.virtnet: failed to create barrier semaphore oleg403-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg403-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg403-client.virtnet: releasing clients oleg403-client.virtnet: 1 208 8.09 MB/sec warmup 1 sec latency 71.583 ms oleg403-client.virtnet: 1 422 7.42 MB/sec warmup 2 sec latency 33.831 ms oleg403-client.virtnet: 1 666 7.05 MB/sec warmup 3 sec latency 77.384 ms oleg403-client.virtnet: 1 804 5.37 MB/sec warmup 4 sec latency 50.012 ms oleg403-client.virtnet: 1 985 4.38 MB/sec warmup 5 sec latency 71.841 ms oleg403-client.virtnet: 1 1172 3.83 MB/sec warmup 6 sec latency 264.692 ms oleg403-client.virtnet: 1 1172 3.29 MB/sec warmup 7 sec latency 1266.355 ms oleg403-client.virtnet: 1 1172 2.87 MB/sec warmup 8 sec latency 2268.203 ms oleg403-client.virtnet: 1 1172 2.55 MB/sec warmup 9 sec latency 3269.173 ms oleg403-client.virtnet: 1 1172 2.30 MB/sec warmup 10 sec latency 4269.342 ms oleg403-client.virtnet: 1 1172 2.09 MB/sec warmup 11 sec latency 5269.503 ms oleg403-client.virtnet: 1 1172 1.92 MB/sec warmup 12 sec latency 6270.024 ms oleg403-client.virtnet: 1 1172 1.77 MB/sec warmup 13 sec latency 7270.788 ms oleg403-client.virtnet: 1 1172 1.64 MB/sec warmup 14 sec latency 8270.944 ms oleg403-client.virtnet: 1 1172 1.53 MB/sec warmup 15 sec latency 9271.534 ms oleg403-client.virtnet: 1 1172 1.44 MB/sec warmup 16 sec latency 10272.019 ms oleg403-client.virtnet: 1 1172 1.35 MB/sec warmup 17 sec latency 11272.432 ms oleg403-client.virtnet: 1 1172 1.28 MB/sec warmup 18 sec latency 12272.552 ms oleg403-client.virtnet: 1 1172 1.21 MB/sec warmup 19 sec latency 13272.669 ms oleg403-client.virtnet: 1 1172 1.15 MB/sec warmup 20 sec latency 14273.554 ms oleg403-client.virtnet: 1 1172 1.10 MB/sec warmup 21 sec latency 15274.555 ms oleg403-client.virtnet: 1 1172 1.05 MB/sec warmup 22 sec latency 16282.825 ms oleg403-client.virtnet: 1 1172 1.00 MB/sec warmup 23 sec latency 17283.014 ms oleg403-client.virtnet: 1 1172 0.96 MB/sec warmup 24 sec latency 18283.233 ms oleg403-client.virtnet: 1 1172 0.92 MB/sec warmup 25 sec latency 19283.395 ms oleg403-client.virtnet: 1 1173 0.88 MB/sec warmup 26 sec latency 20256.334 ms oleg403-client.virtnet: 1 1396 0.96 MB/sec warmup 27 sec latency 52.331 ms oleg403-client.virtnet: 1 1518 0.93 MB/sec warmup 28 sec latency 91.536 ms oleg403-client.virtnet: 1 1687 0.90 MB/sec warmup 29 sec latency 77.787 ms oleg403-client.virtnet: 1 1888 0.88 MB/sec warmup 30 sec latency 88.794 ms oleg403-client.virtnet: 1 2107 0.86 MB/sec warmup 31 sec latency 52.537 ms oleg403-client.virtnet: 1 2235 0.84 MB/sec warmup 32 sec latency 55.304 test_70b fail mds1 2 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server 00:35:13 (1746074113) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:35:28 (1746074128) targets are mounted 00:35:28 (1746074128) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3741696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3689472 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7431168 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server 00:35:43 (1746074143) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 ms oleg403-client.virtnet: 1 2388 0.85 MB/sec warmup 33 sec latency 96.400 ms oleg403-client.virtnet: 1 2497 0.84 MB/sec warmup 34 sec latency 116.866 ms oleg403-client.virtnet: 1 2768 0.91 MB/sec warmup 35 sec latency 106.173 ms oleg403-client.virtnet: 1 2922 0.93 MB/sec warmup 36 sec latency 453.212 ms oleg403-client.virtnet: 1 2922 0.90 MB/sec warmup 37 sec latency 1453.991 ms oleg403-client.virtnet: 1 2922 0.88 MB/sec warmup 38 sec latency 2454.115 ms oleg403-client.virtnet: 1 2922 0.86 MB/sec warmup 39 sec latency 3454.744 ms oleg403-client.virtnet: 1 2922 0.84 MB/sec warmup 40 sec latency 4455.102 ms oleg403-client.virtnet: 1 2922 0.82 MB/sec warmup 41 sec latency 5455.991 ms oleg403-client.virtnet: 1 2922 0.80 MB/sec warmup 42 sec latency 6456.150 ms oleg403-client.virtnet: 1 2922 0.78 MB/sec warmup 43 sec latency 7458.386 ms oleg403-client.virtnet: 1 2922 0.76 MB/sec warmup 44 sec latency 8459.616 ms oleg403-client.virtnet: 1 2922 0.74 MB/sec warmup 45 sec latency 9460.070 ms oleg403-client.virtnet: 1 2922 0.73 MB/sec warmup 46 sec latency 10463.939 ms oleg403-client.virtnet: 1 2922 0.71 MB/sec warmup 47 sec latency 11464.209 ms oleg403-client.virtnet: 1 2922 0.70 MB/sec warmup 48 sec latency 12464.314 ms oleg403-client.virtnet: 1 2922 0.68 MB/sec warmup 49 sec latency 13465.757 ms oleg403-client.virtnet: 1 2922 0.67 MB/sec warmup 50 sec latency 14465.959 ms oleg403-client.virtnet: 1 2922 0.66 MB/sec warmup 51 sec latency 15466.197 ms oleg403-client.virtnet: 1 2922 0.64 MB/sec warmup 52 sec latency 16466.972 ms oleg403-client.virtnet: 1 2922 0.63 MB/sec warmup 53 sec latency 17468.461 ms oleg403-client.virtnet: 1 2922 0.62 MB/sec warmup 54 sec latency 18468.621 ms oleg403-client.virtnet: 1 2922 0.61 MB/sec warmup 55 sec latency 19470.692 ms oleg403-client.virtnet: 1 2925 0.60 MB/sec warmup 56 sec latency 20427.155 ms oleg403-client.virtnet: 1 3172 0.60 MB/sec warmup 57 sec latency 129.006 ms oleg403-client.virtnet: 1 3347 0.61 MB/sec warmup 58 sec latency 112.729 ms oleg403-client.virtnet: 1 3632 0.65 MB/sec warmup 59 sec latency 39.579 ms oleg403-client.virtnet: 1 3955 0.30 MB/sec execute 1 sec latency 49.407 ms oleg403-client.virtnet: 1 4065 0.16 MB/sec execute 2 sec latency 38.122 ms oleg403-client.virtnet: 1 4241 0.17 MB/sec execute 3 sec latency 93.151 ms oleg403-client.virtnet: 1 4360 0.20 MB/sec execute 4 sec latency 83.881 ms oleg403-client.virtnet: 1 4448 0.20 MB/sec execute 5 sec latency 104.070 ms oleg403-client.virtnet: 1 4487 0.19 MB/sec execute 6 sec latency 604.444 ms oleg403-client.virtnet: 1 4487 0.16 MB/sec execute 7 sec latency 1604.586 ms oleg403-client.virtnet: 1 4487 0.14 MB/sec execute 8 sec latency 2604.776 ms oleg403-client.virtnet: 1 4487 0.13 MB/sec execute 9 sec latency 3605.119 ms oleg403-client.virtnet: 1 4487 0.11 MB/sec execute 10 sec latency 4605.408 ms oleg403-client.virtnet: 1 4487 0.10 MB/sec execute 11 sec latency 5608.430 ms oleg403-client.virtnet: 1 4487 0.09 MB/sec execute 12 sec latency 6608.589 ms oleg403-client.virtnet: 1 4487 0.09 MB/sec execute 13 sec latency 7609.814 ms oleg403-client.virtnet: 1 4487 0.08 MB/sec execute 14 sec latency 8610.651 ms oleg403-client.virtnet: 1 4487 0.08 MB/sec execute 15 sec latency 9610.801 ms oleg403-client.virtnet: 1 4487 0.07 MB/sec execute 16 sec latency 10611.550 ms oleg403-client.virtnet: 1 4487 0.07 MB/sec execute 17 sec latency 11611.786 ms oleg403-client.virtnet: 1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:35:57 (1746074157) targets are mounted 00:35:57 (1746074157) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3722240 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7468032 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server 00:36:13 (1746074173) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:36:28 (1746074188) targets are mounted 00:36:28 (1746074188) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3732480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3709952 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7442432 1% /mnt/lustre 4487 0.06 MB/sec execute 18 sec latency 12612.708 ms oleg403-client.virtnet: 1 4487 0.06 MB/sec execute 19 sec latency 13612.832 ms oleg403-client.virtnet: 1 4487 0.06 MB/sec execute 20 sec latency 14612.989 ms oleg403-client.virtnet: 1 4487 0.05 MB/sec execute 21 sec latency 15613.734 ms oleg403-client.virtnet: 1 4487 0.05 MB/sec execute 22 sec latency 16613.967 ms oleg403-client.virtnet: 1 4487 0.05 MB/sec execute 23 sec latency 17614.855 ms oleg403-client.virtnet: 1 4487 0.05 MB/sec execute 24 sec latency 18614.995 ms oleg403-client.virtnet: 1 4487 0.05 MB/sec execute 25 sec latency 19615.146 ms oleg403-client.virtnet: 1 4488 0.04 MB/sec execute 26 sec latency 20613.012 ms oleg403-client.virtnet: 1 4557 0.05 MB/sec execute 27 sec latency 119.093 ms oleg403-client.virtnet: 1 4701 0.08 MB/sec execute 28 sec latency 46.879 ms oleg403-client.virtnet: 1 4827 0.08 MB/sec execute 29 sec latency 48.432 ms oleg403-client.virtnet: 1 5067 0.18 MB/sec execute 30 sec latency 113.099 ms oleg403-client.virtnet: 1 5208 0.18 MB/sec execute 31 sec latency 97.630 ms oleg403-client.virtnet: 1 5346 0.18 MB/sec execute 32 sec latency 36.667 ms oleg403-client.virtnet: 1 5453 0.17 MB/sec execute 33 sec latency 57.068 ms oleg403-client.virtnet: 1 5650 0.18 MB/sec execute 34 sec latency 79.688 ms oleg403-client.virtnet: 1 5857 0.20 MB/sec execute 35 sec latency 50.658 ms oleg403-client.virtnet: 1 5954 0.20 MB/sec execute 36 sec latency 277.745 ms oleg403-client.virtnet: 1 5954 0.20 MB/sec execute 37 sec latency 1278.418 ms oleg403-client.virtnet: 1 5954 0.19 MB/sec execute 38 sec latency 2278.638 ms oleg403-client.virtnet: 1 5954 0.19 MB/sec execute 39 sec latency 3278.940 ms oleg403-client.virtnet: 1 5954 0.18 MB/sec execute 40 sec latency 4279.435 ms oleg403-client.virtnet: 1 5954 0.18 MB/sec execute 41 sec latency 5279.610 ms oleg403-client.virtnet: 1 5954 0.17 MB/sec execute 42 sec latency 6280.837 ms oleg403-client.virtnet: 1 5954 0.17 MB/sec execute 43 sec latency 7280.988 ms oleg403-client.virtnet: 1 5954 0.17 MB/sec execute 44 sec latency 8281.137 ms oleg403-client.virtnet: 1 5954 0.16 MB/sec execute 45 sec latency 9282.133 ms oleg403-client.virtnet: 1 5954 0.16 MB/sec execute 46 sec latency 10283.848 ms oleg403-client.virtnet: 1 5954 0.16 MB/sec execute 47 sec latency 11284.046 ms oleg403-client.virtnet: 1 5954 0.15 MB/sec execute 48 sec latency 12284.211 ms oleg403-client.virtnet: 1 5954 0.15 MB/sec execute 49 sec latency 13284.517 ms oleg403-client.virtnet: 1 5954 0.15 MB/sec execute 50 sec latency 14284.695 ms oleg403-client.virtnet: 1 5954 0.14 MB/sec execute 51 sec latency 15284.884 ms oleg403-client.virtnet: 1 5954 0.14 MB/sec execute 52 sec latency 16285.555 ms oleg403-client.virtnet: 1 5954 0.14 MB/sec execute 53 sec latency 17287.304 ms oleg403-client.virtnet: 1 5954 0.14 MB/sec execute 54 sec latency 18287.632 ms oleg403-client.virtnet: 1 5954 0.13 MB/sec execute 55 sec latency 19288.753 ms oleg403-client.virtnet: 1 5967 0.13 MB/sec execute 56 sec latency 20210.022 ms oleg403-client.virtnet: 1 6168 0.15 MB/sec execute 57 sec latency 80.973 ms oleg403-client.virtnet: 1 6746 0.23 MB/sec execute 58 sec latency 64.789 ms oleg403-client.virtnet: 1 7175 0.30 MB/sec execute 59 sec latency 43.397 ms oleg403-client.virtnet: 1 7438 0.32 MB/sec execute 60 sec latency 20.281 ms oleg403-client.virtnet: 1 7614 0.32 MB/sec execute 61 sec latency 31.032 ms oleg403-client.virtnet: 1 7825 0.test_70b fail mds1 5 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server 00:36:41 (1746074201) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:36:56 (1746074216) targets are mounted 00:36:56 (1746074216) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3747840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3718144 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7465984 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server 00:37:09 (1746074229) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 32 MB/sec execute 62 sec latency 86.386 ms oleg403-client.virtnet: 1 7924 0.32 MB/sec execute 63 sec latency 81.556 ms oleg403-client.virtnet: 1 8050 0.32 MB/sec execute 64 sec latency 287.977 ms oleg403-client.virtnet: 1 8050 0.31 MB/sec execute 65 sec latency 1288.678 ms oleg403-client.virtnet: 1 8050 0.31 MB/sec execute 66 sec latency 2291.284 ms oleg403-client.virtnet: 1 8050 0.30 MB/sec execute 67 sec latency 3292.615 ms oleg403-client.virtnet: 1 8050 0.30 MB/sec execute 68 sec latency 4292.862 ms oleg403-client.virtnet: 1 8050 0.29 MB/sec execute 69 sec latency 5293.085 ms oleg403-client.virtnet: 1 8050 0.29 MB/sec execute 70 sec latency 6293.267 ms oleg403-client.virtnet: 1 8050 0.29 MB/sec execute 71 sec latency 7293.434 ms oleg403-client.virtnet: 1 8050 0.28 MB/sec execute 72 sec latency 8293.612 ms oleg403-client.virtnet: 1 8050 0.28 MB/sec execute 73 sec latency 9295.115 ms oleg403-client.virtnet: 1 8050 0.27 MB/sec execute 74 sec latency 10295.293 ms oleg403-client.virtnet: 1 8050 0.27 MB/sec execute 75 sec latency 11295.457 ms oleg403-client.virtnet: 1 8050 0.27 MB/sec execute 76 sec latency 12295.823 ms oleg403-client.virtnet: 1 8050 0.26 MB/sec execute 77 sec latency 13296.022 ms oleg403-client.virtnet: 1 8050 0.26 MB/sec execute 78 sec latency 14296.540 ms oleg403-client.virtnet: 1 8050 0.26 MB/sec execute 79 sec latency 15297.632 ms oleg403-client.virtnet: 1 8050 0.25 MB/sec execute 80 sec latency 16297.789 ms oleg403-client.virtnet: 1 8050 0.25 MB/sec execute 81 sec latency 17298.174 ms oleg403-client.virtnet: 1 8097 0.25 MB/sec execute 82 sec latency 17562.549 ms oleg403-client.virtnet: 1 8238 0.26 MB/sec execute 83 sec latency 89.016 ms oleg403-client.virtnet: 1 8360 0.26 MB/sec execute 84 sec latency 35.606 ms oleg403-client.virtnet: 1 8613 0.29 MB/sec execute 85 sec latency 69.314 ms oleg403-client.virtnet: 1 8754 0.29 MB/sec execute 86 sec latency 90.204 ms oleg403-client.virtnet: 1 8885 0.28 MB/sec execute 87 sec latency 43.233 ms oleg403-client.virtnet: 1 8993 0.28 MB/sec execute 88 sec latency 48.995 ms oleg403-client.virtnet: 1 9189 0.28 MB/sec execute 89 sec latency 91.889 ms oleg403-client.virtnet: 1 9356 0.28 MB/sec execute 90 sec latency 57.441 ms oleg403-client.virtnet: 1 9510 0.29 MB/sec execute 91 sec latency 109.415 ms oleg403-client.virtnet: 1 9521 0.29 MB/sec execute 92 sec latency 821.555 ms oleg403-client.virtnet: 1 9521 0.29 MB/sec execute 93 sec latency 1821.873 ms oleg403-client.virtnet: 1 9521 0.28 MB/sec execute 94 sec latency 2822.107 ms oleg403-client.virtnet: 1 9521 0.28 MB/sec execute 95 sec latency 3822.375 ms oleg403-client.virtnet: 1 9521 0.28 MB/sec execute 96 sec latency 4823.914 ms oleg403-client.virtnet: 1 9521 0.27 MB/sec execute 97 sec latency 5824.347 ms oleg403-client.virtnet: 1 9521 0.27 MB/sec execute 98 sec latency 6824.521 ms oleg403-client.virtnet: 1 9521 0.27 MB/sec execute 99 sec latency 7827.025 ms oleg403-client.virtnet: 1 9521 0.27 MB/sec execute 100 sec latency 8828.110 ms oleg403-client.virtnet: 1 9521 0.26 MB/sec execute 101 sec latency 9828.360 ms oleg403-client.virtnet: 1 9521 0.26 MB/sec execute 102 sec latency 10828.831 ms oleg403-client.virtnet: 1 9521 0.26 MB/sec execute 103 sec latency 11830.120 ms oleg403-client.virtnet: 1 9521 0.26 MB/sec execute 104 sec latency 12831.106 ms oleg403-client.virtnet: 1 9521 0.25 MB/sec execute 105 sec latency 13831.246 ms oleg403-client.virtnet: 1 9521 0.25 MB/sec exepdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:37:23 (1746074243) targets are mounted 00:37:23 (1746074243) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3729408 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3709952 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7439360 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server 00:37:37 (1746074257) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:37:53 (1746074273) targets are mounted 00:37:53 (1746074273) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3737600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3717120 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7454720 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server cute 106 sec latency 14831.515 ms oleg403-client.virtnet: 1 9521 0.25 MB/sec execute 107 sec latency 15834.937 ms oleg403-client.virtnet: 1 9521 0.25 MB/sec execute 108 sec latency 16835.130 ms oleg403-client.virtnet: 1 9521 0.24 MB/sec execute 109 sec latency 17836.246 ms oleg403-client.virtnet: 1 9521 0.24 MB/sec execute 110 sec latency 18837.041 ms oleg403-client.virtnet: 1 9530 0.24 MB/sec execute 111 sec latency 19799.348 ms oleg403-client.virtnet: 1 9735 0.25 MB/sec execute 112 sec latency 86.705 ms oleg403-client.virtnet: 1 10099 0.28 MB/sec execute 113 sec latency 101.288 ms oleg403-client.virtnet: 1 10472 0.30 MB/sec execute 114 sec latency 69.698 ms oleg403-client.virtnet: 1 10792 0.32 MB/sec execute 115 sec latency 24.905 ms oleg403-client.virtnet: 1 10954 0.33 MB/sec execute 116 sec latency 39.106 ms oleg403-client.virtnet: 1 11104 0.33 MB/sec execute 117 sec latency 42.069 ms oleg403-client.virtnet: 1 11297 0.33 MB/sec execute 118 sec latency 33.704 ms oleg403-client.virtnet: 1 11383 0.33 MB/sec execute 119 sec latency 121.390 ms oleg403-client.virtnet: 1 11425 0.32 MB/sec execute 120 sec latency 506.042 ms oleg403-client.virtnet: 1 11425 0.32 MB/sec execute 121 sec latency 1506.513 ms oleg403-client.virtnet: 1 11425 0.32 MB/sec execute 122 sec latency 2506.822 ms oleg403-client.virtnet: 1 11425 0.32 MB/sec execute 123 sec latency 3507.524 ms oleg403-client.virtnet: 1 11425 0.31 MB/sec execute 124 sec latency 4508.851 ms oleg403-client.virtnet: 1 11425 0.31 MB/sec execute 125 sec latency 5510.175 ms oleg403-client.virtnet: 1 11425 0.31 MB/sec execute 126 sec latency 6510.345 ms oleg403-client.virtnet: 1 11425 0.31 MB/sec execute 127 sec latency 7512.074 ms oleg403-client.virtnet: 1 11425 0.30 MB/sec execute 128 sec latency 8512.258 ms oleg403-client.virtnet: 1 11425 0.30 MB/sec execute 129 sec latency 9512.423 ms oleg403-client.virtnet: 1 11425 0.30 MB/sec execute 130 sec latency 10512.613 ms oleg403-client.virtnet: 1 11425 0.30 MB/sec execute 131 sec latency 11512.704 ms oleg403-client.virtnet: 1 11425 0.29 MB/sec execute 132 sec latency 12513.874 ms oleg403-client.virtnet: 1 11425 0.29 MB/sec execute 133 sec latency 13514.012 ms oleg403-client.virtnet: 1 11425 0.29 MB/sec execute 134 sec latency 14515.064 ms oleg403-client.virtnet: 1 11425 0.29 MB/sec execute 135 sec latency 15515.625 ms oleg403-client.virtnet: 1 11425 0.29 MB/sec execute 136 sec latency 16517.551 ms oleg403-client.virtnet: 1 11458 0.28 MB/sec execute 137 sec latency 17011.443 ms oleg403-client.virtnet: 1 11561 0.28 MB/sec execute 138 sec latency 100.026 ms oleg403-client.virtnet: 1 11639 0.28 MB/sec execute 139 sec latency 102.302 ms oleg403-client.virtnet: 1 11790 0.29 MB/sec execute 140 sec latency 59.744 ms oleg403-client.virtnet: 1 11926 0.29 MB/sec execute 141 sec latency 66.487 ms oleg403-client.virtnet: 1 12137 0.31 MB/sec execute 142 sec latency 70.068 ms oleg403-client.virtnet: 1 12280 0.31 MB/sec execute 143 sec latency 96.191 ms oleg403-client.virtnet: 1 12368 0.30 MB/sec execute 144 sec latency 130.756 ms oleg403-client.virtnet: 1 12461 0.30 MB/sec execute 145 sec latency 56.149 ms oleg403-client.virtnet: 1 12620 0.30 MB/sec execute 146 sec latency 80.124 ms oleg403-client.virtnet: 1 12770 0.30 MB/sec execute 147 sec latency 50.068 ms oleg403-client.virtnet: 1 12918 0.30 MB/sec execute 148 sec latency 74.623 ms oleg403-client.virtnet: 1 12918 0.30 MB/sec execute 149 sec latency 1074.826 ms oleg403-client.virtnet: 1 12918 0.30 MB/sec execute 150 sec latency 2000:38:14 (1746074294) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:38:30 (1746074310) targets are mounted 00:38:30 (1746074310) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3717120 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7462912 1% /mnt/lustre 75.143 ms oleg403-client.virtnet: 1 12918 0.29 MB/sec execute 151 sec latency 3082.958 ms oleg403-client.virtnet: 1 12918 0.29 MB/sec execute 152 sec latency 4083.110 ms oleg403-client.virtnet: 1 12918 0.29 MB/sec execute 153 sec latency 5083.252 ms oleg403-client.virtnet: 1 12918 0.29 MB/sec execute 154 sec latency 6084.877 ms oleg403-client.virtnet: 1 12918 0.29 MB/sec execute 155 sec latency 7085.061 ms oleg403-client.virtnet: 1 12918 0.29 MB/sec execute 156 sec latency 8086.178 ms oleg403-client.virtnet: 1 12918 0.28 MB/sec execute 157 sec latency 9086.339 ms oleg403-client.virtnet: 1 12918 0.28 MB/sec execute 158 sec latency 10087.452 ms oleg403-client.virtnet: 1 12918 0.28 MB/sec execute 159 sec latency 11092.492 ms oleg403-client.virtnet: 1 12918 0.28 MB/sec execute 160 sec latency 12094.098 ms oleg403-client.virtnet: 1 12918 0.28 MB/sec execute 161 sec latency 13094.263 ms oleg403-client.virtnet: 1 12918 0.27 MB/sec execute 162 sec latency 14094.417 ms oleg403-client.virtnet: 1 12918 0.27 MB/sec execute 163 sec latency 15094.572 ms oleg403-client.virtnet: 1 12918 0.27 MB/sec execute 164 sec latency 16094.766 ms oleg403-client.virtnet: 1 12918 0.27 MB/sec execute 165 sec latency 17094.964 ms oleg403-client.virtnet: 1 12918 0.27 MB/sec execute 166 sec latency 18095.166 ms oleg403-client.virtnet: 1 12918 0.27 MB/sec execute 167 sec latency 19095.364 ms oleg403-client.virtnet: 1 12918 0.26 MB/sec execute 168 sec latency 20095.522 ms oleg403-client.virtnet: 1 12918 0.26 MB/sec execute 169 sec latency 21095.672 ms oleg403-client.virtnet: 1 12918 0.26 MB/sec execute 170 sec latency 22096.027 ms oleg403-client.virtnet: 1 12918 0.26 MB/sec execute 171 sec latency 23096.178 ms oleg403-client.virtnet: 1 12918 0.26 MB/sec execute 172 sec latency 24096.338 ms oleg403-client.virtnet: 1 12918 0.26 MB/sec execute 173 sec latency 25100.835 ms oleg403-client.virtnet: 1 12918 0.26 MB/sec execute 174 sec latency 26102.152 ms oleg403-client.virtnet: 1 12918 0.25 MB/sec execute 175 sec latency 27103.343 ms oleg403-client.virtnet: 1 12918 0.25 MB/sec execute 176 sec latency 28103.548 ms oleg403-client.virtnet: 1 12918 0.25 MB/sec execute 177 sec latency 29104.277 ms oleg403-client.virtnet: 1 12918 0.25 MB/sec execute 178 sec latency 30106.715 ms oleg403-client.virtnet: 1 12918 0.25 MB/sec execute 179 sec latency 31109.397 ms oleg403-client.virtnet: 1 12918 0.25 MB/sec execute 180 sec latency 32111.381 ms oleg403-client.virtnet: 1 12918 0.25 MB/sec execute 181 sec latency 33113.509 ms oleg403-client.virtnet: 1 12918 0.24 MB/sec execute 182 sec latency 34113.721 ms oleg403-client.virtnet: 1 12918 0.24 MB/sec execute 183 sec latency 35114.191 ms oleg403-client.virtnet: 1 12918 0.24 MB/sec execute 184 sec latency 36116.789 ms oleg403-client.virtnet: 1 12918 0.24 MB/sec execute 185 sec latency 37117.516 ms oleg403-client.virtnet: 1 12927 0.24 MB/sec execute 186 sec latency 38103.371 ms oleg403-client.virtnet: 1 13049 0.24 MB/sec execute 187 sec latency 151.648 ms oleg403-client.virtnet: 1 13143 0.25 MB/sec execute 188 sec latency 117.298 ms oleg403-client.virtnet: 1 13419 0.26 MB/sec execute 189 sec latency 91.223 ms oleg403-client.virtnet: 1 13640 0.27 MB/sec execute 190 sec latency 117.540 ms oleg403-client.virtnet: 1 14014 0.28 MB/sec execute 191 sec latency 76.885 ms oleg403-client.virtnet: 1 14269 0.29 MB/sec execute 192 sec latency 49.695 ms oleg403-client.virtnet: 1 14415 0.30 MB/sec execute 193 sec latency 48.285 ms oleg403-client.virtnet: 1 14540 0.30 MB/sec execute 194 stest_70b fail mds1 9 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server 00:38:54 (1746074334) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:39:08 (1746074348) targets are mounted 00:39:08 (1746074348) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3716096 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7460864 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg403-server Stopping /mnt/lustre-mds1 (opts:) on oleg403-server 00:39:24 (1746074364) shut down facet: mds1 facet_host: oleg403-server facet_failover_host: oleg403-server Failover mds1 to oleg403-server mount facets: mds1 ec latency 55.753 ms oleg403-client.virtnet: 1 14631 0.30 MB/sec execute 195 sec latency 69.093 ms oleg403-client.virtnet: 1 14731 0.29 MB/sec execute 196 sec latency 44.025 ms oleg403-client.virtnet: 1 14751 0.29 MB/sec execute 197 sec latency 902.816 ms oleg403-client.virtnet: 1 14751 0.29 MB/sec execute 198 sec latency 1904.077 ms oleg403-client.virtnet: 1 14751 0.29 MB/sec execute 199 sec latency 2904.252 ms oleg403-client.virtnet: 1 14751 0.29 MB/sec execute 200 sec latency 3904.409 ms oleg403-client.virtnet: 1 14751 0.29 MB/sec execute 201 sec latency 4905.009 ms oleg403-client.virtnet: 1 14751 0.29 MB/sec execute 202 sec latency 5905.871 ms oleg403-client.virtnet: 1 14751 0.28 MB/sec execute 203 sec latency 6906.749 ms oleg403-client.virtnet: 1 14751 0.28 MB/sec execute 204 sec latency 7906.921 ms oleg403-client.virtnet: 1 14751 0.28 MB/sec execute 205 sec latency 8907.085 ms oleg403-client.virtnet: 1 14751 0.28 MB/sec execute 206 sec latency 9907.790 ms oleg403-client.virtnet: 1 14751 0.28 MB/sec execute 207 sec latency 10907.951 ms oleg403-client.virtnet: 1 14751 0.28 MB/sec execute 208 sec latency 11911.953 ms oleg403-client.virtnet: 1 14751 0.28 MB/sec execute 209 sec latency 12913.021 ms oleg403-client.virtnet: 1 14751 0.27 MB/sec execute 210 sec latency 13913.174 ms oleg403-client.virtnet: 1 14751 0.27 MB/sec execute 211 sec latency 14916.317 ms oleg403-client.virtnet: 1 14751 0.27 MB/sec execute 212 sec latency 15916.472 ms oleg403-client.virtnet: 1 14751 0.27 MB/sec execute 213 sec latency 16916.604 ms oleg403-client.virtnet: 1 14751 0.27 MB/sec execute 214 sec latency 17916.771 ms oleg403-client.virtnet: 1 14751 0.27 MB/sec execute 215 sec latency 18916.931 ms oleg403-client.virtnet: 1 14751 0.27 MB/sec execute 216 sec latency 19917.482 ms oleg403-client.virtnet: 1 14824 0.27 MB/sec execute 217 sec latency 20318.902 ms oleg403-client.virtnet: 1 14928 0.27 MB/sec execute 218 sec latency 143.268 ms oleg403-client.virtnet: 1 14996 0.27 MB/sec execute 219 sec latency 134.394 ms oleg403-client.virtnet: 1 15110 0.27 MB/sec execute 220 sec latency 136.638 ms oleg403-client.virtnet: 1 15241 0.27 MB/sec execute 221 sec latency 114.767 ms oleg403-client.virtnet: 1 15392 0.27 MB/sec execute 222 sec latency 40.140 ms oleg403-client.virtnet: 1 15612 0.28 MB/sec execute 223 sec latency 39.433 ms oleg403-client.virtnet: 1 15817 0.28 MB/sec execute 224 sec latency 48.155 ms oleg403-client.virtnet: 1 15963 0.28 MB/sec execute 225 sec latency 82.919 ms oleg403-client.virtnet: 1 16089 0.28 MB/sec execute 226 sec latency 88.325 ms oleg403-client.virtnet: 1 16089 0.28 MB/sec execute 227 sec latency 1066.379 ms oleg403-client.virtnet: 1 16089 0.28 MB/sec execute 228 sec latency 2067.822 ms oleg403-client.virtnet: 1 16089 0.28 MB/sec execute 229 sec latency 3067.986 ms oleg403-client.virtnet: 1 16089 0.27 MB/sec execute 230 sec latency 4068.327 ms oleg403-client.virtnet: 1 16089 0.27 MB/sec execute 231 sec latency 5068.578 ms oleg403-client.virtnet: 1 16089 0.27 MB/sec execute 232 sec latency 6068.747 ms oleg403-client.virtnet: 1 16089 0.27 MB/sec execute 233 sec latency 7070.526 ms oleg403-client.virtnet: 1 16089 0.27 MB/sec execute 234 sec latency 8070.684 ms oleg403-client.virtnet: 1 16089 0.27 MB/sec execute 235 sec latency 9071.467 ms oleg403-client.virtnet: 1 16089 0.27 MB/sec execute 236 sec latency 10071.626 ms oleg403-client.virtnet: 1 16089 0.27 MB/sec execute 237 sec latency 11071.798 ms oleg403-client.virtnet: 1 16089 0.26 MB/sec execute 238 sec latency Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 00:39:38 (1746074378) targets are mounted 00:39:38 (1746074378) facet_failover done oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 12071.966 ms oleg403-client.virtnet: 1 16089 0.26 MB/sec execute 239 sec latency 13072.124 ms oleg403-client.virtnet: 1 16089 0.26 MB/sec execute 240 sec latency 14072.251 ms oleg403-client.virtnet: 1 16089 0.26 MB/sec execute 241 sec latency 15072.559 ms oleg403-client.virtnet: 1 16089 0.26 MB/sec execute 242 sec latency 16072.744 ms oleg403-client.virtnet: 1 16089 0.26 MB/sec execute 243 sec latency 17073.533 ms oleg403-client.virtnet: 1 16089 0.26 MB/sec execute 244 sec latency 18074.521 ms oleg403-client.virtnet: 1 16089 0.26 MB/sec execute 245 sec latency 19074.768 ms oleg403-client.virtnet: 1 16089 0.26 MB/sec execute 246 sec latency 20074.979 ms oleg403-client.virtnet: 1 16352 0.26 MB/sec execute 247 sec latency 20124.371 ms oleg403-client.virtnet: 1 16546 0.26 MB/sec execute 248 sec latency 93.167 ms oleg403-client.virtnet: 1 16754 0.26 MB/sec execute 249 sec latency 82.586 ms oleg403-client.virtnet: 1 17174 0.28 MB/sec execute 250 sec latency 78.714 ms oleg403-client.virtnet: 1 17571 0.29 MB/sec execute 251 sec latency 84.772 ms oleg403-client.virtnet: 1 17951 0.30 MB/sec execute 252 sec latency 24.053 ms oleg403-client.virtnet: 1 18153 0.30 MB/sec execute 253 sec latency 36.701 ms oleg403-client.virtnet: 1 18364 0.30 MB/sec execute 254 sec latency 20.046 ms oleg403-client.virtnet: 1 18481 0.30 MB/sec execute 255 sec latency 141.632 ms oleg403-client.virtnet: 1 18605 0.30 MB/sec execute 256 sec latency 100.583 ms oleg403-client.virtnet: 1 18736 0.30 MB/sec execute 257 sec latency 87.986 ms oleg403-client.virtnet: 1 18996 0.31 MB/sec execute 258 sec latency 47.173 ms oleg403-client.virtnet: 1 19274 0.32 MB/sec execute 259 sec latency 96.213 ms oleg403-client.virtnet: 1 19465 0.32 MB/sec execute 260 sec latency 54.885 ms oleg403-client.virtnet: 1 19635 0.31 MB/sec execute 261 sec latency 90.404 ms oleg403-client.virtnet: 1 19896 0.31 MB/sec execute 262 sec latency 92.855 ms oleg403-client.virtnet: 1 20130 0.32 MB/sec execute 263 sec latency 62.779 ms oleg403-client.virtnet: 1 20262 0.32 MB/sec execute 264 sec latency 81.999 ms oleg403-client.virtnet: 1 20662 0.33 MB/sec execute 265 sec latency 93.559 ms oleg403-client.virtnet: 1 20988 0.34 MB/sec execute 266 sec latency 80.279 ms oleg403-client.virtnet: 1 21312 0.35 MB/sec execute 267 sec latency 67.141 ms oleg403-client.virtnet: 1 21478 0.35 MB/sec execute 268 sec latency 30.716 ms oleg403-client.virtnet: 1 21674 0.36 MB/sec execute 269 sec latency 26.522 ms oleg403-client.virtnet: 1 21838 0.36 MB/sec execute 270 sec latency 29.982 ms oleg403-client.virtnet: 1 22010 0.35 MB/sec execute 271 sec latency 114.055 ms oleg403-client.virtnet: 1 22122 0.35 MB/sec execute 272 sec latency 95.114 ms oleg403-client.virtnet: 1 22253 0.35 MB/sec execute 273 sec latency 85.555 ms oleg403-client.virtnet: 1 22477 0.36 MB/sec execute 274 sec latency 68.994 ms oleg403-client.virtnet: 1 22799 0.37 MB/sec execute 275 sec latency 27.923 ms oleg403-client.virtnet: 1 22992 0.37 MB/sec execute 276 sec latency 81.358 ms oleg403-client.virtnet: 1 23185 0.37 MB/sec execute 277 sec latency 23.355 ms oleg403-client.virtnet: 1 23414 0.37 MB/sec execute 278 sec latency 58.481 ms oleg403-client.virtnet: 1 23694 0.37 MB/sec execute 279 sec latency 58.853 ms oleg403-client.virtnet: 1 23900 0.37 MB/sec execute 280 sec latency 74.711 ms oleg403-client.virtnet: 1 24252 0.39 MB/sec execute 281 sec latency 98.766 ms oleg403-client.virtnet: 1 24560 0.39 MB/sec execute 282 sec latency 85.741 ms oleg403-client.virtnet: 1 24902 0.40 MB/sec execute 283 sec latency 45.352 ms oleg403-client.virtnet: 1 25126 0.40 MB/sec execute 284 sec latency 28.638 ms oleg403-client.virtnet: 1 25294 0.40 MB/sec execute 285 sec latency 24.743 ms oleg403-client.virtnet: 1 25551 0.40 MB/sec execute 286 sec latency 35.079 ms oleg403-client.virtnet: 1 25675 0.40 MB/sec execute 287 sec latency 87.698 ms oleg403-client.virtnet: 1 25804 0.40 MB/sec execute 288 sec latency 88.199 ms oleg403-client.virtnet: 1 26016 0.40 MB/sec execute 289 sec latency 81.423 ms oleg403-client.virtnet: 1 26299 0.41 MB/sec execute 290 sec latency 22.910 ms oleg403-client.virtnet: 1 26478 0.41 MB/sec execute 291 sec latency 75.241 ms oleg403-client.virtnet: 1 26655 0.41 MB/sec execute 292 sec latency 82.065 ms oleg403-client.virtnet: 1 26846 0.41 MB/sec execute 293 sec latency 78.540 ms oleg403-client.virtnet: 1 27036 0.41 MB/sec execute 294 sec latency 39.672 ms oleg403-client.virtnet: 1 27254 0.41 MB/sec execute 295 sec latency 106.403 ms oleg403-client.virtnet: 1 27436 0.42 MB/sec execute 296 sec latency 81.176 ms oleg403-client.virtnet: 1 27793 0.43 MB/sec execute 297 sec latency 84.303 ms oleg403-client.virtnet: 1 28175 0.43 MB/sec execute 298 sec latency 73.418 ms oleg403-client.virtnet: 1 28565 0.44 MB/sec execute 299 sec latency 29.212 ms oleg403-client.virtnet: 1 cleanup 300 sec oleg403-client.virtnet: 0 cleanup 301 sec oleg403-client.virtnet: oleg403-client.virtnet: Operation Count AvgLat MaxLat oleg403-client.virtnet: ---------------------------------------- oleg403-client.virtnet: NTCreateX 4379 27.123 38103.360 oleg403-client.virtnet: Close 3190 15.148 20612.990 oleg403-client.virtnet: Rename 182 20.745 66.470 oleg403-client.virtnet: Unlink 889 5.062 47.681 oleg403-client.virtnet: Qpathinfo 3931 13.977 20318.883 oleg403-client.virtnet: Qfileinfo 680 0.235 5.753 oleg403-client.virtnet: Qfsinfo 719 0.358 22.469 oleg403-client.virtnet: Sfileinfo 350 71.811 19799.336 oleg403-client.virtnet: Find 1510 0.888 37.441 oleg403-client.virtnet: WriteX 2128 1.989 28.090 oleg403-client.virtnet: ReadX 6751 0.051 14.552 oleg403-client.virtnet: LockX 14 2.902 5.198 oleg403-client.virtnet: UnlockX 14 2.591 4.113 oleg403-client.virtnet: Flush 301 126.271 20124.349 oleg403-client.virtnet: oleg403-client.virtnet: Throughput 0.444221 MB/sec 1 clients 1 procs max_latency=38103.371 ms oleg403-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg403-client.virtnet at Thu May 1 00:40:37 EDT 2025 with return code 0 oleg403-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg403-client.virtnet oleg403-client.virtnet: /mnt/lustre/d70b.replay-single/oleg403-client.virtnet /mnt/lustre/d70b.replay-single/oleg403-client.virtnet oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg403-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg403-client.virtnet: removed directory: 'clients/client0' oleg403-client.virtnet: removed directory: 'clients' oleg403-client.virtnet: removed 'client.txt' oleg403-client.virtnet: /mnt/lustre/d70b.replay-single/oleg403-client.virtnet oleg403-client.virtnet: dbench successfully finished