== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 18:25:10 (1746138310) Starting client oleg115-client.virtnet: -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre Started clients oleg115-client.virtnet: 192.168.201.115@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg115-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found dbench: no process found Started rundbench load pid=6885 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3731456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7468032 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg115-server Stopping /mnt/lustre-mds1 (opts:) on oleg115-server 18:25:24 (1746138324) shut down facet: mds1 facet_host: oleg115-server facet_failover_host: oleg115-server Failover mds1 to oleg115-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 18:25:42 (1746138342) targets are mounted 18:25:42 (1746138342) facet_failover done oleg115-client.virtnet: looking for dbench program oleg115-client.virtnet: /usr/bin/dbench oleg115-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg115-client.virtnet oleg115-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg115-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg115-client.virtnet' oleg115-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg115-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg115-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg115-client.virtnet at Thu May 1 18:25:13 EDT 2025 oleg115-client.virtnet: waiting for dbench pid 6916 oleg115-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg115-client.virtnet: oleg115-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg115-client.virtnet: failed to create barrier semaphore oleg115-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg115-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg115-client.virtnet: releasing clients oleg115-client.virtnet: 1 139 6.02 MB/sec warmup 1 sec latency 125.795 ms oleg115-client.virtnet: 1 240 4.57 MB/sec warmup 2 sec latency 70.248 ms oleg115-client.virtnet: 1 349 4.17 MB/sec warmup 3 sec latency 97.508 ms oleg115-client.virtnet: 1 484 4.24 MB/sec warmup 4 sec latency 75.196 ms oleg115-client.virtnet: 1 649 4.21 MB/sec warmup 5 sec latency 93.481 ms oleg115-client.virtnet: 1 699 3.53 MB/sec warmup 6 sec latency 132.677 ms oleg115-client.virtnet: 1 774 3.05 MB/sec warmup 7 sec latency 125.598 ms oleg115-client.virtnet: 1 808 2.68 MB/sec warmup 8 sec latency 160.023 ms oleg115-client.virtnet: 1 893 2.41 MB/sec warmup 9 sec latency 162.855 ms oleg115-client.virtnet: 1 902 2.17 MB/sec warmup 10 sec latency 947.856 ms oleg115-client.virtnet: 1 902 1.97 MB/sec warmup 11 sec latency 1948.091 ms oleg115-client.virtnet: 1 902 1.81 MB/sec warmup 12 sec latency 2950.700 ms oleg115-client.virtnet: 1 902 1.67 MB/sec warmup 13 sec latency 3950.893 ms oleg115-client.virtnet: 1 902 1.55 MB/sec warmup 14 sec latency 4951.561 ms oleg115-client.virtnet: 1 902 1.44 MB/sec warmup 15 sec latency 5951.831 ms oleg115-client.virtnet: 1 902 1.35 MB/sec warmup 16 sec latency 6953.152 ms oleg115-client.virtnet: 1 902 1.27 MB/sec warmup 17 sec latency 7953.406 ms oleg115-client.virtnet: 1 902 1.20 MB/sec warmup 18 sec latency 8954.997 ms oleg115-client.virtnet: 1 902 1.14 MB/sec warmup 19 sec latency 9955.342 ms oleg115-client.virtnet: 1 902 1.08 MB/sec warmup 20 sec latency 10957.536 ms oleg115-client.virtnet: 1 902 1.03 MB/sec warmup 21 sec latency 11957.735 ms oleg115-client.virtnet: 1 902 0.98 MB/sec warmup 22 sec latency 12968.862 ms oleg115-client.virtnet: 1 902 0.94 MB/sec warmup 23 sec latency 13979.566 ms oleg115-client.virtnet: 1 902 0.90 MB/sec warmup 24 sec latency 14985.375 ms oleg115-client.virtnet: 1 902 0.87 MB/sec warmup 25 sec latency 15995.149 ms oleg115-client.virtnet: 1 902 0.83 MB/sec warmup 26 sec latency 16995.347 ms oleg115-client.virtnet: 1 902 0.80 MB/sec warmup 27 sec latency 17999.290 ms oleg115-client.virtnet: 1 902 0.77 MB/sec warmup 28 sec latency 18999.548 ms oleg115-client.virtnet: 1 902 0.75 MB/sec warmup 29 sec latency 19999.762 ms oleg115-client.virtnet: 1 949 0.73 MB/sec warmup 30 sec latency 20381.891 ms oleg115-client.virtnet: 1 995 0.71 MB/sec warmup 31 sec latency 189.013 ms oleg115-client.virtnet: 1 1126 0.72 MB/sec warmup 32 sec latency 5oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3748864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 38912 3715072 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 52224 7463936 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg115-server Stopping /mnt/lustre-mds1 (opts:) on oleg115-server 18:26:00 (1746138360) shut down facet: mds1 facet_host: oleg115-server facet_failover_host: oleg115-server Failover mds1 to oleg115-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 18:26:17 (1746138377) targets are mounted 18:26:17 (1746138377) facet_failover done oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3731456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3723264 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7454720 1% /mnt/lustre test_70b fail mds1 3 times 9.745 ms oleg115-client.virtnet: 1 1214 0.70 MB/sec warmup 33 sec latency 62.301 ms oleg115-client.virtnet: 1 1384 0.76 MB/sec warmup 34 sec latency 81.534 ms oleg115-client.virtnet: 1 1462 0.74 MB/sec warmup 35 sec latency 71.568 ms oleg115-client.virtnet: 1 1552 0.72 MB/sec warmup 36 sec latency 127.764 ms oleg115-client.virtnet: 1 1634 0.71 MB/sec warmup 37 sec latency 152.389 ms oleg115-client.virtnet: 1 1755 0.69 MB/sec warmup 38 sec latency 65.368 ms oleg115-client.virtnet: 1 1851 0.67 MB/sec warmup 39 sec latency 54.840 ms oleg115-client.virtnet: 1 1926 0.66 MB/sec warmup 40 sec latency 236.597 ms oleg115-client.virtnet: 1 2012 0.65 MB/sec warmup 41 sec latency 69.082 ms oleg115-client.virtnet: 1 2162 0.63 MB/sec warmup 42 sec latency 49.586 ms oleg115-client.virtnet: 1 2291 0.64 MB/sec warmup 43 sec latency 68.184 ms oleg115-client.virtnet: 1 2376 0.63 MB/sec warmup 44 sec latency 177.814 ms oleg115-client.virtnet: 1 2418 0.62 MB/sec warmup 45 sec latency 238.109 ms oleg115-client.virtnet: 1 2428 0.61 MB/sec warmup 46 sec latency 916.939 ms oleg115-client.virtnet: 1 2428 0.60 MB/sec warmup 47 sec latency 1917.363 ms oleg115-client.virtnet: 1 2428 0.58 MB/sec warmup 48 sec latency 2917.571 ms oleg115-client.virtnet: 1 2428 0.57 MB/sec warmup 49 sec latency 3917.789 ms oleg115-client.virtnet: 1 2428 0.56 MB/sec warmup 50 sec latency 4920.183 ms oleg115-client.virtnet: 1 2428 0.55 MB/sec warmup 51 sec latency 5920.427 ms oleg115-client.virtnet: 1 2428 0.54 MB/sec warmup 52 sec latency 6921.759 ms oleg115-client.virtnet: 1 2428 0.53 MB/sec warmup 53 sec latency 7926.972 ms oleg115-client.virtnet: 1 2428 0.52 MB/sec warmup 54 sec latency 8928.047 ms oleg115-client.virtnet: 1 2428 0.51 MB/sec warmup 55 sec latency 9929.048 ms oleg115-client.virtnet: 1 2428 0.50 MB/sec warmup 56 sec latency 10929.281 ms oleg115-client.virtnet: 1 2428 0.49 MB/sec warmup 57 sec latency 11929.509 ms oleg115-client.virtnet: 1 2428 0.48 MB/sec warmup 58 sec latency 12931.662 ms oleg115-client.virtnet: 1 2428 0.48 MB/sec warmup 59 sec latency 13932.332 ms oleg115-client.virtnet: 1 2428 0.00 MB/sec execute 1 sec latency 15933.197 ms oleg115-client.virtnet: 1 2428 0.00 MB/sec execute 2 sec latency 16933.920 ms oleg115-client.virtnet: 1 2428 0.00 MB/sec execute 3 sec latency 17934.308 ms oleg115-client.virtnet: 1 2428 0.00 MB/sec execute 4 sec latency 18935.924 ms oleg115-client.virtnet: 1 2465 0.10 MB/sec execute 5 sec latency 19518.096 ms oleg115-client.virtnet: 1 2575 0.14 MB/sec execute 6 sec latency 176.604 ms oleg115-client.virtnet: 1 2768 0.55 MB/sec execute 7 sec latency 52.555 ms oleg115-client.virtnet: 1 2903 0.59 MB/sec execute 8 sec latency 301.569 ms oleg115-client.virtnet: 1 2982 0.60 MB/sec execute 9 sec latency 183.671 ms oleg115-client.virtnet: 1 3169 0.60 MB/sec execute 10 sec latency 246.515 ms oleg115-client.virtnet: 1 3277 0.66 MB/sec execute 11 sec latency 256.533 ms oleg115-client.virtnet: 1 3432 0.70 MB/sec execute 12 sec latency 78.298 ms oleg115-client.virtnet: 1 3627 0.81 MB/sec execute 13 sec latency 43.587 ms oleg115-client.virtnet: 1 3767 0.83 MB/sec execute 14 sec latency 58.071 ms oleg115-client.virtnet: 1 3829 0.78 MB/sec execute 15 sec latency 81.078 ms oleg115-client.virtnet: 1 3899 0.75 MB/sec execute 16 sec latency 120.582 ms oleg115-client.virtnet: 1 3984 0.70 MB/sec execute 17 sec latency 66.744 ms oleg115-client.virtnet: 1 4064 0.66 MB/seFailing mds1 on oleg115-server Stopping /mnt/lustre-mds1 (opts:) on oleg115-server 18:26:35 (1746138395) shut down facet: mds1 facet_host: oleg115-server facet_failover_host: oleg115-server Failover mds1 to oleg115-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 18:26:53 (1746138413) targets are mounted 18:26:53 (1746138413) facet_failover done oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3716096 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7461888 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg115-server Stopping /mnt/lustre-mds1 (opts:) on oleg115-server 18:27:10 (1746138430) shut down facet: mds1 facet_host: oleg115-server facet_failover_host: oleg115-server c execute 18 sec latency 50.260 ms oleg115-client.virtnet: 1 4172 0.63 MB/sec execute 19 sec latency 68.566 ms oleg115-client.virtnet: 1 4246 0.61 MB/sec execute 20 sec latency 122.337 ms oleg115-client.virtnet: 1 4246 0.58 MB/sec execute 21 sec latency 1014.912 ms oleg115-client.virtnet: 1 4246 0.55 MB/sec execute 22 sec latency 2015.145 ms oleg115-client.virtnet: 1 4246 0.53 MB/sec execute 23 sec latency 3015.515 ms oleg115-client.virtnet: 1 4246 0.51 MB/sec execute 24 sec latency 4016.031 ms oleg115-client.virtnet: 1 4246 0.49 MB/sec execute 25 sec latency 5016.487 ms oleg115-client.virtnet: 1 4246 0.47 MB/sec execute 26 sec latency 6019.451 ms oleg115-client.virtnet: 1 4246 0.45 MB/sec execute 27 sec latency 7019.658 ms oleg115-client.virtnet: 1 4246 0.43 MB/sec execute 28 sec latency 8020.285 ms oleg115-client.virtnet: 1 4246 0.42 MB/sec execute 29 sec latency 9021.004 ms oleg115-client.virtnet: 1 4246 0.41 MB/sec execute 30 sec latency 10021.215 ms oleg115-client.virtnet: 1 4246 0.39 MB/sec execute 31 sec latency 11021.788 ms oleg115-client.virtnet: 1 4246 0.38 MB/sec execute 32 sec latency 12023.834 ms oleg115-client.virtnet: 1 4246 0.37 MB/sec execute 33 sec latency 13024.601 ms oleg115-client.virtnet: 1 4246 0.36 MB/sec execute 34 sec latency 14026.370 ms oleg115-client.virtnet: 1 4246 0.35 MB/sec execute 35 sec latency 15026.561 ms oleg115-client.virtnet: 1 4246 0.34 MB/sec execute 36 sec latency 16028.800 ms oleg115-client.virtnet: 1 4246 0.33 MB/sec execute 37 sec latency 17029.666 ms oleg115-client.virtnet: 1 4283 0.32 MB/sec execute 38 sec latency 17358.796 ms oleg115-client.virtnet: 1 4333 0.32 MB/sec execute 39 sec latency 228.922 ms oleg115-client.virtnet: 1 4394 0.31 MB/sec execute 40 sec latency 136.855 ms oleg115-client.virtnet: 1 4483 0.31 MB/sec execute 41 sec latency 166.846 ms oleg115-client.virtnet: 1 4546 0.31 MB/sec execute 42 sec latency 170.800 ms oleg115-client.virtnet: 1 4625 0.32 MB/sec execute 43 sec latency 196.389 ms oleg115-client.virtnet: 1 4700 0.32 MB/sec execute 44 sec latency 114.638 ms oleg115-client.virtnet: 1 4780 0.31 MB/sec execute 45 sec latency 77.992 ms oleg115-client.virtnet: 1 4862 0.31 MB/sec execute 46 sec latency 61.202 ms oleg115-client.virtnet: 1 5017 0.36 MB/sec execute 47 sec latency 114.369 ms oleg115-client.virtnet: 1 5103 0.36 MB/sec execute 48 sec latency 128.793 ms oleg115-client.virtnet: 1 5199 0.35 MB/sec execute 49 sec latency 194.970 ms oleg115-client.virtnet: 1 5323 0.35 MB/sec execute 50 sec latency 65.764 ms oleg115-client.virtnet: 1 5436 0.34 MB/sec execute 51 sec latency 62.849 ms oleg115-client.virtnet: 1 5579 0.34 MB/sec execute 52 sec latency 125.290 ms oleg115-client.virtnet: 1 5712 0.33 MB/sec execute 53 sec latency 58.711 ms oleg115-client.virtnet: 1 5819 0.33 MB/sec execute 54 sec latency 103.445 ms oleg115-client.virtnet: 1 5906 0.34 MB/sec execute 55 sec latency 313.543 ms oleg115-client.virtnet: 1 5906 0.34 MB/sec execute 56 sec latency 1319.154 ms oleg115-client.virtnet: 1 5906 0.33 MB/sec execute 57 sec latency 2321.379 ms oleg115-client.virtnet: 1 5906 0.33 MB/sec execute 58 sec latency 3322.280 ms oleg115-client.virtnet: 1 5906 0.32 MB/sec execute 59 sec latency 4322.497 ms oleg115-client.virtnet: 1 5906 0.31 MB/sec execute 60 sec latency 5322.969 ms oleg115-client.virtnet: 1 5906 0.31 MB/sec execute 61 sec latency 6326.363 ms oleg115-client.virtnet: 1 5906 0.30 MB/sec execute 62 sec laFailover mds1 to oleg115-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 18:27:28 (1746138448) targets are mounted 18:27:28 (1746138448) facet_failover done oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3725312 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3713024 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7438336 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg115-server Stopping /mnt/lustre-mds1 (opts:) on oleg115-server 18:27:45 (1746138465) shut down facet: mds1 facet_host: oleg115-server facet_failover_host: oleg115-server Failover mds1 to oleg115-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 tency 7327.167 ms oleg115-client.virtnet: 1 5906 0.30 MB/sec execute 63 sec latency 8327.903 ms oleg115-client.virtnet: 1 5906 0.30 MB/sec execute 64 sec latency 9328.122 ms oleg115-client.virtnet: 1 5906 0.29 MB/sec execute 65 sec latency 10329.187 ms oleg115-client.virtnet: 1 5906 0.29 MB/sec execute 66 sec latency 11329.491 ms oleg115-client.virtnet: 1 5906 0.28 MB/sec execute 67 sec latency 12329.866 ms oleg115-client.virtnet: 1 5906 0.28 MB/sec execute 68 sec latency 13330.261 ms oleg115-client.virtnet: 1 5906 0.27 MB/sec execute 69 sec latency 14331.365 ms oleg115-client.virtnet: 1 5906 0.27 MB/sec execute 70 sec latency 15332.774 ms oleg115-client.virtnet: 1 5906 0.27 MB/sec execute 71 sec latency 16333.337 ms oleg115-client.virtnet: 1 5918 0.26 MB/sec execute 72 sec latency 16993.888 ms oleg115-client.virtnet: 1 5962 0.26 MB/sec execute 73 sec latency 236.363 ms oleg115-client.virtnet: 1 6013 0.26 MB/sec execute 74 sec latency 284.870 ms oleg115-client.virtnet: 1 6129 0.27 MB/sec execute 75 sec latency 153.553 ms oleg115-client.virtnet: 1 6320 0.30 MB/sec execute 76 sec latency 57.573 ms oleg115-client.virtnet: 1 6473 0.32 MB/sec execute 77 sec latency 169.563 ms oleg115-client.virtnet: 1 6537 0.31 MB/sec execute 78 sec latency 279.286 ms oleg115-client.virtnet: 1 6724 0.32 MB/sec execute 79 sec latency 303.546 ms oleg115-client.virtnet: 1 6825 0.33 MB/sec execute 80 sec latency 246.450 ms oleg115-client.virtnet: 1 6919 0.33 MB/sec execute 81 sec latency 197.184 ms oleg115-client.virtnet: 1 7120 0.36 MB/sec execute 82 sec latency 53.264 ms oleg115-client.virtnet: 1 7239 0.36 MB/sec execute 83 sec latency 44.870 ms oleg115-client.virtnet: 1 7355 0.37 MB/sec execute 84 sec latency 58.275 ms oleg115-client.virtnet: 1 7447 0.37 MB/sec execute 85 sec latency 45.983 ms oleg115-client.virtnet: 1 7514 0.36 MB/sec execute 86 sec latency 56.859 ms oleg115-client.virtnet: 1 7578 0.36 MB/sec execute 87 sec latency 71.931 ms oleg115-client.virtnet: 1 7697 0.35 MB/sec execute 88 sec latency 44.673 ms oleg115-client.virtnet: 1 7780 0.35 MB/sec execute 89 sec latency 239.553 ms oleg115-client.virtnet: 1 7834 0.35 MB/sec execute 90 sec latency 162.280 ms oleg115-client.virtnet: 1 7844 0.35 MB/sec execute 91 sec latency 601.285 ms oleg115-client.virtnet: 1 7844 0.34 MB/sec execute 92 sec latency 1602.337 ms oleg115-client.virtnet: 1 7844 0.34 MB/sec execute 93 sec latency 2602.802 ms oleg115-client.virtnet: 1 7844 0.33 MB/sec execute 94 sec latency 3606.646 ms oleg115-client.virtnet: 1 7844 0.33 MB/sec execute 95 sec latency 4607.453 ms oleg115-client.virtnet: 1 7844 0.33 MB/sec execute 96 sec latency 5607.743 ms oleg115-client.virtnet: 1 7844 0.32 MB/sec execute 97 sec latency 6607.925 ms oleg115-client.virtnet: 1 7844 0.32 MB/sec execute 98 sec latency 7608.736 ms oleg115-client.virtnet: 1 7844 0.32 MB/sec execute 99 sec latency 8609.747 ms oleg115-client.virtnet: 1 7844 0.31 MB/sec execute 100 sec latency 9610.046 ms oleg115-client.virtnet: 1 7844 0.31 MB/sec execute 101 sec latency 10612.842 ms oleg115-client.virtnet: 1 7844 0.31 MB/sec execute 102 sec latency 11612.972 ms oleg115-client.virtnet: 1 7844 0.30 MB/sec execute 103 sec latency 12613.332 ms oleg115-client.virtnet: 1 7844 0.30 MB/sec execute 104 sec latency 13613.469 ms oleg115-client.virtnet: 1 7844 0.30 MB/sec execute 105 sec latency 14613.948 ms oleg115-client.virtnet: 1 7844 0.30 MB/sec execute 106 sec latency 15614.672 msoleg115-server: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 18:28:03 (1746138483) targets are mounted 18:28:03 (1746138483) facet_failover done oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3715072 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7460864 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg115-server Stopping /mnt/lustre-mds1 (opts:) on oleg115-server 18:28:22 (1746138502) shut down facet: mds1 facet_host: oleg115-server facet_failover_host: oleg115-server Failover mds1 to oleg115-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 18:28:40 (1746138520) targets are mounted 18:28:40 (1746138520) facet_failover done oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg115-client.virtnet: 1 7848 0.29 MB/sec execute 107 sec latency 16460.100 ms oleg115-client.virtnet: 1 7907 0.29 MB/sec execute 108 sec latency 205.822 ms oleg115-client.virtnet: 1 7962 0.29 MB/sec execute 109 sec latency 143.059 ms oleg115-client.virtnet: 1 8012 0.29 MB/sec execute 110 sec latency 242.083 ms oleg115-client.virtnet: 1 8088 0.29 MB/sec execute 111 sec latency 163.857 ms oleg115-client.virtnet: 1 8151 0.29 MB/sec execute 112 sec latency 176.053 ms oleg115-client.virtnet: 1 8247 0.29 MB/sec execute 113 sec latency 53.451 ms oleg115-client.virtnet: 1 8329 0.29 MB/sec execute 114 sec latency 78.735 ms oleg115-client.virtnet: 1 8376 0.29 MB/sec execute 115 sec latency 85.456 ms oleg115-client.virtnet: 1 8530 0.31 MB/sec execute 116 sec latency 118.238 ms oleg115-client.virtnet: 1 8625 0.31 MB/sec execute 117 sec latency 137.481 ms oleg115-client.virtnet: 1 8740 0.31 MB/sec execute 118 sec latency 67.179 ms oleg115-client.virtnet: 1 8852 0.31 MB/sec execute 119 sec latency 125.809 ms oleg115-client.virtnet: 1 8982 0.30 MB/sec execute 120 sec latency 37.321 ms oleg115-client.virtnet: 1 9078 0.30 MB/sec execute 121 sec latency 259.458 ms oleg115-client.virtnet: 1 9204 0.30 MB/sec execute 122 sec latency 80.881 ms oleg115-client.virtnet: 1 9330 0.30 MB/sec execute 123 sec latency 75.438 ms oleg115-client.virtnet: 1 9454 0.31 MB/sec execute 124 sec latency 234.886 ms oleg115-client.virtnet: 1 9492 0.30 MB/sec execute 125 sec latency 346.931 ms oleg115-client.virtnet: 1 9545 0.30 MB/sec execute 126 sec latency 163.525 ms oleg115-client.virtnet: 1 9577 0.30 MB/sec execute 127 sec latency 756.130 ms oleg115-client.virtnet: 1 9577 0.30 MB/sec execute 128 sec latency 1756.341 ms oleg115-client.virtnet: 1 9577 0.30 MB/sec execute 129 sec latency 2756.535 ms oleg115-client.virtnet: 1 9577 0.30 MB/sec execute 130 sec latency 3756.717 ms oleg115-client.virtnet: 1 9577 0.30 MB/sec execute 131 sec latency 4757.000 ms oleg115-client.virtnet: 1 9577 0.29 MB/sec execute 132 sec latency 5757.170 ms oleg115-client.virtnet: 1 9577 0.29 MB/sec execute 133 sec latency 6758.338 ms oleg115-client.virtnet: 1 9577 0.29 MB/sec execute 134 sec latency 7758.513 ms oleg115-client.virtnet: 1 9577 0.29 MB/sec execute 135 sec latency 8759.519 ms oleg115-client.virtnet: 1 9577 0.28 MB/sec execute 136 sec latency 9759.706 ms oleg115-client.virtnet: 1 9577 0.28 MB/sec execute 137 sec latency 10760.153 ms oleg115-client.virtnet: 1 9577 0.28 MB/sec execute 138 sec latency 11762.651 ms oleg115-client.virtnet: 1 9577 0.28 MB/sec execute 139 sec latency 12762.804 ms oleg115-client.virtnet: 1 9577 0.28 MB/sec execute 140 sec latency 13763.034 ms oleg115-client.virtnet: 1 9577 0.27 MB/sec execute 141 sec latency 14774.124 ms oleg115-client.virtnet: 1 9577 0.27 MB/sec execute 142 sec latency 15774.734 ms oleg115-client.virtnet: 1 9577 0.27 MB/sec execute 143 sec latency 16778.601 ms oleg115-client.virtnet: 1 9577 0.27 MB/sec execute 144 sec latency 17779.847 ms oleg115-client.virtnet: 1 9577 0.27 MB/sec execute 145 sec latency 18780.089 ms oleg115-client.virtnet: 1 9577 0.27 MB/sec execute 146 sec latency 19780.518 ms oleg115-client.virtnet: 1 9581 0.26 MB/sec execute 147 sec latency 20716.152 ms oleg115-client.virtnet: 1 9735 0.27 MB/sec execute 148 sec latency 102.531 ms oleg115-client.virtnet: 1 9997 0.29 MB/sec execute 149 sec latency 170.482 ms oleg115-client.virtnet: 1 10078 0.29 MB/sec execute 150 sec latency 264.913 ms oleg115-client.virtnemdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3716096 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7460864 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg115-server Stopping /mnt/lustre-mds1 (opts:) on oleg115-server 18:28:58 (1746138538) shut down facet: mds1 facet_host: oleg115-server facet_failover_host: oleg115-server Failover mds1 to oleg115-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 18:29:16 (1746138556) targets are mounted 18:29:16 (1746138556) facet_failover done oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3714048 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7460864 1% /mnt/lustre test_70b fail mds1 8 times t: 1 10123 0.29 MB/sec execute 151 sec latency 229.978 ms oleg115-client.virtnet: 1 10359 0.30 MB/sec execute 152 sec latency 190.040 ms oleg115-client.virtnet: 1 10465 0.30 MB/sec execute 153 sec latency 103.330 ms oleg115-client.virtnet: 1 10675 0.32 MB/sec execute 154 sec latency 48.033 ms oleg115-client.virtnet: 1 10767 0.31 MB/sec execute 155 sec latency 65.399 ms oleg115-client.virtnet: 1 10875 0.32 MB/sec execute 156 sec latency 52.988 ms oleg115-client.virtnet: 1 10942 0.32 MB/sec execute 157 sec latency 74.915 ms oleg115-client.virtnet: 1 11010 0.32 MB/sec execute 158 sec latency 87.356 ms oleg115-client.virtnet: 1 11061 0.32 MB/sec execute 159 sec latency 124.546 ms oleg115-client.virtnet: 1 11139 0.31 MB/sec execute 160 sec latency 72.596 ms oleg115-client.virtnet: 1 11268 0.31 MB/sec execute 161 sec latency 50.328 ms oleg115-client.virtnet: 1 11330 0.31 MB/sec execute 162 sec latency 248.825 ms oleg115-client.virtnet: 1 11381 0.31 MB/sec execute 163 sec latency 131.844 ms oleg115-client.virtnet: 1 11391 0.31 MB/sec execute 164 sec latency 556.103 ms oleg115-client.virtnet: 1 11391 0.31 MB/sec execute 165 sec latency 1556.266 ms oleg115-client.virtnet: 1 11391 0.30 MB/sec execute 166 sec latency 2556.419 ms oleg115-client.virtnet: 1 11391 0.30 MB/sec execute 167 sec latency 3557.348 ms oleg115-client.virtnet: 1 11391 0.30 MB/sec execute 168 sec latency 4557.546 ms oleg115-client.virtnet: 1 11391 0.30 MB/sec execute 169 sec latency 5558.108 ms oleg115-client.virtnet: 1 11391 0.30 MB/sec execute 170 sec latency 6558.312 ms oleg115-client.virtnet: 1 11391 0.30 MB/sec execute 171 sec latency 7559.938 ms oleg115-client.virtnet: 1 11391 0.29 MB/sec execute 172 sec latency 8560.408 ms oleg115-client.virtnet: 1 11391 0.29 MB/sec execute 173 sec latency 9560.601 ms oleg115-client.virtnet: 1 11391 0.29 MB/sec execute 174 sec latency 10561.034 ms oleg115-client.virtnet: 1 11391 0.29 MB/sec execute 175 sec latency 11561.465 ms oleg115-client.virtnet: 1 11391 0.29 MB/sec execute 176 sec latency 12562.310 ms oleg115-client.virtnet: 1 11391 0.29 MB/sec execute 177 sec latency 13564.422 ms oleg115-client.virtnet: 1 11391 0.28 MB/sec execute 178 sec latency 14565.214 ms oleg115-client.virtnet: 1 11391 0.28 MB/sec execute 179 sec latency 15566.506 ms oleg115-client.virtnet: 1 11391 0.28 MB/sec execute 180 sec latency 16566.947 ms oleg115-client.virtnet: 1 11391 0.28 MB/sec execute 181 sec latency 17567.136 ms oleg115-client.virtnet: 1 11393 0.28 MB/sec execute 182 sec latency 18550.767 ms oleg115-client.virtnet: 1 11458 0.28 MB/sec execute 183 sec latency 113.112 ms oleg115-client.virtnet: 1 11547 0.28 MB/sec execute 184 sec latency 117.262 ms oleg115-client.virtnet: 1 11610 0.28 MB/sec execute 185 sec latency 140.862 ms oleg115-client.virtnet: 1 11698 0.28 MB/sec execute 186 sec latency 96.566 ms oleg115-client.virtnet: 1 11792 0.28 MB/sec execute 187 sec latency 59.501 ms oleg115-client.virtnet: 1 11870 0.28 MB/sec execute 188 sec latency 99.039 ms oleg115-client.virtnet: 1 12074 0.29 MB/sec execute 189 sec latency 45.435 ms oleg115-client.virtnet: 1 12159 0.29 MB/sec execute 190 sec latency 150.413 ms oleg115-client.virtnet: 1 12274 0.29 MB/sec execute 191 sec latency 58.683 ms oleg115-client.virtnet: 1 12341 0.29 MB/sec execute 192 sec latency 240.405 ms oleg115-client.virtnet: 1 12440 0.29 MB/sec execute 193 sec latency 97.080 ms oleg115-client.virtnet: 1 12529 0.29 MB/sec execute 194 sec latency 75.981 ms oleg115-client.virtnet: 1 12660 0.29 MB/sec Failing mds1 on oleg115-server Stopping /mnt/lustre-mds1 (opts:) on oleg115-server 18:29:34 (1746138574) shut down facet: mds1 facet_host: oleg115-server facet_failover_host: oleg115-server Failover mds1 to oleg115-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 18:29:52 (1746138592) targets are mounted 18:29:52 (1746138592) facet_failover done oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3739648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3696640 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7436288 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg115-server Stopping /mnt/lustre-mds1 (opts:) on oleg115-server 18:30:08 (1746138608) shut down facet: mds1 facet_host: oleg115-server facet_failover_host: oleg115-server execute 195 sec latency 108.227 ms oleg115-client.virtnet: 1 12765 0.29 MB/sec execute 196 sec latency 58.976 ms oleg115-client.virtnet: 1 12839 0.28 MB/sec execute 197 sec latency 56.721 ms oleg115-client.virtnet: 1 12949 0.29 MB/sec execute 198 sec latency 70.072 ms oleg115-client.virtnet: 1 12959 0.29 MB/sec execute 199 sec latency 891.559 ms oleg115-client.virtnet: 1 12959 0.28 MB/sec execute 200 sec latency 1892.791 ms oleg115-client.virtnet: 1 12959 0.28 MB/sec execute 201 sec latency 2893.210 ms oleg115-client.virtnet: 1 12959 0.28 MB/sec execute 202 sec latency 3893.577 ms oleg115-client.virtnet: 1 12959 0.28 MB/sec execute 203 sec latency 4894.017 ms oleg115-client.virtnet: 1 12959 0.28 MB/sec execute 204 sec latency 5897.863 ms oleg115-client.virtnet: 1 12959 0.28 MB/sec execute 205 sec latency 6899.734 ms oleg115-client.virtnet: 1 12959 0.28 MB/sec execute 206 sec latency 7899.936 ms oleg115-client.virtnet: 1 12959 0.28 MB/sec execute 207 sec latency 8903.135 ms oleg115-client.virtnet: 1 12959 0.27 MB/sec execute 208 sec latency 9903.565 ms oleg115-client.virtnet: 1 12959 0.27 MB/sec execute 209 sec latency 10904.720 ms oleg115-client.virtnet: 1 12959 0.27 MB/sec execute 210 sec latency 11906.536 ms oleg115-client.virtnet: 1 12959 0.27 MB/sec execute 211 sec latency 12906.692 ms oleg115-client.virtnet: 1 12959 0.27 MB/sec execute 212 sec latency 13907.661 ms oleg115-client.virtnet: 1 12959 0.27 MB/sec execute 213 sec latency 14908.398 ms oleg115-client.virtnet: 1 12959 0.27 MB/sec execute 214 sec latency 15910.071 ms oleg115-client.virtnet: 1 12959 0.26 MB/sec execute 215 sec latency 16911.476 ms oleg115-client.virtnet: 1 12959 0.26 MB/sec execute 216 sec latency 17911.726 ms oleg115-client.virtnet: 1 12999 0.26 MB/sec execute 217 sec latency 18484.135 ms oleg115-client.virtnet: 1 13060 0.26 MB/sec execute 218 sec latency 184.110 ms oleg115-client.virtnet: 1 13133 0.26 MB/sec execute 219 sec latency 214.357 ms oleg115-client.virtnet: 1 13285 0.27 MB/sec execute 220 sec latency 70.612 ms oleg115-client.virtnet: 1 13566 0.28 MB/sec execute 221 sec latency 127.020 ms oleg115-client.virtnet: 1 13650 0.28 MB/sec execute 222 sec latency 172.064 ms oleg115-client.virtnet: 1 13882 0.28 MB/sec execute 223 sec latency 105.898 ms oleg115-client.virtnet: 1 14035 0.29 MB/sec execute 224 sec latency 112.041 ms oleg115-client.virtnet: 1 14234 0.30 MB/sec execute 225 sec latency 53.249 ms oleg115-client.virtnet: 1 14399 0.31 MB/sec execute 226 sec latency 51.729 ms oleg115-client.virtnet: 1 14487 0.30 MB/sec execute 227 sec latency 50.278 ms oleg115-client.virtnet: 1 14583 0.30 MB/sec execute 228 sec latency 51.603 ms oleg115-client.virtnet: 1 14667 0.30 MB/sec execute 229 sec latency 51.812 ms oleg115-client.virtnet: 1 14789 0.30 MB/sec execute 230 sec latency 41.192 ms oleg115-client.virtnet: 1 14881 0.30 MB/sec execute 231 sec latency 148.534 ms oleg115-client.virtnet: 1 14930 0.30 MB/sec execute 232 sec latency 174.101 ms oleg115-client.virtnet: 1 14972 0.30 MB/sec execute 233 sec latency 394.417 ms oleg115-client.virtnet: 1 14972 0.30 MB/sec execute 234 sec latency 1394.589 ms oleg115-client.virtnet: 1 14972 0.30 MB/sec execute 235 sec latency 2394.950 ms oleg115-client.virtnet: 1 14972 0.30 MB/sec execute 236 sec latency 3395.611 ms oleg115-client.virtnet: 1 14972 0.29 MB/sec execute 237 sec latency 4396.140 ms oleg115-client.virtnet: 1 14972 0.29 MB/sec execute 238 sec latency 5398.224 ms oleg115-client.virtnet: 1 14972 0.29 MB/sec execute 239 sec latency 6Failover mds1 to oleg115-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 18:30:26 (1746138626) targets are mounted 18:30:26 (1746138626) facet_failover done oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 398.614 ms oleg115-client.virtnet: 1 14972 0.29 MB/sec execute 240 sec latency 7399.883 ms oleg115-client.virtnet: 1 14972 0.29 MB/sec execute 241 sec latency 8411.182 ms oleg115-client.virtnet: 1 14972 0.29 MB/sec execute 242 sec latency 9411.455 ms oleg115-client.virtnet: 1 14972 0.29 MB/sec execute 243 sec latency 10412.701 ms oleg115-client.virtnet: 1 14972 0.29 MB/sec execute 244 sec latency 11415.175 ms oleg115-client.virtnet: 1 14972 0.28 MB/sec execute 245 sec latency 12422.426 ms oleg115-client.virtnet: 1 14972 0.28 MB/sec execute 246 sec latency 13425.463 ms oleg115-client.virtnet: 1 14972 0.28 MB/sec execute 247 sec latency 14425.747 ms oleg115-client.virtnet: 1 14972 0.28 MB/sec execute 248 sec latency 15425.988 ms oleg115-client.virtnet: 1 14972 0.28 MB/sec execute 249 sec latency 16427.918 ms oleg115-client.virtnet: 1 14972 0.28 MB/sec execute 250 sec latency 17430.945 ms oleg115-client.virtnet: 1 14972 0.28 MB/sec execute 251 sec latency 18431.124 ms oleg115-client.virtnet: 1 14972 0.28 MB/sec execute 252 sec latency 19431.469 ms oleg115-client.virtnet: 1 15005 0.28 MB/sec execute 253 sec latency 19476.266 ms oleg115-client.virtnet: 1 15091 0.28 MB/sec execute 254 sec latency 88.867 ms oleg115-client.virtnet: 1 15148 0.27 MB/sec execute 255 sec latency 149.270 ms oleg115-client.virtnet: 1 15234 0.28 MB/sec execute 256 sec latency 164.185 ms oleg115-client.virtnet: 1 15335 0.28 MB/sec execute 257 sec latency 69.573 ms oleg115-client.virtnet: 1 15413 0.28 MB/sec execute 258 sec latency 91.022 ms oleg115-client.virtnet: 1 15580 0.29 MB/sec execute 259 sec latency 53.865 ms oleg115-client.virtnet: 1 15678 0.29 MB/sec execute 260 sec latency 68.647 ms oleg115-client.virtnet: 1 15833 0.29 MB/sec execute 261 sec latency 109.687 ms oleg115-client.virtnet: 1 15928 0.28 MB/sec execute 262 sec latency 142.651 ms oleg115-client.virtnet: 1 16067 0.28 MB/sec execute 263 sec latency 57.930 ms oleg115-client.virtnet: 1 16175 0.28 MB/sec execute 264 sec latency 175.290 ms oleg115-client.virtnet: 1 16337 0.28 MB/sec execute 265 sec latency 43.353 ms oleg115-client.virtnet: 1 16529 0.29 MB/sec execute 266 sec latency 43.673 ms oleg115-client.virtnet: 1 16597 0.29 MB/sec execute 267 sec latency 166.183 ms oleg115-client.virtnet: 1 16681 0.29 MB/sec execute 268 sec latency 165.190 ms oleg115-client.virtnet: 1 16949 0.30 MB/sec execute 269 sec latency 61.204 ms oleg115-client.virtnet: 1 17117 0.30 MB/sec execute 270 sec latency 176.825 ms oleg115-client.virtnet: 1 17367 0.30 MB/sec execute 271 sec latency 124.651 ms oleg115-client.virtnet: 1 17543 0.31 MB/sec execute 272 sec latency 113.607 ms oleg115-client.virtnet: 1 17745 0.32 MB/sec execute 273 sec latency 49.379 ms oleg115-client.virtnet: 1 17859 0.32 MB/sec execute 274 sec latency 50.569 ms oleg115-client.virtnet: 1 17968 0.32 MB/sec execute 275 sec latency 45.643 ms oleg115-client.virtnet: 1 18055 0.32 MB/sec execute 276 sec latency 52.846 ms oleg115-client.virtnet: 1 18150 0.32 MB/sec execute 277 sec latency 48.398 ms oleg115-client.virtnet: 1 18214 0.32 MB/sec execute 278 sec latency 60.487 ms oleg115-client.virtnet: 1 18337 0.32 MB/sec execute 279 sec latency 37.166 ms oleg115-client.virtnet: 1 18430 0.32 MB/sec execute 280 sec latency 243.465 ms oleg115-client.virtnet: 1 18488 0.32 MB/sec execute 281 sec latency 180.300 ms oleg115-client.virtnet: 1 18553 0.32 MB/sec execute 282 sec latency 160.535 ms oleg115-client.virtnet: 1 18639 0.31 MB/sec execute 283 sec latency 149.382 ms oleg115-client.virtnet: 1 18720 0.31 MB/sec execute 284 sec latency 144.080 ms oleg115-client.virtnet: 1 18819 0.32 MB/sec execute 285 sec latency 172.586 ms oleg115-client.virtnet: 1 18969 0.32 MB/sec execute 286 sec latency 34.422 ms oleg115-client.virtnet: 1 19156 0.33 MB/sec execute 287 sec latency 43.886 ms oleg115-client.virtnet: 1 19243 0.32 MB/sec execute 288 sec latency 176.794 ms oleg115-client.virtnet: 1 19316 0.32 MB/sec execute 289 sec latency 66.642 ms oleg115-client.virtnet: 1 19382 0.32 MB/sec execute 290 sec latency 94.094 ms oleg115-client.virtnet: 1 19473 0.32 MB/sec execute 291 sec latency 115.926 ms oleg115-client.virtnet: 1 19555 0.32 MB/sec execute 292 sec latency 65.524 ms oleg115-client.virtnet: 1 19639 0.32 MB/sec execute 293 sec latency 160.460 ms oleg115-client.virtnet: 1 19761 0.32 MB/sec execute 294 sec latency 75.035 ms oleg115-client.virtnet: 1 19908 0.32 MB/sec execute 295 sec latency 45.984 ms oleg115-client.virtnet: 1 20029 0.32 MB/sec execute 296 sec latency 60.298 ms oleg115-client.virtnet: 1 20161 0.32 MB/sec execute 297 sec latency 128.884 ms oleg115-client.virtnet: 1 20213 0.32 MB/sec execute 298 sec latency 211.940 ms oleg115-client.virtnet: 1 20345 0.32 MB/sec execute 299 sec latency 159.178 ms oleg115-client.virtnet: 1 cleanup 300 sec oleg115-client.virtnet: 1 cleanup 301 sec oleg115-client.virtnet: 0 cleanup 301 sec oleg115-client.virtnet: oleg115-client.virtnet: Operation Count AvgLat MaxLat oleg115-client.virtnet: ---------------------------------------- oleg115-client.virtnet: NTCreateX 3113 31.769 19476.235 oleg115-client.virtnet: Close 2288 13.339 16993.852 oleg115-client.virtnet: Rename 134 189.505 19518.071 oleg115-client.virtnet: Unlink 627 9.712 38.289 oleg115-client.virtnet: Qpathinfo 2832 7.038 125.193 oleg115-client.virtnet: Qfileinfo 496 0.366 13.400 oleg115-client.virtnet: Qfsinfo 523 0.903 30.223 oleg115-client.virtnet: Sfileinfo 256 385.257 20716.137 oleg115-client.virtnet: Find 1094 1.652 86.360 oleg115-client.virtnet: WriteX 1541 3.748 55.435 oleg115-client.virtnet: ReadX 4983 0.110 36.920 oleg115-client.virtnet: LockX 10 6.742 14.101 oleg115-client.virtnet: UnlockX 10 8.354 17.953 oleg115-client.virtnet: Flush 217 121.833 346.911 oleg115-client.virtnet: oleg115-client.virtnet: Throughput 0.322203 MB/sec 1 clients 1 procs max_latency=20716.152 ms oleg115-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg115-client.virtnet at Thu May 1 18:31:15 EDT 2025 with return code 0 oleg115-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg115-client.virtnet oleg115-client.virtnet: /mnt/lustre/d70b.replay-single/oleg115-client.virtnet /mnt/lustre/d70b.replay-single/oleg115-client.virtnet oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg115-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg115-client.virtnet: removed directory: 'clients/client0' oleg115-client.virtnet: removed directory: 'clients' oleg115-client.virtnet: removed 'client.txt' oleg115-client.virtnet: /mnt/lustre/d70b.replay-single/oleg115-client.virtnet oleg115-client.virtnet: dbench successfully finished