== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 06:00:26 (1743501626) Starting client oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre Started clients oleg631-client.virtnet: 192.168.206.131@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg631-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found dbench: no process found Started rundbench load pid=125726 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3840 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 32768 3689472 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3729408 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 35840 7418880 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 06:00:46 (1743501646) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 oleg631-client.virtnet: looking for dbench program oleg631-client.virtnet: /usr/bin/dbench oleg631-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg631-client.virtnet oleg631-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg631-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg631-client.virtnet' oleg631-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg631-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg631-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg631-client.virtnet at Tue Apr 1 06:00:29 EDT 2025 oleg631-client.virtnet: waiting for dbench pid 125756 oleg631-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg631-client.virtnet: oleg631-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg631-client.virtnet: failed to create barrier semaphore oleg631-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg631-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg631-client.virtnet: releasing clients oleg631-client.virtnet: 1 176 7.06 MB/sec warmup 1 sec latency 91.357 ms oleg631-client.virtnet: 1 306 5.61 MB/sec warmup 2 sec latency 48.090 ms oleg631-client.virtnet: 1 437 5.13 MB/sec warmup 3 sec latency 92.607 ms oleg631-client.virtnet: 1 639 5.26 MB/sec warmup 4 sec latency 70.307 ms oleg631-client.virtnet: 1 721 4.26 MB/sec warmup 5 sec latency 183.179 ms oleg631-client.virtnet: 1 786 3.57 MB/sec warmup 6 sec latency 159.982 ms oleg631-client.virtnet: 1 864 3.08 MB/sec warmup 7 sec latency 236.849 ms oleg631-client.virtnet: 1 957 2.73 MB/sec warmup 8 sec latency 152.762 ms oleg631-client.virtnet: 1 1128 2.55 MB/sec warmup 9 sec latency 163.178 ms oleg631-client.virtnet: 1 1410 2.60 MB/sec warmup 10 sec latency 24.335 ms oleg631-client.virtnet: 1 1560 2.37 MB/sec warmup 11 sec latency 233.276 ms oleg631-client.virtnet: 1 1689 2.18 MB/sec warmup 12 sec latency 127.724 ms oleg631-client.virtnet: 1 1868 2.03 MB/sec warmup 13 sec latency 36.845 ms oleg631-client.virtnet: 1 1984 1.89 MB/sec warmup 14 sec latency 172.754 ms oleg631-client.virtnet: 1 2145 1.78 MB/sec warmup 15 sec latency 336.995 ms oleg631-client.virtnet: 1 2145 1.67 MB/sec warmup 16 sec latency 1337.264 ms oleg631-client.virtnet: 1 2145 1.57 MB/sec warmup 17 sec latency 2337.601 ms oleg631-client.virtnet: 1 2145 1.48 MB/sec warmup 18 sec latency 3337.971 ms oleg631-client.virtnet: 1 2145 1.40 MB/sec warmup 19 sec latency 4338.156 ms oleg631-client.virtnet: 1 2145 1.33 MB/sec warmup 20 sec latency 5338.340 ms oleg631-client.virtnet: 1 2145 1.27 MB/sec warmup 21 sec latency 6338.681 ms oleg631-client.virtnet: 1 2145 1.21 MB/sec warmup 22 sec latency 7338.865 ms oleg631-client.virtnet: 1 2145 1.16 MB/sec warmup 23 sec latency 8339.126 ms oleg631-client.virtnet: 1 2145 1.11 MB/sec warmup 24 sec latency 9339.326 ms oleg631-client.virtnet: 1 2145 1.07 MB/sec warmup 25 sec latency 10339.641 ms oleg631-client.virtnet: 1 2145 1.03 MB/sec warmup 26 sec latency 11339.827 ms oleg631-client.virtnet: 1 2145 0.99 MB/sec warmup 27 sec latency 12340.115 ms oleg631-client.virtnet: 1 2145 0.95 MB/sec warmup 28 sec latency 13340.357 ms oleg631-client.virtnet: 1 2145 0.92 MB/sec warmup 29 sec latency 14340.673 ms oleg631-client.virtnet: 1 2145 0.89 MB/sec warmup 30 sec latency 15341.014 ms oleg631-client.virtnet: 1 2145 0.86 MB/sec warmup 31 sec latency 16341.212 ms oleg631-client.virtnet: 1 2145 0.83 MB/sec warmup 32 sec latency 17341.532Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 06:01:09 (1743501669) targets are mounted 06:01:09 (1743501669) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3840 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3714048 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3748864 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7462912 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 06:01:45 (1743501705) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server ms oleg631-client.virtnet: 1 2145 0.81 MB/sec warmup 33 sec latency 18341.875 ms oleg631-client.virtnet: 1 2145 0.78 MB/sec warmup 34 sec latency 19342.181 ms oleg631-client.virtnet: 1 2145 0.76 MB/sec warmup 35 sec latency 20342.455 ms oleg631-client.virtnet: 1 2145 0.74 MB/sec warmup 36 sec latency 21342.717 ms oleg631-client.virtnet: 1 2145 0.72 MB/sec warmup 37 sec latency 22343.014 ms oleg631-client.virtnet: 1 2145 0.70 MB/sec warmup 38 sec latency 23343.261 ms oleg631-client.virtnet: 1 2145 0.68 MB/sec warmup 39 sec latency 24343.523 ms oleg631-client.virtnet: 1 2145 0.67 MB/sec warmup 40 sec latency 25343.748 ms oleg631-client.virtnet: 1 2145 0.65 MB/sec warmup 41 sec latency 26344.095 ms oleg631-client.virtnet: 1 2145 0.64 MB/sec warmup 42 sec latency 27344.348 ms oleg631-client.virtnet: 1 2145 0.62 MB/sec warmup 43 sec latency 28344.536 ms oleg631-client.virtnet: 1 2145 0.61 MB/sec warmup 44 sec latency 29344.794 ms oleg631-client.virtnet: 1 2145 0.59 MB/sec warmup 45 sec latency 30345.090 ms oleg631-client.virtnet: 1 2145 0.58 MB/sec warmup 46 sec latency 31345.321 ms oleg631-client.virtnet: 1 2145 0.57 MB/sec warmup 47 sec latency 32345.555 ms oleg631-client.virtnet: 1 2145 0.56 MB/sec warmup 48 sec latency 33345.801 ms oleg631-client.virtnet: 1 2145 0.54 MB/sec warmup 49 sec latency 34346.069 ms oleg631-client.virtnet: 1 2145 0.53 MB/sec warmup 50 sec latency 35346.340 ms oleg631-client.virtnet: 1 2145 0.52 MB/sec warmup 51 sec latency 36347.851 ms oleg631-client.virtnet: 1 2191 0.51 MB/sec warmup 52 sec latency 36781.364 ms oleg631-client.virtnet: 1 2376 0.53 MB/sec warmup 53 sec latency 212.820 ms oleg631-client.virtnet: 1 2477 0.53 MB/sec warmup 54 sec latency 271.767 ms oleg631-client.virtnet: 1 2759 0.58 MB/sec warmup 55 sec latency 204.813 ms oleg631-client.virtnet: 1 2987 0.60 MB/sec warmup 56 sec latency 128.902 ms oleg631-client.virtnet: 1 3212 0.60 MB/sec warmup 57 sec latency 246.556 ms oleg631-client.virtnet: 1 3491 0.66 MB/sec warmup 58 sec latency 92.799 ms oleg631-client.virtnet: 1 3696 0.66 MB/sec warmup 59 sec latency 24.449 ms oleg631-client.virtnet: 1 4010 0.30 MB/sec execute 1 sec latency 44.897 ms oleg631-client.virtnet: 1 4147 0.16 MB/sec execute 2 sec latency 48.819 ms oleg631-client.virtnet: 1 4250 0.18 MB/sec execute 3 sec latency 215.650 ms oleg631-client.virtnet: 1 4333 0.18 MB/sec execute 4 sec latency 168.112 ms oleg631-client.virtnet: 1 4379 0.17 MB/sec execute 5 sec latency 219.541 ms oleg631-client.virtnet: 1 4483 0.19 MB/sec execute 6 sec latency 193.986 ms oleg631-client.virtnet: 1 4552 0.18 MB/sec execute 7 sec latency 256.393 ms oleg631-client.virtnet: 1 4739 0.30 MB/sec execute 8 sec latency 69.753 ms oleg631-client.virtnet: 1 4986 0.60 MB/sec execute 9 sec latency 42.376 ms oleg631-client.virtnet: 1 5123 0.55 MB/sec execute 10 sec latency 114.225 ms oleg631-client.virtnet: 1 5208 0.50 MB/sec execute 11 sec latency 200.477 ms oleg631-client.virtnet: 1 5344 0.47 MB/sec execute 12 sec latency 55.335 ms oleg631-client.virtnet: 1 5453 0.44 MB/sec execute 13 sec latency 165.351 ms oleg631-client.virtnet: 1 5539 0.41 MB/sec execute 14 sec latency 731.088 ms oleg631-client.virtnet: 1 5539 0.39 MB/sec execute 15 sec latency 1731.454 ms oleg631-client.virtnet: 1 5539 0.36 MB/sec execute 16 sec latency 2731.740 ms oleg631-client.virtnet: 1 5539 0.34 MB/sec execute 17 sec latency 3732.016 ms oleg631-client.virtnet: 1 5Failover mds1 to oleg631-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 06:02:08 (1743501728) targets are mounted 06:02:08 (1743501728) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3840 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3707904 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3741696 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7449600 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 539 0.32 MB/sec execute 18 sec latency 4732.558 ms oleg631-client.virtnet: 1 5539 0.31 MB/sec execute 19 sec latency 5732.875 ms oleg631-client.virtnet: 1 5539 0.29 MB/sec execute 20 sec latency 6733.152 ms oleg631-client.virtnet: 1 5539 0.28 MB/sec execute 21 sec latency 7733.436 ms oleg631-client.virtnet: 1 5539 0.26 MB/sec execute 22 sec latency 8733.685 ms oleg631-client.virtnet: 1 5539 0.25 MB/sec execute 23 sec latency 9734.064 ms oleg631-client.virtnet: 1 5539 0.24 MB/sec execute 24 sec latency 10734.606 ms oleg631-client.virtnet: 1 5539 0.23 MB/sec execute 25 sec latency 11734.845 ms oleg631-client.virtnet: 1 5539 0.22 MB/sec execute 26 sec latency 12735.064 ms oleg631-client.virtnet: 1 5539 0.21 MB/sec execute 27 sec latency 13735.333 ms oleg631-client.virtnet: 1 5539 0.21 MB/sec execute 28 sec latency 14735.954 ms oleg631-client.virtnet: 1 5539 0.20 MB/sec execute 29 sec latency 15736.970 ms oleg631-client.virtnet: 1 5539 0.19 MB/sec execute 30 sec latency 16737.231 ms oleg631-client.virtnet: 1 5539 0.19 MB/sec execute 31 sec latency 17737.576 ms oleg631-client.virtnet: 1 5539 0.18 MB/sec execute 32 sec latency 18738.126 ms oleg631-client.virtnet: 1 5539 0.18 MB/sec execute 33 sec latency 19738.474 ms oleg631-client.virtnet: 1 5547 0.17 MB/sec execute 34 sec latency 20576.399 ms oleg631-client.virtnet: 1 5718 0.17 MB/sec execute 35 sec latency 75.943 ms oleg631-client.virtnet: 1 5907 0.20 MB/sec execute 36 sec latency 207.585 ms oleg631-client.virtnet: 1 5989 0.20 MB/sec execute 37 sec latency 178.967 ms oleg631-client.virtnet: 1 6160 0.22 MB/sec execute 38 sec latency 198.581 ms oleg631-client.virtnet: 1 6490 0.33 MB/sec execute 39 sec latency 192.480 ms oleg631-client.virtnet: 1 6779 0.34 MB/sec execute 40 sec latency 222.114 ms oleg631-client.virtnet: 1 7066 0.43 MB/sec execute 41 sec latency 118.597 ms oleg631-client.virtnet: 1 7260 0.43 MB/sec execute 42 sec latency 25.808 ms oleg631-client.virtnet: 1 7422 0.44 MB/sec execute 43 sec latency 34.326 ms oleg631-client.virtnet: 1 7554 0.44 MB/sec execute 44 sec latency 37.183 ms oleg631-client.virtnet: 1 7669 0.43 MB/sec execute 45 sec latency 56.316 ms oleg631-client.virtnet: 1 7785 0.43 MB/sec execute 46 sec latency 241.776 ms oleg631-client.virtnet: 1 7846 0.42 MB/sec execute 47 sec latency 163.371 ms oleg631-client.virtnet: 1 7960 0.42 MB/sec execute 48 sec latency 149.895 ms oleg631-client.virtnet: 1 8047 0.41 MB/sec execute 49 sec latency 177.725 ms oleg631-client.virtnet: 1 8244 0.43 MB/sec execute 50 sec latency 114.005 ms oleg631-client.virtnet: 1 8514 0.48 MB/sec execute 51 sec latency 61.396 ms oleg631-client.virtnet: 1 8674 0.47 MB/sec execute 52 sec latency 161.350 ms oleg631-client.virtnet: 1 8803 0.47 MB/sec execute 53 sec latency 206.396 ms oleg631-client.virtnet: 1 8936 0.46 MB/sec execute 54 sec latency 36.546 ms oleg631-client.virtnet: 1 9111 0.45 MB/sec execute 55 sec latency 133.685 ms oleg631-client.virtnet: 1 9364 0.45 MB/sec execute 56 sec latency 33.836 ms oleg631-client.virtnet: 1 9505 0.46 MB/sec execute 57 sec latency 117.560 ms oleg631-client.virtnet: 1 9593 0.47 MB/sec execute 58 sec latency 187.960 ms oleg631-client.virtnet: 1 9979 0.52 MB/sec execute 59 sec latency 154.301 ms oleg631-client.virtnet: 1 10106 0.53 MB/sec execute 60 sec latency 131.691 ms oleg631-client.virtnet: 1 10106 0.52 MB/sec execute 61 sec latency 1075.060 ms oleg631-client.virtnet: 1 10106 0.52 MB/sec execut06:02:31 (1743501751) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 06:02:55 (1743501775) targets are mounted 06:02:55 (1743501775) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3840 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3710976 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7456768 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg631-server e 62 sec latency 2075.298 ms oleg631-client.virtnet: 1 10106 0.51 MB/sec execute 63 sec latency 3075.541 ms oleg631-client.virtnet: 1 10106 0.50 MB/sec execute 64 sec latency 4075.765 ms oleg631-client.virtnet: 1 10106 0.49 MB/sec execute 65 sec latency 5075.975 ms oleg631-client.virtnet: 1 10106 0.48 MB/sec execute 66 sec latency 6076.248 ms oleg631-client.virtnet: 1 10106 0.48 MB/sec execute 67 sec latency 7076.454 ms oleg631-client.virtnet: 1 10106 0.47 MB/sec execute 68 sec latency 8076.753 ms oleg631-client.virtnet: 1 10106 0.46 MB/sec execute 69 sec latency 9077.019 ms oleg631-client.virtnet: 1 10106 0.46 MB/sec execute 70 sec latency 10077.331 ms oleg631-client.virtnet: 1 10106 0.45 MB/sec execute 71 sec latency 11077.575 ms oleg631-client.virtnet: 1 10106 0.44 MB/sec execute 72 sec latency 12077.850 ms oleg631-client.virtnet: 1 10106 0.44 MB/sec execute 73 sec latency 13078.067 ms oleg631-client.virtnet: 1 10106 0.43 MB/sec execute 74 sec latency 14078.336 ms oleg631-client.virtnet: 1 10106 0.43 MB/sec execute 75 sec latency 15078.606 ms oleg631-client.virtnet: 1 10106 0.42 MB/sec execute 76 sec latency 16078.904 ms oleg631-client.virtnet: 1 10106 0.42 MB/sec execute 77 sec latency 17079.152 ms oleg631-client.virtnet: 1 10106 0.41 MB/sec execute 78 sec latency 18079.422 ms oleg631-client.virtnet: 1 10106 0.41 MB/sec execute 79 sec latency 19079.710 ms oleg631-client.virtnet: 1 10106 0.40 MB/sec execute 80 sec latency 20079.984 ms oleg631-client.virtnet: 1 10296 0.40 MB/sec execute 81 sec latency 20474.311 ms oleg631-client.virtnet: 1 10471 0.41 MB/sec execute 82 sec latency 255.004 ms oleg631-client.virtnet: 1 10732 0.45 MB/sec execute 83 sec latency 28.996 ms oleg631-client.virtnet: 1 10882 0.46 MB/sec execute 84 sec latency 56.007 ms oleg631-client.virtnet: 1 11038 0.45 MB/sec execute 85 sec latency 28.507 ms oleg631-client.virtnet: 1 11217 0.45 MB/sec execute 86 sec latency 62.504 ms oleg631-client.virtnet: 1 11332 0.45 MB/sec execute 87 sec latency 184.342 ms oleg631-client.virtnet: 1 11393 0.44 MB/sec execute 88 sec latency 158.852 ms oleg631-client.virtnet: 1 11463 0.44 MB/sec execute 89 sec latency 202.814 ms oleg631-client.virtnet: 1 11575 0.44 MB/sec execute 90 sec latency 104.200 ms oleg631-client.virtnet: 1 11719 0.44 MB/sec execute 91 sec latency 137.393 ms oleg631-client.virtnet: 1 11880 0.44 MB/sec execute 92 sec latency 33.005 ms oleg631-client.virtnet: 1 12084 0.47 MB/sec execute 93 sec latency 42.933 ms oleg631-client.virtnet: 1 12213 0.46 MB/sec execute 94 sec latency 240.652 ms oleg631-client.virtnet: 1 12361 0.46 MB/sec execute 95 sec latency 154.434 ms oleg631-client.virtnet: 1 12496 0.46 MB/sec execute 96 sec latency 65.148 ms oleg631-client.virtnet: 1 12682 0.45 MB/sec execute 97 sec latency 146.181 ms oleg631-client.virtnet: 1 12887 0.45 MB/sec execute 98 sec latency 30.824 ms oleg631-client.virtnet: 1 13030 0.46 MB/sec execute 99 sec latency 296.298 ms oleg631-client.virtnet: 1 13117 0.46 MB/sec execute 100 sec latency 230.361 ms oleg631-client.virtnet: 1 13260 0.46 MB/sec execute 101 sec latency 181.186 ms oleg631-client.virtnet: 1 13581 0.50 MB/sec execute 102 sec latency 183.619 ms oleg631-client.virtnet: 1 13926 0.51 MB/sec execute 103 sec latency 105.429 ms oleg631-client.virtnet: 1 14179 0.54 MB/sec execute 104 sec latency 72.282 ms oleg631-client.virtnet: 1 14319 0.54 MB/sec execute 105 sec latency 34.931 ms oleg631-client.virtnet: 1 14448 0.54 MB/sec execute 106 sec latency 57.373 Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 06:03:19 (1743501799) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 06:03:51 (1743501831) targets are mounted 06:03:51 (1743501831) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec ms oleg631-client.virtnet: 1 14586 0.54 MB/sec execute 107 sec latency 39.494 ms oleg631-client.virtnet: 1 14622 0.53 MB/sec execute 108 sec latency 702.280 ms oleg631-client.virtnet: 1 14622 0.53 MB/sec execute 109 sec latency 1702.514 ms oleg631-client.virtnet: 1 14622 0.52 MB/sec execute 110 sec latency 2702.824 ms oleg631-client.virtnet: 1 14622 0.52 MB/sec execute 111 sec latency 3703.052 ms oleg631-client.virtnet: 1 14622 0.51 MB/sec execute 112 sec latency 4703.343 ms oleg631-client.virtnet: 1 14622 0.51 MB/sec execute 113 sec latency 5703.579 ms oleg631-client.virtnet: 1 14622 0.51 MB/sec execute 114 sec latency 6703.893 ms oleg631-client.virtnet: 1 14622 0.50 MB/sec execute 115 sec latency 7704.186 ms oleg631-client.virtnet: 1 14622 0.50 MB/sec execute 116 sec latency 8704.515 ms oleg631-client.virtnet: 1 14622 0.49 MB/sec execute 117 sec latency 9704.851 ms oleg631-client.virtnet: 1 14622 0.49 MB/sec execute 118 sec latency 10705.262 ms oleg631-client.virtnet: 1 14622 0.48 MB/sec execute 119 sec latency 11705.552 ms oleg631-client.virtnet: 1 14622 0.48 MB/sec execute 120 sec latency 12705.754 ms oleg631-client.virtnet: 1 14622 0.48 MB/sec execute 121 sec latency 13706.032 ms oleg631-client.virtnet: 1 14622 0.47 MB/sec execute 122 sec latency 14706.309 ms oleg631-client.virtnet: 1 14622 0.47 MB/sec execute 123 sec latency 15706.536 ms oleg631-client.virtnet: 1 14622 0.46 MB/sec execute 124 sec latency 16706.805 ms oleg631-client.virtnet: 1 14622 0.46 MB/sec execute 125 sec latency 17707.065 ms oleg631-client.virtnet: 1 14622 0.46 MB/sec execute 126 sec latency 18707.324 ms oleg631-client.virtnet: 1 14622 0.45 MB/sec execute 127 sec latency 19707.615 ms oleg631-client.virtnet: 1 14622 0.45 MB/sec execute 128 sec latency 20707.870 ms oleg631-client.virtnet: 1 14622 0.45 MB/sec execute 129 sec latency 21708.126 ms oleg631-client.virtnet: 1 14622 0.44 MB/sec execute 130 sec latency 22708.403 ms oleg631-client.virtnet: 1 14622 0.44 MB/sec execute 131 sec latency 23708.650 ms oleg631-client.virtnet: 1 14622 0.44 MB/sec execute 132 sec latency 24708.874 ms oleg631-client.virtnet: 1 14622 0.43 MB/sec execute 133 sec latency 25709.136 ms oleg631-client.virtnet: 1 14622 0.43 MB/sec execute 134 sec latency 26709.552 ms oleg631-client.virtnet: 1 14622 0.43 MB/sec execute 135 sec latency 27709.828 ms oleg631-client.virtnet: 1 14622 0.42 MB/sec execute 136 sec latency 28710.117 ms oleg631-client.virtnet: 1 14622 0.42 MB/sec execute 137 sec latency 29710.313 ms oleg631-client.virtnet: 1 14622 0.42 MB/sec execute 138 sec latency 30711.324 ms oleg631-client.virtnet: 1 14622 0.41 MB/sec execute 139 sec latency 31711.630 ms oleg631-client.virtnet: 1 14622 0.41 MB/sec execute 140 sec latency 32711.898 ms oleg631-client.virtnet: 1 14622 0.41 MB/sec execute 141 sec latency 33712.169 ms oleg631-client.virtnet: 1 14622 0.41 MB/sec execute 142 sec latency 34712.528 ms oleg631-client.virtnet: 1 14622 0.40 MB/sec execute 143 sec latency 35712.808 ms oleg631-client.virtnet: 1 14622 0.40 MB/sec execute 144 sec latency 36712.993 ms oleg631-client.virtnet: 1 14691 0.40 MB/sec execute 145 sec latency 37029.750 ms oleg631-client.virtnet: 1 14802 0.39 MB/sec execute 146 sec latency 55.663 ms oleg631-client.virtnet: 1 14879 0.39 MB/sec execute 147 sec latency 365.585 ms oleg631-client.virtnet: 1 14954 0.39 MB/sec execute 148 sec latency 122.380 ms oleg631-client.virtnet: 1 15005 0.39 MB/sec execute 149 sec latency 244.395 ms oleg631-client.virtnet: 1 15098 0.39 MB/sec execute 150 sec lUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3840 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3712000 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7457792 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 06:04:16 (1743501856) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 06:04:39 (1743501879) targets are mounted 06:04:39 (1743501879) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid atency 134.907 ms oleg631-client.virtnet: 1 15185 0.39 MB/sec execute 151 sec latency 136.613 ms oleg631-client.virtnet: 1 15400 0.39 MB/sec execute 152 sec latency 180.877 ms oleg631-client.virtnet: 1 15624 0.41 MB/sec execute 153 sec latency 47.463 ms oleg631-client.virtnet: 1 15781 0.41 MB/sec execute 154 sec latency 185.281 ms oleg631-client.virtnet: 1 15892 0.41 MB/sec execute 155 sec latency 219.306 ms oleg631-client.virtnet: 1 16026 0.40 MB/sec execute 156 sec latency 42.815 ms oleg631-client.virtnet: 1 16175 0.40 MB/sec execute 157 sec latency 187.658 ms oleg631-client.virtnet: 1 16348 0.40 MB/sec execute 158 sec latency 28.155 ms oleg631-client.virtnet: 1 16541 0.41 MB/sec execute 159 sec latency 153.702 ms oleg631-client.virtnet: 1 16683 0.41 MB/sec execute 160 sec latency 185.374 ms oleg631-client.virtnet: 1 16963 0.43 MB/sec execute 161 sec latency 120.789 ms oleg631-client.virtnet: 1 17174 0.43 MB/sec execute 162 sec latency 154.673 ms oleg631-client.virtnet: 1 17467 0.44 MB/sec execute 163 sec latency 148.156 ms oleg631-client.virtnet: 1 17727 0.46 MB/sec execute 164 sec latency 85.102 ms oleg631-client.virtnet: 1 17727 0.45 MB/sec execute 165 sec latency 1052.309 ms oleg631-client.virtnet: 1 17727 0.45 MB/sec execute 166 sec latency 2052.996 ms oleg631-client.virtnet: 1 17727 0.45 MB/sec execute 167 sec latency 3053.251 ms oleg631-client.virtnet: 1 17727 0.45 MB/sec execute 168 sec latency 4053.940 ms oleg631-client.virtnet: 1 17727 0.44 MB/sec execute 169 sec latency 5054.382 ms oleg631-client.virtnet: 1 17727 0.44 MB/sec execute 170 sec latency 6054.715 ms oleg631-client.virtnet: 1 17727 0.44 MB/sec execute 171 sec latency 7054.989 ms oleg631-client.virtnet: 1 17727 0.44 MB/sec execute 172 sec latency 8055.269 ms oleg631-client.virtnet: 1 17727 0.43 MB/sec execute 173 sec latency 9055.517 ms oleg631-client.virtnet: 1 17727 0.43 MB/sec execute 174 sec latency 10055.743 ms oleg631-client.virtnet: 1 17727 0.43 MB/sec execute 175 sec latency 11056.001 ms oleg631-client.virtnet: 1 17727 0.43 MB/sec execute 176 sec latency 12056.311 ms oleg631-client.virtnet: 1 17727 0.42 MB/sec execute 177 sec latency 13056.519 ms oleg631-client.virtnet: 1 17727 0.42 MB/sec execute 178 sec latency 14056.983 ms oleg631-client.virtnet: 1 17727 0.42 MB/sec execute 179 sec latency 15057.329 ms oleg631-client.virtnet: 1 17727 0.42 MB/sec execute 180 sec latency 16057.713 ms oleg631-client.virtnet: 1 17727 0.41 MB/sec execute 181 sec latency 17057.951 ms oleg631-client.virtnet: 1 17727 0.41 MB/sec execute 182 sec latency 18058.314 ms oleg631-client.virtnet: 1 17727 0.41 MB/sec execute 183 sec latency 19058.837 ms oleg631-client.virtnet: 1 17727 0.41 MB/sec execute 184 sec latency 20059.723 ms oleg631-client.virtnet: 1 17732 0.41 MB/sec execute 185 sec latency 20888.377 ms oleg631-client.virtnet: 1 17898 0.41 MB/sec execute 186 sec latency 28.119 ms oleg631-client.virtnet: 1 18043 0.41 MB/sec execute 187 sec latency 31.774 ms oleg631-client.virtnet: 1 18170 0.41 MB/sec execute 188 sec latency 42.329 ms oleg631-client.virtnet: 1 18296 0.41 MB/sec execute 189 sec latency 63.188 ms oleg631-client.virtnet: 1 18420 0.41 MB/sec execute 190 sec latency 70.924 ms oleg631-client.virtnet: 1 18476 0.40 MB/sec execute 191 sec latency 298.201 ms oleg631-client.virtnet: 1 18546 0.40 MB/sec execute 192 sec latency 160.468 ms oleg631-client.virtnet: 1 18630 0.40 MB/sec execute 193 sec latency 199.764 ms oleg631-client.virtnet: 1 18733 0.40 MB/sec execute 194 sec latency 127.408 ms oleg63mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3968 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3706880 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7452672 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 06:05:03 (1743501903) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1-client.virtnet: 1 18883 0.40 MB/sec execute 195 sec latency 138.597 ms oleg631-client.virtnet: 1 19011 0.40 MB/sec execute 196 sec latency 63.553 ms oleg631-client.virtnet: 1 19232 0.42 MB/sec execute 197 sec latency 40.856 ms oleg631-client.virtnet: 1 19338 0.41 MB/sec execute 198 sec latency 145.933 ms oleg631-client.virtnet: 1 19486 0.41 MB/sec execute 199 sec latency 135.525 ms oleg631-client.virtnet: 1 19637 0.41 MB/sec execute 200 sec latency 124.729 ms oleg631-client.virtnet: 1 19896 0.41 MB/sec execute 201 sec latency 72.569 ms oleg631-client.virtnet: 1 20130 0.41 MB/sec execute 202 sec latency 26.100 ms oleg631-client.virtnet: 1 20201 0.41 MB/sec execute 203 sec latency 196.153 ms oleg631-client.virtnet: 1 20385 0.42 MB/sec execute 204 sec latency 201.809 ms oleg631-client.virtnet: 1 20702 0.44 MB/sec execute 205 sec latency 263.402 ms oleg631-client.virtnet: 1 20989 0.44 MB/sec execute 206 sec latency 202.352 ms oleg631-client.virtnet: 1 21316 0.46 MB/sec execute 207 sec latency 65.902 ms oleg631-client.virtnet: 1 21485 0.45 MB/sec execute 208 sec latency 31.444 ms oleg631-client.virtnet: 1 21633 0.46 MB/sec execute 209 sec latency 43.945 ms oleg631-client.virtnet: 1 21797 0.46 MB/sec execute 210 sec latency 26.056 ms oleg631-client.virtnet: 1 21920 0.45 MB/sec execute 211 sec latency 242.276 ms oleg631-client.virtnet: 1 21920 0.45 MB/sec execute 212 sec latency 1242.540 ms oleg631-client.virtnet: 1 21920 0.45 MB/sec execute 213 sec latency 2242.794 ms oleg631-client.virtnet: 1 21920 0.45 MB/sec execute 214 sec latency 3243.073 ms oleg631-client.virtnet: 1 21920 0.45 MB/sec execute 215 sec latency 4243.313 ms oleg631-client.virtnet: 1 21920 0.44 MB/sec execute 216 sec latency 5243.594 ms oleg631-client.virtnet: 1 21920 0.44 MB/sec execute 217 sec latency 6243.837 ms oleg631-client.virtnet: 1 21920 0.44 MB/sec execute 218 sec latency 7244.108 ms oleg631-client.virtnet: 1 21920 0.44 MB/sec execute 219 sec latency 8244.464 ms oleg631-client.virtnet: 1 21920 0.44 MB/sec execute 220 sec latency 9244.714 ms oleg631-client.virtnet: 1 21920 0.43 MB/sec execute 221 sec latency 10245.085 ms oleg631-client.virtnet: 1 21920 0.43 MB/sec execute 222 sec latency 11245.508 ms oleg631-client.virtnet: 1 21920 0.43 MB/sec execute 223 sec latency 12245.777 ms oleg631-client.virtnet: 1 21920 0.43 MB/sec execute 224 sec latency 13246.112 ms oleg631-client.virtnet: 1 21920 0.43 MB/sec execute 225 sec latency 14246.476 ms oleg631-client.virtnet: 1 21920 0.42 MB/sec execute 226 sec latency 15246.729 ms oleg631-client.virtnet: 1 21920 0.42 MB/sec execute 227 sec latency 16246.931 ms oleg631-client.virtnet: 1 21920 0.42 MB/sec execute 228 sec latency 17247.253 ms oleg631-client.virtnet: 1 21920 0.42 MB/sec execute 229 sec latency 18247.524 ms oleg631-client.virtnet: 1 21920 0.42 MB/sec execute 230 sec latency 19247.786 ms oleg631-client.virtnet: 1 21920 0.42 MB/sec execute 231 sec latency 20248.000 ms oleg631-client.virtnet: 1 21920 0.41 MB/sec execute 232 sec latency 21248.278 ms oleg631-client.virtnet: 1 21920 0.41 MB/sec execute 233 sec latency 22248.737 ms oleg631-client.virtnet: 1 21920 0.41 MB/sec execute 234 sec latency 23248.999 ms oleg631-client.virtnet: 1 21920 0.41 MB/sec execute 235 sec latency 24249.308 ms oleg631-client.virtnet: 1 21920 0.41 MB/sec execute 236 sec latency 25249.646 ms oleg631-client.virtnet: 1 21920 0.40 MB/sec execute 237 sec latency 26249.913 ms oleg631-client.virtnet: 1 21920 0.40 MB/sec execute 238 sec latency 27250.215 ms oleg631-client.violeg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 06:05:34 (1743501934) targets are mounted 06:05:34 (1743501934) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec rtnet: 1 21920 0.40 MB/sec execute 239 sec latency 28250.469 ms oleg631-client.virtnet: 1 21920 0.40 MB/sec execute 240 sec latency 29250.728 ms oleg631-client.virtnet: 1 21920 0.40 MB/sec execute 241 sec latency 30251.071 ms oleg631-client.virtnet: 1 21920 0.40 MB/sec execute 242 sec latency 31251.460 ms oleg631-client.virtnet: 1 21920 0.39 MB/sec execute 243 sec latency 32253.412 ms oleg631-client.virtnet: 1 21920 0.39 MB/sec execute 244 sec latency 33253.685 ms oleg631-client.virtnet: 1 21920 0.39 MB/sec execute 245 sec latency 34253.910 ms oleg631-client.virtnet: 1 21920 0.39 MB/sec execute 246 sec latency 35254.176 ms oleg631-client.virtnet: 1 21920 0.39 MB/sec execute 247 sec latency 36254.471 ms oleg631-client.virtnet: 1 21920 0.39 MB/sec execute 248 sec latency 37254.716 ms oleg631-client.virtnet: 1 21920 0.39 MB/sec execute 249 sec latency 38255.676 ms oleg631-client.virtnet: 1 21920 0.38 MB/sec execute 250 sec latency 39256.141 ms oleg631-client.virtnet: 1 21920 0.38 MB/sec execute 251 sec latency 40256.345 ms oleg631-client.virtnet: 1 22001 0.38 MB/sec execute 252 sec latency 40369.871 ms oleg631-client.virtnet: 1 22057 0.38 MB/sec execute 253 sec latency 310.164 ms oleg631-client.virtnet: 1 22125 0.38 MB/sec execute 254 sec latency 190.409 ms oleg631-client.virtnet: 1 22213 0.38 MB/sec execute 255 sec latency 132.941 ms oleg631-client.virtnet: 1 22304 0.38 MB/sec execute 256 sec latency 140.190 ms oleg631-client.virtnet: 1 22479 0.38 MB/sec execute 257 sec latency 108.627 ms oleg631-client.virtnet: 1 22627 0.38 MB/sec execute 258 sec latency 43.759 ms oleg631-client.virtnet: 1 22902 0.39 MB/sec execute 259 sec latency 125.925 ms oleg631-client.virtnet: 1 23043 0.39 MB/sec execute 260 sec latency 122.397 ms oleg631-client.virtnet: 1 23216 0.39 MB/sec execute 261 sec latency 171.885 ms oleg631-client.virtnet: 1 23489 0.39 MB/sec execute 262 sec latency 95.364 ms oleg631-client.virtnet: 1 23726 0.39 MB/sec execute 263 sec latency 128.729 ms oleg631-client.virtnet: 1 23921 0.39 MB/sec execute 264 sec latency 123.323 ms oleg631-client.virtnet: 1 24305 0.41 MB/sec execute 265 sec latency 145.602 ms oleg631-client.virtnet: 1 24662 0.41 MB/sec execute 266 sec latency 112.274 ms oleg631-client.virtnet: 1 25004 0.43 MB/sec execute 267 sec latency 42.269 ms oleg631-client.virtnet: 1 25247 0.43 MB/sec execute 268 sec latency 17.651 ms oleg631-client.virtnet: 1 25432 0.43 MB/sec execute 269 sec latency 36.374 ms oleg631-client.virtnet: 1 25588 0.43 MB/sec execute 270 sec latency 315.451 ms oleg631-client.virtnet: 1 25684 0.43 MB/sec execute 271 sec latency 140.430 ms oleg631-client.virtnet: 1 25838 0.43 MB/sec execute 272 sec latency 103.664 ms oleg631-client.virtnet: 1 26062 0.43 MB/sec execute 273 sec latency 93.167 ms oleg631-client.virtnet: 1 26363 0.44 MB/sec execute 274 sec latency 91.460 ms oleg631-client.virtnet: 1 26541 0.44 MB/sec execute 275 sec latency 158.758 ms oleg631-client.virtnet: 1 26765 0.44 MB/sec execute 276 sec latency 19.056 ms oleg631-client.virtnet: 1 27028 0.44 MB/sec execute 277 sec latency 119.911 ms oleg631-client.virtnet: 1 27252 0.44 MB/sec execute 278 sec latency 196.668 ms oleg631-client.virtnet: 1 27456 0.44 MB/sec execute 279 sec latency 164.839 ms oleg631-client.virtnet: 1 27850 0.46 MB/sec execute 280 sec latency 129.517 ms oleg631-client.virtnet: 1 28187 0.46 MB/sec execute 281 sec latency 113.505 ms oleg631-client.virtnet: 1 28479 0.47 MB/sec execute 282 sec latency 60.997 ms oleg631-client.virtnet: 1 28688 0.47 MB/sec execute 283 sec latency 24.601 ms oleg631-client.virtnet: 1 28890 0.47 MB/sec execute 284 sec latency 25.194 ms oleg631-client.virtnet: 1 29093 0.47 MB/sec execute 285 sec latency 256.765 ms oleg631-client.virtnet: 1 29169 0.47 MB/sec execute 286 sec latency 299.349 ms oleg631-client.virtnet: 1 29300 0.47 MB/sec execute 287 sec latency 174.101 ms oleg631-client.virtnet: 1 29407 0.47 MB/sec execute 288 sec latency 135.860 ms oleg631-client.virtnet: 1 29652 0.47 MB/sec execute 289 sec latency 59.048 ms oleg631-client.virtnet: 1 29910 0.48 MB/sec execute 290 sec latency 159.816 ms oleg631-client.virtnet: 1 30093 0.48 MB/sec execute 291 sec latency 107.937 ms oleg631-client.virtnet: 1 30278 0.48 MB/sec execute 292 sec latency 23.813 ms oleg631-client.virtnet: 1 30463 0.48 MB/sec execute 293 sec latency 134.212 ms oleg631-client.virtnet: 1 30653 0.48 MB/sec execute 294 sec latency 32.369 ms oleg631-client.virtnet: 1 30835 0.48 MB/sec execute 295 sec latency 187.242 ms oleg631-client.virtnet: 1 31029 0.48 MB/sec execute 296 sec latency 175.464 ms oleg631-client.virtnet: 1 31408 0.49 MB/sec execute 297 sec latency 147.968 ms oleg631-client.virtnet: 1 31787 0.50 MB/sec execute 298 sec latency 106.775 ms oleg631-client.virtnet: 1 32113 0.51 MB/sec execute 299 sec latency 18.902 ms oleg631-client.virtnet: 1 cleanup 300 sec oleg631-client.virtnet: 0 cleanup 301 sec oleg631-client.virtnet: oleg631-client.virtnet: Operation Count AvgLat MaxLat oleg631-client.virtnet: ---------------------------------------- oleg631-client.virtnet: NTCreateX 4953 25.717 37029.724 oleg631-client.virtnet: Close 3629 8.691 20888.360 oleg631-client.virtnet: Rename 208 21.696 63.501 oleg631-client.virtnet: Unlink 1002 68.386 40369.853 oleg631-client.virtnet: Qpathinfo 4468 3.901 63.163 oleg631-client.virtnet: Qfileinfo 777 0.428 11.592 oleg631-client.virtnet: Qfsinfo 821 0.296 34.349 oleg631-client.virtnet: Sfileinfo 400 13.999 63.686 oleg631-client.virtnet: Find 1725 1.791 73.972 oleg631-client.virtnet: WriteX 2431 2.041 19.727 oleg631-client.virtnet: ReadX 7713 0.046 5.354 oleg631-client.virtnet: LockX 16 2.852 6.530 oleg631-client.virtnet: UnlockX 16 2.555 4.576 oleg631-client.virtnet: Flush 344 103.608 365.565 oleg631-client.virtnet: oleg631-client.virtnet: Throughput 0.50831 MB/sec 1 clients 1 procs max_latency=40369.871 ms oleg631-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg631-client.virtnet at Tue Apr 1 06:06:30 EDT 2025 with return code 0 oleg631-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg631-client.virtnet oleg631-client.virtnet: /mnt/lustre/d70b.replay-single/oleg631-client.virtnet /mnt/lustre/d70b.replay-single/oleg631-client.virtnet oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp/WORD' oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp/PARADOX' oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp/EXCEL' oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp/ACCESS' oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp/SEED' oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp/PM' oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp/WORDPRO' oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp/PWRPNT' oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp/COREL' oleg631-client.virtnet: removed directory 'clients/client0/~dmtmp' oleg631-client.virtnet: removed directory 'clients/client0' oleg631-client.virtnet: removed directory 'clients' oleg631-client.virtnet: removed 'client.txt' oleg631-client.virtnet: /mnt/lustre/d70b.replay-single/oleg631-client.virtnet oleg631-client.virtnet: dbench successfully finished