== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 18:28:31 (1717021711) Starting client oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre Started clients oleg256-client.virtnet: 192.168.202.156@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg256-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=12279 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3584 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3716096 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7452672 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server reboot facets: mds1 Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3840 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3687424 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3723264 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7410688 1% /mnt/lustre oleg256-client.virtnet: looking for dbench program oleg256-client.virtnet: /usr/bin/dbench oleg256-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg256-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg256-client.virtnet' oleg256-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg256-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg256-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg256-client.virtnet at Wed May 29 18:28:33 EDT 2024 oleg256-client.virtnet: waiting for dbench pid 12304 oleg256-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg256-client.virtnet: oleg256-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg256-client.virtnet: failed to create barrier semaphore oleg256-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg256-client.virtnet: releasing clients oleg256-client.virtnet: 1 223 8.62 MB/sec warmup 1 sec latency 48.479 ms oleg256-client.virtnet: 1 476 8.33 MB/sec warmup 2 sec latency 32.607 ms oleg256-client.virtnet: 1 671 7.05 MB/sec warmup 3 sec latency 88.059 ms oleg256-client.virtnet: 1 765 5.33 MB/sec warmup 4 sec latency 87.439 ms oleg256-client.virtnet: 1 910 4.35 MB/sec warmup 5 sec latency 80.463 ms oleg256-client.virtnet: 1 1086 3.82 MB/sec warmup 6 sec latency 103.825 ms oleg256-client.virtnet: 1 1180 3.29 MB/sec warmup 7 sec latency 740.042 ms oleg256-client.virtnet: 1 1180 2.87 MB/sec warmup 8 sec latency 1740.870 ms oleg256-client.virtnet: 1 1180 2.56 MB/sec warmup 9 sec latency 2741.084 ms oleg256-client.virtnet: 1 1180 2.30 MB/sec warmup 10 sec latency 3742.070 ms oleg256-client.virtnet: 1 1180 2.09 MB/sec warmup 11 sec latency 4743.250 ms oleg256-client.virtnet: 1 1180 1.92 MB/sec warmup 12 sec latency 5745.430 ms oleg256-client.virtnet: 1 1180 1.77 MB/sec warmup 13 sec latency 6746.795 ms oleg256-client.virtnet: 1 1180 1.64 MB/sec warmup 14 sec latency 7746.942 ms oleg256-client.virtnet: 1 1180 1.53 MB/sec warmup 15 sec latency 8747.117 ms oleg256-client.virtnet: 1 1180 1.44 MB/sec warmup 16 sec latency 9747.672 ms oleg256-client.virtnet: 1 1180 1.35 MB/sec warmup 17 sec latency 10748.108 ms oleg256-client.virtnet: 1 1180 1.28 MB/sec warmup 18 sec latency 11753.926 ms oleg256-client.virtnet: 1 1180 1.21 MB/sec warmup 19 sec latency 12755.581 ms oleg256-client.virtnet: 1 1180 1.15 MB/sec warmup 20 sec latency 13755.749 ms oleg256-client.virtnet: 1 1180 1.09 MB/sec warmup 21 sec latency 14755.876 ms oleg256-client.virtnet: 1 1180 1.05 MB/sec warmup 22 sec latency 15756.026 ms oleg256-client.virtnet: 1 1180 1.00 MB/sec warmup 23 sec latency 16756.186 ms oleg256-client.virtnet: 1 1229 0.96 MB/sec warmup 24 sec latency 17490.456 ms oleg256-client.virtnet: 1 1481 1.04 MB/sec warmup 25 sec latency 118.268 ms oleg256-client.virtnet: 1 1632 1.01 MB/sec warmup 26 sec latency 173.831 ms oleg256-client.virtnet: 1 1757 0.97 MB/sec warmup 27 sec latency 165.688 ms oleg256-client.virtnet: 1 1886 0.94 MB/sec warmup 28 sec latency 148.181 ms oleg256-client.virtnet: 1 2063 0.92 MB/sec warmup 29 sec latency 86.755 ms oleg256-client.virtnet: 1 2345 0.93 MB/sec warmup 30 sec latency 27.697 ms oleg256-client.virtnet: 1 2477 0.93 MB/sec warmup 31 sec latency 98.310 ms oleg256-client.virtnet: 1 2784 1.00 MB/sec warmup 32 sec latency 155.862 ms oleg256-client.virtnet: 1 3000 1.02 MB/sec warmup 33 stest_70b fail mds1 2 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server reboot facets: mds1 Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3840 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7464960 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server reboot facets: mds1 ec latency 125.671 ms oleg256-client.virtnet: 1 3359 1.04 MB/sec warmup 34 sec latency 114.061 ms oleg256-client.virtnet: 1 3636 1.10 MB/sec warmup 35 sec latency 37.638 ms oleg256-client.virtnet: 1 3774 1.10 MB/sec warmup 36 sec latency 239.671 ms oleg256-client.virtnet: 1 3774 1.07 MB/sec warmup 37 sec latency 1239.946 ms oleg256-client.virtnet: 1 3774 1.05 MB/sec warmup 38 sec latency 2240.115 ms oleg256-client.virtnet: 1 3774 1.02 MB/sec warmup 39 sec latency 3241.177 ms oleg256-client.virtnet: 1 3774 0.99 MB/sec warmup 40 sec latency 4241.339 ms oleg256-client.virtnet: 1 3774 0.97 MB/sec warmup 41 sec latency 5241.500 ms oleg256-client.virtnet: 1 3774 0.95 MB/sec warmup 42 sec latency 6242.152 ms oleg256-client.virtnet: 1 3774 0.92 MB/sec warmup 43 sec latency 7242.340 ms oleg256-client.virtnet: 1 3774 0.90 MB/sec warmup 44 sec latency 8243.036 ms oleg256-client.virtnet: 1 3774 0.88 MB/sec warmup 45 sec latency 9243.302 ms oleg256-client.virtnet: 1 3774 0.86 MB/sec warmup 46 sec latency 10243.473 ms oleg256-client.virtnet: 1 3774 0.85 MB/sec warmup 47 sec latency 11245.420 ms oleg256-client.virtnet: 1 3774 0.83 MB/sec warmup 48 sec latency 12245.762 ms oleg256-client.virtnet: 1 3774 0.81 MB/sec warmup 49 sec latency 13246.134 ms oleg256-client.virtnet: 1 3774 0.79 MB/sec warmup 50 sec latency 14246.637 ms oleg256-client.virtnet: 1 3774 0.78 MB/sec warmup 51 sec latency 15246.809 ms oleg256-client.virtnet: 1 3774 0.76 MB/sec warmup 52 sec latency 16246.978 ms oleg256-client.virtnet: 1 3774 0.75 MB/sec warmup 53 sec latency 17248.967 ms oleg256-client.virtnet: 1 3774 0.74 MB/sec warmup 54 sec latency 18249.632 ms oleg256-client.virtnet: 1 3774 0.72 MB/sec warmup 55 sec latency 19249.767 ms oleg256-client.virtnet: 1 3801 0.71 MB/sec warmup 56 sec latency 19974.701 ms oleg256-client.virtnet: 1 3928 0.70 MB/sec warmup 57 sec latency 51.589 ms oleg256-client.virtnet: 1 4108 0.69 MB/sec warmup 58 sec latency 23.537 ms oleg256-client.virtnet: 1 4236 0.68 MB/sec warmup 59 sec latency 194.391 ms oleg256-client.virtnet: 1 4362 0.17 MB/sec execute 1 sec latency 214.454 ms oleg256-client.virtnet: 1 4446 0.18 MB/sec execute 2 sec latency 131.093 ms oleg256-client.virtnet: 1 4578 0.34 MB/sec execute 3 sec latency 84.029 ms oleg256-client.virtnet: 1 4825 0.45 MB/sec execute 4 sec latency 31.539 ms oleg256-client.virtnet: 1 5065 0.95 MB/sec execute 5 sec latency 188.244 ms oleg256-client.virtnet: 1 5218 0.82 MB/sec execute 6 sec latency 70.764 ms oleg256-client.virtnet: 1 5421 0.72 MB/sec execute 7 sec latency 39.452 ms oleg256-client.virtnet: 1 5614 0.67 MB/sec execute 8 sec latency 80.717 ms oleg256-client.virtnet: 1 5752 0.61 MB/sec execute 9 sec latency 406.367 ms oleg256-client.virtnet: 1 5752 0.55 MB/sec execute 10 sec latency 1406.591 ms oleg256-client.virtnet: 1 5752 0.50 MB/sec execute 11 sec latency 2407.197 ms oleg256-client.virtnet: 1 5752 0.45 MB/sec execute 12 sec latency 3408.508 ms oleg256-client.virtnet: 1 5752 0.42 MB/sec execute 13 sec latency 4408.676 ms oleg256-client.virtnet: 1 5752 0.39 MB/sec execute 14 sec latency 5408.979 ms oleg256-client.virtnet: 1 5752 0.36 MB/sec execute 15 sec latency 6409.566 ms oleg256-client.virtnet: 1 5752 0.34 MB/sec execute 16 sec latency 7410.952 ms oleg256-client.virtnet: 1 5752 0.32 MB/sec execute 17 sec latency 8414.788 ms oleg256-client.virtnet: 1 5752 0.30 MB/sec execute 18 sec latency 9415.081 ms oleg256-client.virtFailover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3840 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7456768 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server reboot facets: mds1 net: 1 5752 0.29 MB/sec execute 19 sec latency 10415.480 ms oleg256-client.virtnet: 1 5752 0.27 MB/sec execute 20 sec latency 11416.150 ms oleg256-client.virtnet: 1 5752 0.26 MB/sec execute 21 sec latency 12416.863 ms oleg256-client.virtnet: 1 5752 0.25 MB/sec execute 22 sec latency 13418.549 ms oleg256-client.virtnet: 1 5752 0.24 MB/sec execute 23 sec latency 14419.835 ms oleg256-client.virtnet: 1 5752 0.23 MB/sec execute 24 sec latency 15419.992 ms oleg256-client.virtnet: 1 5752 0.22 MB/sec execute 25 sec latency 16420.154 ms oleg256-client.virtnet: 1 5752 0.21 MB/sec execute 26 sec latency 17420.338 ms oleg256-client.virtnet: 1 5752 0.20 MB/sec execute 27 sec latency 18420.728 ms oleg256-client.virtnet: 1 5752 0.19 MB/sec execute 28 sec latency 19421.624 ms oleg256-client.virtnet: 1 5752 0.19 MB/sec execute 29 sec latency 20422.169 ms oleg256-client.virtnet: 1 5752 0.18 MB/sec execute 30 sec latency 21422.341 ms oleg256-client.virtnet: 1 5752 0.18 MB/sec execute 31 sec latency 22422.485 ms oleg256-client.virtnet: 1 5752 0.17 MB/sec execute 32 sec latency 23423.013 ms oleg256-client.virtnet: 1 5752 0.17 MB/sec execute 33 sec latency 24423.717 ms oleg256-client.virtnet: 1 5752 0.16 MB/sec execute 34 sec latency 25425.126 ms oleg256-client.virtnet: 1 5752 0.16 MB/sec execute 35 sec latency 26425.314 ms oleg256-client.virtnet: 1 5752 0.15 MB/sec execute 36 sec latency 27425.603 ms oleg256-client.virtnet: 1 5752 0.15 MB/sec execute 37 sec latency 28426.401 ms oleg256-client.virtnet: 1 5752 0.14 MB/sec execute 38 sec latency 29426.812 ms oleg256-client.virtnet: 1 5887 0.17 MB/sec execute 39 sec latency 30013.016 ms oleg256-client.virtnet: 1 5996 0.17 MB/sec execute 40 sec latency 119.306 ms oleg256-client.virtnet: 1 6162 0.19 MB/sec execute 41 sec latency 78.041 ms oleg256-client.virtnet: 1 6486 0.29 MB/sec execute 42 sec latency 107.790 ms oleg256-client.virtnet: 1 6859 0.33 MB/sec execute 43 sec latency 101.074 ms oleg256-client.virtnet: 1 7175 0.39 MB/sec execute 44 sec latency 69.401 ms oleg256-client.virtnet: 1 7356 0.41 MB/sec execute 45 sec latency 37.916 ms oleg256-client.virtnet: 1 7503 0.41 MB/sec execute 46 sec latency 31.819 ms oleg256-client.virtnet: 1 7695 0.40 MB/sec execute 47 sec latency 33.973 ms oleg256-client.virtnet: 1 7823 0.40 MB/sec execute 48 sec latency 92.334 ms oleg256-client.virtnet: 1 7853 0.39 MB/sec execute 49 sec latency 433.239 ms oleg256-client.virtnet: 1 7853 0.38 MB/sec execute 50 sec latency 1433.564 ms oleg256-client.virtnet: 1 7853 0.38 MB/sec execute 51 sec latency 2434.545 ms oleg256-client.virtnet: 1 7853 0.37 MB/sec execute 52 sec latency 3434.852 ms oleg256-client.virtnet: 1 7853 0.36 MB/sec execute 53 sec latency 4435.024 ms oleg256-client.virtnet: 1 7853 0.35 MB/sec execute 54 sec latency 5435.258 ms oleg256-client.virtnet: 1 7853 0.35 MB/sec execute 55 sec latency 6435.426 ms oleg256-client.virtnet: 1 7853 0.34 MB/sec execute 56 sec latency 7435.582 ms oleg256-client.virtnet: 1 7853 0.34 MB/sec execute 57 sec latency 8436.026 ms oleg256-client.virtnet: 1 7853 0.33 MB/sec execute 58 sec latency 9436.197 ms oleg256-client.virtnet: 1 7853 0.32 MB/sec execute 59 sec latency 10436.449 ms oleg256-client.virtnet: 1 7853 0.32 MB/sec execute 60 sec latency 11436.935 ms oleg256-client.virtnet: 1 7853 0.31 MB/sec execute 61 sec latency 12437.317 ms oleg256-client.virtnet: 1 7853 0.31 MB/sec execute 62 sec latency 13437.475 ms oleg256-clientFailover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3840 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3714048 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7464960 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server reboot facets: mds1 Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 .virtnet: 1 7853 0.30 MB/sec execute 63 sec latency 14437.810 ms oleg256-client.virtnet: 1 7853 0.30 MB/sec execute 64 sec latency 15438.586 ms oleg256-client.virtnet: 1 7853 0.29 MB/sec execute 65 sec latency 16440.259 ms oleg256-client.virtnet: 1 7853 0.29 MB/sec execute 66 sec latency 17440.467 ms oleg256-client.virtnet: 1 7853 0.29 MB/sec execute 67 sec latency 18441.995 ms oleg256-client.virtnet: 1 7853 0.28 MB/sec execute 68 sec latency 19442.647 ms oleg256-client.virtnet: 1 7853 0.28 MB/sec execute 69 sec latency 20442.834 ms oleg256-client.virtnet: 1 7853 0.27 MB/sec execute 70 sec latency 21442.995 ms oleg256-client.virtnet: 1 7853 0.27 MB/sec execute 71 sec latency 22443.266 ms oleg256-client.virtnet: 1 7853 0.27 MB/sec execute 72 sec latency 23443.740 ms oleg256-client.virtnet: 1 7853 0.26 MB/sec execute 73 sec latency 24444.301 ms oleg256-client.virtnet: 1 7853 0.26 MB/sec execute 74 sec latency 25444.492 ms oleg256-client.virtnet: 1 7878 0.26 MB/sec execute 75 sec latency 26327.467 ms oleg256-client.virtnet: 1 7985 0.26 MB/sec execute 76 sec latency 121.413 ms oleg256-client.virtnet: 1 8097 0.26 MB/sec execute 77 sec latency 145.427 ms oleg256-client.virtnet: 1 8308 0.27 MB/sec execute 78 sec latency 33.301 ms oleg256-client.virtnet: 1 8590 0.30 MB/sec execute 79 sec latency 37.308 ms oleg256-client.virtnet: 1 8727 0.30 MB/sec execute 80 sec latency 113.569 ms oleg256-client.virtnet: 1 8913 0.30 MB/sec execute 81 sec latency 104.807 ms oleg256-client.virtnet: 1 9109 0.30 MB/sec execute 82 sec latency 103.461 ms oleg256-client.virtnet: 1 9315 0.30 MB/sec execute 83 sec latency 29.093 ms oleg256-client.virtnet: 1 9532 0.31 MB/sec execute 84 sec latency 95.744 ms oleg256-client.virtnet: 1 9710 0.32 MB/sec execute 85 sec latency 117.276 ms oleg256-client.virtnet: 1 9862 0.35 MB/sec execute 86 sec latency 809.672 ms oleg256-client.virtnet: 1 9862 0.34 MB/sec execute 87 sec latency 1811.203 ms oleg256-client.virtnet: 1 9862 0.34 MB/sec execute 88 sec latency 2812.195 ms oleg256-client.virtnet: 1 9862 0.33 MB/sec execute 89 sec latency 3812.509 ms oleg256-client.virtnet: 1 9862 0.33 MB/sec execute 90 sec latency 4812.732 ms oleg256-client.virtnet: 1 9862 0.33 MB/sec execute 91 sec latency 5813.955 ms oleg256-client.virtnet: 1 9862 0.32 MB/sec execute 92 sec latency 6814.228 ms oleg256-client.virtnet: 1 9862 0.32 MB/sec execute 93 sec latency 7815.224 ms oleg256-client.virtnet: 1 9862 0.32 MB/sec execute 94 sec latency 8817.281 ms oleg256-client.virtnet: 1 9862 0.31 MB/sec execute 95 sec latency 9817.524 ms oleg256-client.virtnet: 1 9862 0.31 MB/sec execute 96 sec latency 10818.156 ms oleg256-client.virtnet: 1 9862 0.31 MB/sec execute 97 sec latency 11819.731 ms oleg256-client.virtnet: 1 9862 0.30 MB/sec execute 98 sec latency 12820.324 ms oleg256-client.virtnet: 1 9862 0.30 MB/sec execute 99 sec latency 13820.514 ms oleg256-client.virtnet: 1 9862 0.30 MB/sec execute 100 sec latency 14820.847 ms oleg256-client.virtnet: 1 9862 0.29 MB/sec execute 101 sec latency 15822.801 ms oleg256-client.virtnet: 1 9862 0.29 MB/sec execute 102 sec latency 16823.123 ms oleg256-client.virtnet: 1 9862 0.29 MB/sec execute 103 sec latency 17823.367 ms oleg256-client.virtnet: 1 9862 0.29 MB/sec execute 104 sec latency 18823.656 ms oleg256-client.virtnet: 1 9862 0.28 MB/sec execute 105 sec latency 19824.054 ms oleg256-client.virtnet: 1 9862 0.28 MB/sec execute 106 sec latency 20825.405 ms oleg256-coleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3840 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3723264 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7477248 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server reboot facets: mds1 Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 lient.virtnet: 1 9862 0.28 MB/sec execute 107 sec latency 21828.873 ms oleg256-client.virtnet: 1 9862 0.28 MB/sec execute 108 sec latency 22831.106 ms oleg256-client.virtnet: 1 9862 0.27 MB/sec execute 109 sec latency 23831.461 ms oleg256-client.virtnet: 1 9862 0.27 MB/sec execute 110 sec latency 24831.809 ms oleg256-client.virtnet: 1 9862 0.27 MB/sec execute 111 sec latency 25832.463 ms oleg256-client.virtnet: 1 9862 0.27 MB/sec execute 112 sec latency 26833.037 ms oleg256-client.virtnet: 1 9862 0.26 MB/sec execute 113 sec latency 27833.569 ms oleg256-client.virtnet: 1 9862 0.26 MB/sec execute 114 sec latency 28833.748 ms oleg256-client.virtnet: 1 9862 0.26 MB/sec execute 115 sec latency 29834.694 ms oleg256-client.virtnet: 1 9862 0.26 MB/sec execute 116 sec latency 30834.860 ms oleg256-client.virtnet: 1 9868 0.25 MB/sec execute 117 sec latency 31754.711 ms oleg256-client.virtnet: 1 10111 0.27 MB/sec execute 118 sec latency 98.952 ms oleg256-client.virtnet: 1 10463 0.28 MB/sec execute 119 sec latency 80.615 ms oleg256-client.virtnet: 1 10735 0.30 MB/sec execute 120 sec latency 56.103 ms oleg256-client.virtnet: 1 10926 0.31 MB/sec execute 121 sec latency 47.205 ms oleg256-client.virtnet: 1 11061 0.31 MB/sec execute 122 sec latency 38.400 ms oleg256-client.virtnet: 1 11206 0.31 MB/sec execute 123 sec latency 42.263 ms oleg256-client.virtnet: 1 11337 0.31 MB/sec execute 124 sec latency 146.845 ms oleg256-client.virtnet: 1 11451 0.31 MB/sec execute 125 sec latency 120.405 ms oleg256-client.virtnet: 1 11535 0.31 MB/sec execute 126 sec latency 131.783 ms oleg256-client.virtnet: 1 11638 0.31 MB/sec execute 127 sec latency 202.318 ms oleg256-client.virtnet: 1 11638 0.30 MB/sec execute 128 sec latency 1147.873 ms oleg256-client.virtnet: 1 11638 0.30 MB/sec execute 129 sec latency 2148.103 ms oleg256-client.virtnet: 1 11638 0.30 MB/sec execute 130 sec latency 3148.513 ms oleg256-client.virtnet: 1 11638 0.30 MB/sec execute 131 sec latency 4148.687 ms oleg256-client.virtnet: 1 11638 0.29 MB/sec execute 132 sec latency 5148.986 ms oleg256-client.virtnet: 1 11638 0.29 MB/sec execute 133 sec latency 6149.852 ms oleg256-client.virtnet: 1 11638 0.29 MB/sec execute 134 sec latency 7150.189 ms oleg256-client.virtnet: 1 11638 0.29 MB/sec execute 135 sec latency 8150.512 ms oleg256-client.virtnet: 1 11638 0.29 MB/sec execute 136 sec latency 9150.741 ms oleg256-client.virtnet: 1 11638 0.28 MB/sec execute 137 sec latency 10151.040 ms oleg256-client.virtnet: 1 11638 0.28 MB/sec execute 138 sec latency 11151.227 ms oleg256-client.virtnet: 1 11638 0.28 MB/sec execute 139 sec latency 12151.455 ms oleg256-client.virtnet: 1 11638 0.28 MB/sec execute 140 sec latency 13152.890 ms oleg256-client.virtnet: 1 11638 0.28 MB/sec execute 141 sec latency 14153.165 ms oleg256-client.virtnet: 1 11638 0.27 MB/sec execute 142 sec latency 15154.088 ms oleg256-client.virtnet: 1 11638 0.27 MB/sec execute 143 sec latency 16154.327 ms oleg256-client.virtnet: 1 11638 0.27 MB/sec execute 144 sec latency 17154.511 ms oleg256-client.virtnet: 1 11638 0.27 MB/sec execute 145 sec latency 18154.708 ms oleg256-client.virtnet: 1 11638 0.27 MB/sec execute 146 sec latency 19154.947 ms oleg256-client.virtnet: 1 11638 0.26 MB/sec execute 147 sec latency 20155.142 ms oleg256-client.virtnet: 1 11638 0.26 MB/sec execute 148 sec latency 21155.855 ms oleg256-client.virtnet: 1 11638 0.26 MB/sec execute 149 sec latency 22156.076 ms oleg256-client.virtnet: 1 11638 0.26 MB/sec execute 150 sec latency 23156.306 ms olegoleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3840 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3712000 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3740672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7452672 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server reboot facets: mds1 Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 256-client.virtnet: 1 11638 0.26 MB/sec execute 151 sec latency 24156.479 ms oleg256-client.virtnet: 1 11638 0.26 MB/sec execute 152 sec latency 25160.307 ms oleg256-client.virtnet: 1 11638 0.25 MB/sec execute 153 sec latency 26160.477 ms oleg256-client.virtnet: 1 11638 0.25 MB/sec execute 154 sec latency 27160.833 ms oleg256-client.virtnet: 1 11638 0.25 MB/sec execute 155 sec latency 28161.363 ms oleg256-client.virtnet: 1 11638 0.25 MB/sec execute 156 sec latency 29161.841 ms oleg256-client.virtnet: 1 11638 0.25 MB/sec execute 157 sec latency 30163.907 ms oleg256-client.virtnet: 1 11638 0.25 MB/sec execute 158 sec latency 31164.079 ms oleg256-client.virtnet: 1 11652 0.24 MB/sec execute 159 sec latency 31949.286 ms oleg256-client.virtnet: 1 11881 0.25 MB/sec execute 160 sec latency 29.167 ms oleg256-client.virtnet: 1 12137 0.27 MB/sec execute 161 sec latency 44.070 ms oleg256-client.virtnet: 1 12274 0.27 MB/sec execute 162 sec latency 115.441 ms oleg256-client.virtnet: 1 12527 0.27 MB/sec execute 163 sec latency 74.837 ms oleg256-client.virtnet: 1 12807 0.27 MB/sec execute 164 sec latency 87.124 ms oleg256-client.virtnet: 1 13021 0.27 MB/sec execute 165 sec latency 132.957 ms oleg256-client.virtnet: 1 13120 0.27 MB/sec execute 166 sec latency 126.660 ms oleg256-client.virtnet: 1 13419 0.29 MB/sec execute 167 sec latency 133.053 ms oleg256-client.virtnet: 1 13662 0.30 MB/sec execute 168 sec latency 113.486 ms oleg256-client.virtnet: 1 14000 0.31 MB/sec execute 169 sec latency 108.483 ms oleg256-client.virtnet: 1 14002 0.31 MB/sec execute 170 sec latency 985.301 ms oleg256-client.virtnet: 1 14002 0.31 MB/sec execute 171 sec latency 1985.723 ms oleg256-client.virtnet: 1 14002 0.30 MB/sec execute 172 sec latency 2985.891 ms oleg256-client.virtnet: 1 14002 0.30 MB/sec execute 173 sec latency 3986.186 ms oleg256-client.virtnet: 1 14002 0.30 MB/sec execute 174 sec latency 4986.863 ms oleg256-client.virtnet: 1 14002 0.30 MB/sec execute 175 sec latency 5987.046 ms oleg256-client.virtnet: 1 14002 0.30 MB/sec execute 176 sec latency 6987.278 ms oleg256-client.virtnet: 1 14002 0.30 MB/sec execute 177 sec latency 7987.441 ms oleg256-client.virtnet: 1 14002 0.29 MB/sec execute 178 sec latency 8987.611 ms oleg256-client.virtnet: 1 14002 0.29 MB/sec execute 179 sec latency 9988.888 ms oleg256-client.virtnet: 1 14002 0.29 MB/sec execute 180 sec latency 10989.112 ms oleg256-client.virtnet: 1 14002 0.29 MB/sec execute 181 sec latency 11991.070 ms oleg256-client.virtnet: 1 14002 0.29 MB/sec execute 182 sec latency 12991.235 ms oleg256-client.virtnet: 1 14002 0.29 MB/sec execute 183 sec latency 13993.051 ms oleg256-client.virtnet: 1 14002 0.28 MB/sec execute 184 sec latency 14993.442 ms oleg256-client.virtnet: 1 14002 0.28 MB/sec execute 185 sec latency 15993.860 ms oleg256-client.virtnet: 1 14002 0.28 MB/sec execute 186 sec latency 16994.060 ms oleg256-client.virtnet: 1 14002 0.28 MB/sec execute 187 sec latency 17995.212 ms oleg256-client.virtnet: 1 14002 0.28 MB/sec execute 188 sec latency 18995.381 ms oleg256-client.virtnet: 1 14002 0.28 MB/sec execute 189 sec latency 19995.565 ms oleg256-client.virtnet: 1 14002 0.28 MB/sec execute 190 sec latency 20996.588 ms oleg256-client.virtnet: 1 14002 0.27 MB/sec execute 191 sec latency 21996.787 ms oleg256-client.virtnet: 1 14002 0.27 MB/sec execute 192 sec latency 22997.075 ms oleg256-client.virtnet: 1 14002 0.27 MB/sec execute 193 sec latency 23998.170 ms oleg256-client.virtnet: 1 14002 0.27 MB/sec execute 194 sec latency 24999.217 msoleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 44032 3720192 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7471104 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server reboot facets: mds1 Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3730432 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7443456 1% /mnt/lustre test_70b fail mds1 9 times oleg256-client.virtnet: 1 14002 0.27 MB/sec execute 195 sec latency 25999.741 ms oleg256-client.virtnet: 1 14002 0.27 MB/sec execute 196 sec latency 27002.096 ms oleg256-client.virtnet: 1 14002 0.27 MB/sec execute 197 sec latency 28002.260 ms oleg256-client.virtnet: 1 14002 0.26 MB/sec execute 198 sec latency 29002.523 ms oleg256-client.virtnet: 1 14002 0.26 MB/sec execute 199 sec latency 30002.840 ms oleg256-client.virtnet: 1 14002 0.26 MB/sec execute 200 sec latency 31003.608 ms oleg256-client.virtnet: 1 14010 0.26 MB/sec execute 201 sec latency 31958.823 ms oleg256-client.virtnet: 1 14314 0.28 MB/sec execute 202 sec latency 26.647 ms oleg256-client.virtnet: 1 14491 0.28 MB/sec execute 203 sec latency 31.005 ms oleg256-client.virtnet: 1 14626 0.28 MB/sec execute 204 sec latency 47.397 ms oleg256-client.virtnet: 1 14824 0.28 MB/sec execute 205 sec latency 24.431 ms oleg256-client.virtnet: 1 14968 0.28 MB/sec execute 206 sec latency 100.279 ms oleg256-client.virtnet: 1 15110 0.28 MB/sec execute 207 sec latency 81.240 ms oleg256-client.virtnet: 1 15336 0.28 MB/sec execute 208 sec latency 70.395 ms oleg256-client.virtnet: 1 15622 0.30 MB/sec execute 209 sec latency 35.809 ms oleg256-client.virtnet: 1 15781 0.30 MB/sec execute 210 sec latency 81.702 ms oleg256-client.virtnet: 1 15820 0.29 MB/sec execute 211 sec latency 817.936 ms oleg256-client.virtnet: 1 15820 0.29 MB/sec execute 212 sec latency 1818.164 ms oleg256-client.virtnet: 1 15820 0.29 MB/sec execute 213 sec latency 2818.578 ms oleg256-client.virtnet: 1 15820 0.29 MB/sec execute 214 sec latency 3819.461 ms oleg256-client.virtnet: 1 15820 0.29 MB/sec execute 215 sec latency 4819.622 ms oleg256-client.virtnet: 1 15820 0.29 MB/sec execute 216 sec latency 5820.945 ms oleg256-client.virtnet: 1 15820 0.29 MB/sec execute 217 sec latency 6821.104 ms oleg256-client.virtnet: 1 15820 0.29 MB/sec execute 218 sec latency 7822.634 ms oleg256-client.virtnet: 1 15820 0.28 MB/sec execute 219 sec latency 8822.852 ms oleg256-client.virtnet: 1 15820 0.28 MB/sec execute 220 sec latency 9825.073 ms oleg256-client.virtnet: 1 15820 0.28 MB/sec execute 221 sec latency 10825.393 ms oleg256-client.virtnet: 1 15820 0.28 MB/sec execute 222 sec latency 11829.240 ms oleg256-client.virtnet: 1 15820 0.28 MB/sec execute 223 sec latency 12829.465 ms oleg256-client.virtnet: 1 15820 0.28 MB/sec execute 224 sec latency 13830.479 ms oleg256-client.virtnet: 1 15820 0.28 MB/sec execute 225 sec latency 14832.019 ms oleg256-client.virtnet: 1 15820 0.28 MB/sec execute 226 sec latency 15832.432 ms oleg256-client.virtnet: 1 15833 0.27 MB/sec execute 227 sec latency 16741.372 ms oleg256-client.virtnet: 1 16033 0.27 MB/sec execute 228 sec latency 63.785 ms oleg256-client.virtnet: 1 16240 0.27 MB/sec execute 229 sec latency 168.938 ms oleg256-client.virtnet: 1 16534 0.28 MB/sec execute 230 sec latency 55.355 ms oleg256-client.virtnet: 1 16632 0.28 MB/sec execute 231 sec latency 142.336 ms oleg256-client.virtnet: 1 16796 0.28 MB/sec execute 232 sec latency 115.757 ms oleg256-client.virtnet: 1 17124 0.30 MB/sec execute 233 sec latency 100.240 ms oleg256-client.virtnet: 1 17450 0.30 MB/sec execute 234 sec latency 123.782 ms oleg256-client.virtnet: 1 17775 0.32 MB/sec execute 235 sec latency 52.964 ms oleg256-client.virtnet: 1 17991 0.32 MB/sec execute 236 sec latency 34.733 ms oleg256-client.virtnet: 1 18130 0.32 MB/sec execute 237 sec latency 31.719 ms oleg256-client.virtnet: 1 18268 0.32 MB/sec execute 238 sec latency 28.169 ms oleg256-client.virtnet: Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server reboot facets: mds1 Failover mds1 to oleg256-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 18446 0.32 MB/sec execute 239 sec latency 120.969 ms oleg256-client.virtnet: 1 18535 0.32 MB/sec execute 240 sec latency 132.239 ms oleg256-client.virtnet: 1 18554 0.32 MB/sec execute 241 sec latency 471.426 ms oleg256-client.virtnet: 1 18554 0.32 MB/sec execute 242 sec latency 1471.655 ms oleg256-client.virtnet: 1 18554 0.32 MB/sec execute 243 sec latency 2471.983 ms oleg256-client.virtnet: 1 18554 0.31 MB/sec execute 244 sec latency 3472.574 ms oleg256-client.virtnet: 1 18554 0.31 MB/sec execute 245 sec latency 4472.733 ms oleg256-client.virtnet: 1 18554 0.31 MB/sec execute 246 sec latency 5473.218 ms oleg256-client.virtnet: 1 18554 0.31 MB/sec execute 247 sec latency 6474.311 ms oleg256-client.virtnet: 1 18554 0.31 MB/sec execute 248 sec latency 7475.209 ms oleg256-client.virtnet: 1 18554 0.31 MB/sec execute 249 sec latency 8475.493 ms oleg256-client.virtnet: 1 18554 0.31 MB/sec execute 250 sec latency 9478.309 ms oleg256-client.virtnet: 1 18554 0.31 MB/sec execute 251 sec latency 10481.206 ms oleg256-client.virtnet: 1 18554 0.30 MB/sec execute 252 sec latency 11486.684 ms oleg256-client.virtnet: 1 18554 0.30 MB/sec execute 253 sec latency 12486.850 ms oleg256-client.virtnet: 1 18554 0.30 MB/sec execute 254 sec latency 13486.981 ms oleg256-client.virtnet: 1 18554 0.30 MB/sec execute 255 sec latency 14487.153 ms oleg256-client.virtnet: 1 18554 0.30 MB/sec execute 256 sec latency 15488.491 ms oleg256-client.virtnet: 1 18554 0.30 MB/sec execute 257 sec latency 16488.940 ms oleg256-client.virtnet: 1 18554 0.30 MB/sec execute 258 sec latency 17490.176 ms oleg256-client.virtnet: 1 18554 0.30 MB/sec execute 259 sec latency 18490.467 ms oleg256-client.virtnet: 1 18554 0.29 MB/sec execute 260 sec latency 19492.015 ms oleg256-client.virtnet: 1 18554 0.29 MB/sec execute 261 sec latency 20492.235 ms oleg256-client.virtnet: 1 18554 0.29 MB/sec execute 262 sec latency 21493.102 ms oleg256-client.virtnet: 1 18554 0.29 MB/sec execute 263 sec latency 22493.261 ms oleg256-client.virtnet: 1 18554 0.29 MB/sec execute 264 sec latency 23493.428 ms oleg256-client.virtnet: 1 18554 0.29 MB/sec execute 265 sec latency 24493.659 ms oleg256-client.virtnet: 1 18554 0.29 MB/sec execute 266 sec latency 25493.853 ms oleg256-client.virtnet: 1 18554 0.29 MB/sec execute 267 sec latency 26494.021 ms oleg256-client.virtnet: 1 18554 0.29 MB/sec execute 268 sec latency 27495.258 ms oleg256-client.virtnet: 1 18554 0.28 MB/sec execute 269 sec latency 28496.589 ms oleg256-client.virtnet: 1 18554 0.28 MB/sec execute 270 sec latency 29497.042 ms oleg256-client.virtnet: 1 18554 0.28 MB/sec execute 271 sec latency 30498.157 ms oleg256-client.virtnet: 1 18650 0.28 MB/sec execute 272 sec latency 30803.709 ms oleg256-client.virtnet: 1 18824 0.29 MB/sec execute 273 sec latency 72.516 ms oleg256-client.virtnet: 1 19129 0.30 MB/sec execute 274 sec latency 34.807 ms oleg256-client.virtnet: 1 19318 0.30 MB/sec execute 275 sec latency 113.882 ms oleg256-client.virtnet: 1 19471 0.29 MB/sec execute 276 sec latency 86.175 ms oleg256-client.virtnet: 1 19637 0.29 MB/sec execute 277 sec latency 96.305 ms oleg256-client.virtnet: 1 19902 0.29 MB/sec execute 278 sec latency 65.232 ms oleg256-client.virtnet: 1 20153 0.30 MB/sec execute 279 sec latency 118.566 ms oleg256-client.virtnet: 1 20271 0.30 MB/sec execute 280 sec latency 138.105 ms oleg256-client.virtnet: 1 20702 0.32 MB/sec execute 281 sec latency 90.948 ms oleg256-client.virtnet: 1 21052 0.32 MB/sec execute 282 sec latency 90.863 ms oleg256-client.virtnet: 1 21352 0.33 MB/sec execute 283 sec latency 59.196 ms oleg256-client.virtnet: 1 21570 0.33 MB/sec execute 284 sec latency 38.141 ms oleg256-client.virtnet: 1 21733 0.33 MB/sec execute 285 sec latency 29.481 ms oleg256-client.virtnet: 1 21976 0.33 MB/sec execute 286 sec latency 23.361 ms oleg256-client.virtnet: 1 22069 0.33 MB/sec execute 287 sec latency 119.602 ms oleg256-client.virtnet: 1 22208 0.33 MB/sec execute 288 sec latency 103.101 ms oleg256-client.virtnet: 1 22375 0.34 MB/sec execute 289 sec latency 93.718 ms oleg256-client.virtnet: 1 22606 0.34 MB/sec execute 290 sec latency 23.842 ms oleg256-client.virtnet: 1 22884 0.35 MB/sec execute 291 sec latency 105.005 ms oleg256-client.virtnet: 1 23065 0.34 MB/sec execute 292 sec latency 67.624 ms oleg256-client.virtnet: 1 23278 0.34 MB/sec execute 293 sec latency 78.464 ms oleg256-client.virtnet: 1 23585 0.34 MB/sec execute 294 sec latency 19.526 ms oleg256-client.virtnet: 1 23773 0.35 MB/sec execute 295 sec latency 105.518 ms oleg256-client.virtnet: 1 24107 0.36 MB/sec execute 296 sec latency 84.811 ms oleg256-client.virtnet: 1 24520 0.37 MB/sec execute 297 sec latency 81.515 ms oleg256-client.virtnet: 1 24890 0.38 MB/sec execute 298 sec latency 84.320 ms oleg256-client.virtnet: 1 25126 0.38 MB/sec execute 299 sec latency 28.026 ms oleg256-client.virtnet: 1 cleanup 300 sec oleg256-client.virtnet: 0 cleanup 301 sec oleg256-client.virtnet: oleg256-client.virtnet: Operation Count AvgLat MaxLat oleg256-client.virtnet: ---------------------------------------- oleg256-client.virtnet: NTCreateX 3624 42.584 31958.812 oleg256-client.virtnet: Close 2680 2.988 16.160 oleg256-client.virtnet: Rename 155 17.412 66.948 oleg256-client.virtnet: Unlink 706 4.836 20.345 oleg256-client.virtnet: Qpathinfo 3267 12.955 31754.695 oleg256-client.virtnet: Qfileinfo 579 0.200 4.398 oleg256-client.virtnet: Qfsinfo 600 0.345 13.632 oleg256-client.virtnet: Sfileinfo 296 113.770 30803.697 oleg256-client.virtnet: Find 1265 0.802 24.412 oleg256-client.virtnet: WriteX 1804 1.877 21.970 oleg256-client.virtnet: ReadX 5751 0.018 3.658 oleg256-client.virtnet: LockX 12 3.323 5.768 oleg256-client.virtnet: UnlockX 12 3.452 6.707 oleg256-client.virtnet: Flush 252 201.094 31949.273 oleg256-client.virtnet: oleg256-client.virtnet: Throughput 0.381471 MB/sec 1 clients 1 procs max_latency=31958.823 ms oleg256-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg256-client.virtnet at Wed May 29 18:34:34 EDT 2024 with return code 0 oleg256-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: /mnt/lustre/d70b.replay-single/oleg256-client.virtnet /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg256-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg256-client.virtnet: removed directory: 'clients/client0' oleg256-client.virtnet: removed directory: 'clients' oleg256-client.virtnet: removed 'client.txt' oleg256-client.virtnet: /mnt/lustre/d70b.replay-single/oleg256-client.virtnet oleg256-client.virtnet: dbench successfully finished