== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 09:35:49 (1747402549) Starting client oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre Started clients oleg303-client.virtnet: 192.168.203.103@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg303-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=12267 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3687424 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3716096 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 34816 7403520 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server reboot facets: mds1 Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3685376 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 45056 7424000 1% /mnt/lustre test_70b fail mds1 2 times oleg303-client.virtnet: looking for dbench program oleg303-client.virtnet: /usr/bin/dbench oleg303-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg303-client.virtnet oleg303-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg303-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg303-client.virtnet' oleg303-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg303-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg303-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg303-client.virtnet at Fri May 16 09:35:50 EDT 2025 oleg303-client.virtnet: waiting for dbench pid 12291 oleg303-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg303-client.virtnet: oleg303-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg303-client.virtnet: failed to create barrier semaphore oleg303-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg303-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg303-client.virtnet: releasing clients oleg303-client.virtnet: 1 504 17.47 MB/sec warmup 1 sec latency 17.170 ms oleg303-client.virtnet: 1 845 10.78 MB/sec warmup 2 sec latency 79.158 ms oleg303-client.virtnet: 1 1147 7.66 MB/sec warmup 3 sec latency 46.701 ms oleg303-client.virtnet: 1 1564 6.53 MB/sec warmup 4 sec latency 36.043 ms oleg303-client.virtnet: 1 1954 5.29 MB/sec warmup 5 sec latency 31.343 ms oleg303-client.virtnet: 1 2389 4.66 MB/sec warmup 6 sec latency 403.887 ms oleg303-client.virtnet: 1 2389 3.99 MB/sec warmup 7 sec latency 1404.123 ms oleg303-client.virtnet: 1 2389 3.50 MB/sec warmup 8 sec latency 2404.412 ms oleg303-client.virtnet: 1 2389 3.11 MB/sec warmup 9 sec latency 3404.707 ms oleg303-client.virtnet: 1 2389 2.80 MB/sec warmup 10 sec latency 4405.004 ms oleg303-client.virtnet: 1 2389 2.54 MB/sec warmup 11 sec latency 5405.255 ms oleg303-client.virtnet: 1 2389 2.33 MB/sec warmup 12 sec latency 6405.520 ms oleg303-client.virtnet: 1 2389 2.15 MB/sec warmup 13 sec latency 7405.849 ms oleg303-client.virtnet: 1 2389 2.00 MB/sec warmup 14 sec latency 8406.440 ms oleg303-client.virtnet: 1 2389 1.86 MB/sec warmup 15 sec latency 9406.731 ms oleg303-client.virtnet: 1 2389 1.75 MB/sec warmup 16 sec latency 10407.016 ms oleg303-client.virtnet: 1 2389 1.64 MB/sec warmup 17 sec latency 11407.248 ms oleg303-client.virtnet: 1 2389 1.55 MB/sec warmup 18 sec latency 12407.632 ms oleg303-client.virtnet: 1 2389 1.47 MB/sec warmup 19 sec latency 13407.895 ms oleg303-client.virtnet: 1 2389 1.40 MB/sec warmup 20 sec latency 14408.203 ms oleg303-client.virtnet: 1 2389 1.33 MB/sec warmup 21 sec latency 15408.544 ms oleg303-client.virtnet: 1 2389 1.27 MB/sec warmup 22 sec latency 16408.732 ms oleg303-client.virtnet: 1 2389 1.22 MB/sec warmup 23 sec latency 17408.874 ms oleg303-client.virtnet: 1 2389 1.16 MB/sec warmup 24 sec latency 18409.121 ms oleg303-client.virtnet: 1 2389 1.12 MB/sec warmup 25 sec latency 19409.376 ms oleg303-client.virtnet: 1 2916 1.29 MB/sec warmup 26 sec latency 19498.970 ms oleg303-client.virtnet: 1 3624 1.43 MB/sec warmup 27 sec latency 35.462 ms oleg303-client.virtnet: 1 3969 1.43 MB/sec warmup 28 sec latency 16.550 ms oleg303-client.virtnet: 1 4294 1.39 MB/sec warmup 29 sec latency 31.174 ms oleg303-client.virtnet: 1 4684 1.40 MB/sec warmup 30 sec latency 29.612 ms oleg303-client.virtnet: 1 5335 1.46 MB/sec warmup 31 sec latency 22.854 ms oleg303-client.virtnet: 1 5865 1.46 MB/sec warmup 32 sec latency 24.377Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server reboot facets: mds1 Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3693568 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3739648 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7433216 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server reboot facets: mds1 ms oleg303-client.virtnet: 1 6319 1.55 MB/sec warmup 33 sec latency 270.857 ms oleg303-client.virtnet: 1 6319 1.50 MB/sec warmup 34 sec latency 1271.091 ms oleg303-client.virtnet: 1 6319 1.46 MB/sec warmup 35 sec latency 2271.323 ms oleg303-client.virtnet: 1 6319 1.42 MB/sec warmup 36 sec latency 3271.557 ms oleg303-client.virtnet: 1 6319 1.38 MB/sec warmup 37 sec latency 4271.785 ms oleg303-client.virtnet: 1 6319 1.34 MB/sec warmup 38 sec latency 5272.029 ms oleg303-client.virtnet: 1 6319 1.31 MB/sec warmup 39 sec latency 6272.219 ms oleg303-client.virtnet: 1 6319 1.28 MB/sec warmup 40 sec latency 7272.404 ms oleg303-client.virtnet: 1 6319 1.25 MB/sec warmup 41 sec latency 8272.688 ms oleg303-client.virtnet: 1 6319 1.22 MB/sec warmup 42 sec latency 9272.942 ms oleg303-client.virtnet: 1 6319 1.19 MB/sec warmup 43 sec latency 10273.174 ms oleg303-client.virtnet: 1 6319 1.16 MB/sec warmup 44 sec latency 11273.329 ms oleg303-client.virtnet: 1 6319 1.13 MB/sec warmup 45 sec latency 12273.520 ms oleg303-client.virtnet: 1 6319 1.11 MB/sec warmup 46 sec latency 13273.714 ms oleg303-client.virtnet: 1 6319 1.09 MB/sec warmup 47 sec latency 14273.864 ms oleg303-client.virtnet: 1 6319 1.06 MB/sec warmup 48 sec latency 15274.054 ms oleg303-client.virtnet: 1 6319 1.04 MB/sec warmup 49 sec latency 16274.267 ms oleg303-client.virtnet: 1 6319 1.02 MB/sec warmup 50 sec latency 17274.471 ms oleg303-client.virtnet: 1 6319 1.00 MB/sec warmup 51 sec latency 18274.594 ms oleg303-client.virtnet: 1 6319 0.98 MB/sec warmup 52 sec latency 19274.753 ms oleg303-client.virtnet: 1 6319 0.96 MB/sec warmup 53 sec latency 20274.961 ms oleg303-client.virtnet: 1 6319 0.95 MB/sec warmup 54 sec latency 21275.242 ms oleg303-client.virtnet: 1 6319 0.93 MB/sec warmup 55 sec latency 22275.487 ms oleg303-client.virtnet: 1 6319 0.91 MB/sec warmup 56 sec latency 23275.682 ms oleg303-client.virtnet: 1 6319 0.90 MB/sec warmup 57 sec latency 24275.931 ms oleg303-client.virtnet: 1 6319 0.88 MB/sec warmup 58 sec latency 25276.143 ms oleg303-client.virtnet: 1 6319 0.87 MB/sec warmup 59 sec latency 26276.402 ms oleg303-client.virtnet: 1 6319 0.00 MB/sec execute 1 sec latency 28276.758 ms oleg303-client.virtnet: 1 6319 0.00 MB/sec execute 2 sec latency 29276.862 ms oleg303-client.virtnet: 1 7241 2.26 MB/sec execute 3 sec latency 29310.913 ms oleg303-client.virtnet: 1 7637 2.03 MB/sec execute 4 sec latency 13.709 ms oleg303-client.virtnet: 1 8026 1.79 MB/sec execute 5 sec latency 33.277 ms oleg303-client.virtnet: 1 8487 2.19 MB/sec execute 6 sec latency 29.427 ms oleg303-client.virtnet: 1 8928 1.93 MB/sec execute 7 sec latency 47.728 ms oleg303-client.virtnet: 1 9430 1.87 MB/sec execute 8 sec latency 34.980 ms oleg303-client.virtnet: 1 10122 2.30 MB/sec execute 9 sec latency 28.813 ms oleg303-client.virtnet: 1 10440 2.25 MB/sec execute 10 sec latency 755.944 ms oleg303-client.virtnet: 1 10440 2.05 MB/sec execute 11 sec latency 1756.143 ms oleg303-client.virtnet: 1 10440 1.88 MB/sec execute 12 sec latency 2756.340 ms oleg303-client.virtnet: 1 10440 1.73 MB/sec execute 13 sec latency 3756.537 ms oleg303-client.virtnet: 1 10440 1.61 MB/sec execute 14 sec latency 4756.681 ms oleg303-client.virtnet: 1 10440 1.50 MB/sec execute 15 sec latency 5756.936 ms oleg303-client.virtnet: 1 10440 1.41 MB/sec execute 16 sec latency 6757.089 ms oleg303-client.virtnet: 1 10440 1.33 MB/sec execute 17 sec latency 7757.305 ms oleg303-client.virtneFailover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3715072 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3740672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7455744 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server reboot facets: mds1 Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 t: 1 10440 1.25 MB/sec execute 18 sec latency 8757.517 ms oleg303-client.virtnet: 1 10440 1.19 MB/sec execute 19 sec latency 9757.707 ms oleg303-client.virtnet: 1 10440 1.13 MB/sec execute 20 sec latency 10757.942 ms oleg303-client.virtnet: 1 10440 1.07 MB/sec execute 21 sec latency 11758.176 ms oleg303-client.virtnet: 1 10440 1.02 MB/sec execute 22 sec latency 12758.462 ms oleg303-client.virtnet: 1 10440 0.98 MB/sec execute 23 sec latency 13758.700 ms oleg303-client.virtnet: 1 10440 0.94 MB/sec execute 24 sec latency 14758.978 ms oleg303-client.virtnet: 1 10440 0.90 MB/sec execute 25 sec latency 15759.309 ms oleg303-client.virtnet: 1 10440 0.87 MB/sec execute 26 sec latency 16759.613 ms oleg303-client.virtnet: 1 10440 0.83 MB/sec execute 27 sec latency 17759.895 ms oleg303-client.virtnet: 1 10440 0.81 MB/sec execute 28 sec latency 18760.089 ms oleg303-client.virtnet: 1 10440 0.78 MB/sec execute 29 sec latency 19760.199 ms oleg303-client.virtnet: 1 10440 0.75 MB/sec execute 30 sec latency 20760.406 ms oleg303-client.virtnet: 1 10440 0.73 MB/sec execute 31 sec latency 21760.719 ms oleg303-client.virtnet: 1 10440 0.70 MB/sec execute 32 sec latency 22761.034 ms oleg303-client.virtnet: 1 10440 0.68 MB/sec execute 33 sec latency 23761.356 ms oleg303-client.virtnet: 1 10440 0.66 MB/sec execute 34 sec latency 24761.657 ms oleg303-client.virtnet: 1 10440 0.64 MB/sec execute 35 sec latency 25761.989 ms oleg303-client.virtnet: 1 10440 0.63 MB/sec execute 36 sec latency 26762.245 ms oleg303-client.virtnet: 1 10440 0.61 MB/sec execute 37 sec latency 27762.628 ms oleg303-client.virtnet: 1 10440 0.59 MB/sec execute 38 sec latency 28762.929 ms oleg303-client.virtnet: 1 10440 0.58 MB/sec execute 39 sec latency 29763.103 ms oleg303-client.virtnet: 1 11029 0.68 MB/sec execute 40 sec latency 29875.165 ms oleg303-client.virtnet: 1 11506 0.68 MB/sec execute 41 sec latency 34.133 ms oleg303-client.virtnet: 1 11918 0.70 MB/sec execute 42 sec latency 30.796 ms oleg303-client.virtnet: 1 12341 0.76 MB/sec execute 43 sec latency 36.045 ms oleg303-client.virtnet: 1 12777 0.75 MB/sec execute 44 sec latency 39.103 ms oleg303-client.virtnet: 1 13107 0.76 MB/sec execute 45 sec latency 51.727 ms oleg303-client.virtnet: 1 13847 0.88 MB/sec execute 46 sec latency 38.322 ms oleg303-client.virtnet: 1 14282 0.96 MB/sec execute 47 sec latency 487.823 ms oleg303-client.virtnet: 1 14282 0.94 MB/sec execute 48 sec latency 1488.072 ms oleg303-client.virtnet: 1 14282 0.92 MB/sec execute 49 sec latency 2488.255 ms oleg303-client.virtnet: 1 14282 0.90 MB/sec execute 50 sec latency 3488.435 ms oleg303-client.virtnet: 1 14282 0.88 MB/sec execute 51 sec latency 4488.624 ms oleg303-client.virtnet: 1 14282 0.86 MB/sec execute 52 sec latency 5488.870 ms oleg303-client.virtnet: 1 14282 0.85 MB/sec execute 53 sec latency 6489.201 ms oleg303-client.virtnet: 1 14282 0.83 MB/sec execute 54 sec latency 7489.436 ms oleg303-client.virtnet: 1 14282 0.82 MB/sec execute 55 sec latency 8489.651 ms oleg303-client.virtnet: 1 14282 0.80 MB/sec execute 56 sec latency 9489.916 ms oleg303-client.virtnet: 1 14282 0.79 MB/sec execute 57 sec latency 10490.211 ms oleg303-client.virtnet: 1 14282 0.77 MB/sec execute 58 sec latency 11490.467 ms oleg303-client.virtnet: 1 14282 0.76 MB/sec execute 59 sec latency 12490.635 ms oleg303-client.virtnet: 1 14282 0.75 MB/sec execute 60 sec latency 13490.812 ms oleg303-client.virtnet: 1 14282 0.74 MB/sec execute 61 sec latency 14490.974 ms oleg303-clieoleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 44032 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7458816 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server reboot facets: mds1 Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 nt.virtnet: 1 14282 0.72 MB/sec execute 62 sec latency 15491.116 ms oleg303-client.virtnet: 1 14282 0.71 MB/sec execute 63 sec latency 16491.269 ms oleg303-client.virtnet: 1 14282 0.70 MB/sec execute 64 sec latency 17491.498 ms oleg303-client.virtnet: 1 14282 0.69 MB/sec execute 65 sec latency 18491.708 ms oleg303-client.virtnet: 1 14282 0.68 MB/sec execute 66 sec latency 19492.009 ms oleg303-client.virtnet: 1 14282 0.67 MB/sec execute 67 sec latency 20492.318 ms oleg303-client.virtnet: 1 14282 0.66 MB/sec execute 68 sec latency 21492.592 ms oleg303-client.virtnet: 1 14282 0.65 MB/sec execute 69 sec latency 22492.904 ms oleg303-client.virtnet: 1 14282 0.64 MB/sec execute 70 sec latency 23493.258 ms oleg303-client.virtnet: 1 14282 0.63 MB/sec execute 71 sec latency 24493.522 ms oleg303-client.virtnet: 1 14282 0.62 MB/sec execute 72 sec latency 25493.900 ms oleg303-client.virtnet: 1 14282 0.62 MB/sec execute 73 sec latency 26494.272 ms oleg303-client.virtnet: 1 14282 0.61 MB/sec execute 74 sec latency 27494.573 ms oleg303-client.virtnet: 1 14282 0.60 MB/sec execute 75 sec latency 28494.905 ms oleg303-client.virtnet: 1 14282 0.59 MB/sec execute 76 sec latency 29495.168 ms oleg303-client.virtnet: 1 14618 0.60 MB/sec execute 77 sec latency 29700.747 ms oleg303-client.virtnet: 1 15104 0.60 MB/sec execute 78 sec latency 41.401 ms oleg303-client.virtnet: 1 15467 0.61 MB/sec execute 79 sec latency 29.767 ms oleg303-client.virtnet: 1 15885 0.65 MB/sec execute 80 sec latency 37.898 ms oleg303-client.virtnet: 1 16569 0.66 MB/sec execute 81 sec latency 25.998 ms oleg303-client.virtnet: 1 17127 0.72 MB/sec execute 82 sec latency 42.673 ms oleg303-client.virtnet: 1 17943 0.79 MB/sec execute 83 sec latency 21.334 ms oleg303-client.virtnet: 1 18043 0.78 MB/sec execute 84 sec latency 583.899 ms oleg303-client.virtnet: 1 18043 0.77 MB/sec execute 85 sec latency 1584.204 ms oleg303-client.virtnet: 1 18043 0.76 MB/sec execute 86 sec latency 2584.524 ms oleg303-client.virtnet: 1 18043 0.75 MB/sec execute 87 sec latency 3584.796 ms oleg303-client.virtnet: 1 18043 0.74 MB/sec execute 88 sec latency 4585.041 ms oleg303-client.virtnet: 1 18043 0.73 MB/sec execute 89 sec latency 5585.280 ms oleg303-client.virtnet: 1 18043 0.72 MB/sec execute 90 sec latency 6585.608 ms oleg303-client.virtnet: 1 18043 0.72 MB/sec execute 91 sec latency 7585.830 ms oleg303-client.virtnet: 1 18043 0.71 MB/sec execute 92 sec latency 8586.035 ms oleg303-client.virtnet: 1 18043 0.70 MB/sec execute 93 sec latency 9586.228 ms oleg303-client.virtnet: 1 18043 0.69 MB/sec execute 94 sec latency 10586.507 ms oleg303-client.virtnet: 1 18043 0.69 MB/sec execute 95 sec latency 11586.711 ms oleg303-client.virtnet: 1 18043 0.68 MB/sec execute 96 sec latency 12586.916 ms oleg303-client.virtnet: 1 18043 0.67 MB/sec execute 97 sec latency 13587.131 ms oleg303-client.virtnet: 1 18043 0.66 MB/sec execute 98 sec latency 14587.325 ms oleg303-client.virtnet: 1 18043 0.66 MB/sec execute 99 sec latency 15587.519 ms oleg303-client.virtnet: 1 18043 0.65 MB/sec execute 100 sec latency 16587.709 ms oleg303-client.virtnet: 1 18043 0.65 MB/sec execute 101 sec latency 17587.891 ms oleg303-client.virtnet: 1 18043 0.64 MB/sec execute 102 sec latency 18588.042 ms oleg303-client.virtnet: 1 18043 0.63 MB/sec execute 103 sec latency 19588.273 ms oleg303-client.virtnet: 1 18043 0.63 MB/sec execute 104 sec latency 20588.441 ms oleg303-client.virtnet: 1 18043 0.62 MB/sec execute 105 sec latency 21588.678 ms ooleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3724288 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3747840 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7472128 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server reboot facets: mds1 Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 leg303-client.virtnet: 1 18043 0.61 MB/sec execute 106 sec latency 22588.961 ms oleg303-client.virtnet: 1 18043 0.61 MB/sec execute 107 sec latency 23589.141 ms oleg303-client.virtnet: 1 18043 0.60 MB/sec execute 108 sec latency 24589.413 ms oleg303-client.virtnet: 1 18043 0.60 MB/sec execute 109 sec latency 25589.622 ms oleg303-client.virtnet: 1 18043 0.59 MB/sec execute 110 sec latency 26589.873 ms oleg303-client.virtnet: 1 18043 0.59 MB/sec execute 111 sec latency 27590.113 ms oleg303-client.virtnet: 1 18043 0.58 MB/sec execute 112 sec latency 28590.390 ms oleg303-client.virtnet: 1 18043 0.58 MB/sec execute 113 sec latency 29590.549 ms oleg303-client.virtnet: 1 18368 0.57 MB/sec execute 114 sec latency 29960.608 ms oleg303-client.virtnet: 1 18733 0.58 MB/sec execute 115 sec latency 38.302 ms oleg303-client.virtnet: 1 19173 0.61 MB/sec execute 116 sec latency 31.693 ms oleg303-client.virtnet: 1 19575 0.61 MB/sec execute 117 sec latency 46.082 ms oleg303-client.virtnet: 1 20161 0.61 MB/sec execute 118 sec latency 26.081 ms oleg303-client.virtnet: 1 20792 0.66 MB/sec execute 119 sec latency 30.543 ms oleg303-client.virtnet: 1 21585 0.70 MB/sec execute 120 sec latency 20.545 ms oleg303-client.virtnet: 1 21837 0.70 MB/sec execute 121 sec latency 478.665 ms oleg303-client.virtnet: 1 21837 0.69 MB/sec execute 122 sec latency 1478.931 ms oleg303-client.virtnet: 1 21837 0.69 MB/sec execute 123 sec latency 2479.135 ms oleg303-client.virtnet: 1 21837 0.68 MB/sec execute 124 sec latency 3479.356 ms oleg303-client.virtnet: 1 21837 0.68 MB/sec execute 125 sec latency 4479.549 ms oleg303-client.virtnet: 1 21837 0.67 MB/sec execute 126 sec latency 5479.762 ms oleg303-client.virtnet: 1 21837 0.67 MB/sec execute 127 sec latency 6479.999 ms oleg303-client.virtnet: 1 21837 0.66 MB/sec execute 128 sec latency 7480.199 ms oleg303-client.virtnet: 1 21837 0.66 MB/sec execute 129 sec latency 8480.440 ms oleg303-client.virtnet: 1 21837 0.65 MB/sec execute 130 sec latency 9480.663 ms oleg303-client.virtnet: 1 21837 0.65 MB/sec execute 131 sec latency 10480.898 ms oleg303-client.virtnet: 1 21837 0.64 MB/sec execute 132 sec latency 11481.182 ms oleg303-client.virtnet: 1 21837 0.64 MB/sec execute 133 sec latency 12481.395 ms oleg303-client.virtnet: 1 21837 0.63 MB/sec execute 134 sec latency 13481.549 ms oleg303-client.virtnet: 1 21837 0.63 MB/sec execute 135 sec latency 14481.760 ms oleg303-client.virtnet: 1 21837 0.62 MB/sec execute 136 sec latency 15481.954 ms oleg303-client.virtnet: 1 21837 0.62 MB/sec execute 137 sec latency 16482.243 ms oleg303-client.virtnet: 1 21837 0.61 MB/sec execute 138 sec latency 17482.448 ms oleg303-client.virtnet: 1 21837 0.61 MB/sec execute 139 sec latency 18482.610 ms oleg303-client.virtnet: 1 21837 0.60 MB/sec execute 140 sec latency 19482.906 ms oleg303-client.virtnet: 1 21837 0.60 MB/sec execute 141 sec latency 20483.167 ms oleg303-client.virtnet: 1 21837 0.60 MB/sec execute 142 sec latency 21483.347 ms oleg303-client.virtnet: 1 21837 0.59 MB/sec execute 143 sec latency 22483.579 ms oleg303-client.virtnet: 1 21837 0.59 MB/sec execute 144 sec latency 23483.817 ms oleg303-client.virtnet: 1 21837 0.58 MB/sec execute 145 sec latency 24484.077 ms oleg303-client.virtnet: 1 21837 0.58 MB/sec execute 146 sec latency 25484.317 ms oleg303-client.virtnet: 1 21837 0.58 MB/sec execute 147 sec latency 26484.572 ms oleg303-client.virtnet: 1 21837 0.57 MB/sec execute 148 sec latency 27484.760 ms oleg303-client.virtnet: 1 21837 0.57 MB/sec execute 149 sec latency 284oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7448576 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server reboot facets: mds1 Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3714048 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3729408 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7443456 1% /mnt/lustre 84.912 ms oleg303-client.virtnet: 1 21837 0.56 MB/sec execute 150 sec latency 29485.099 ms oleg303-client.virtnet: 1 22185 0.56 MB/sec execute 151 sec latency 29803.059 ms oleg303-client.virtnet: 1 22843 0.59 MB/sec execute 152 sec latency 27.937 ms oleg303-client.virtnet: 1 23590 0.59 MB/sec execute 153 sec latency 22.100 ms oleg303-client.virtnet: 1 24535 0.64 MB/sec execute 154 sec latency 20.629 ms oleg303-client.virtnet: 1 25277 0.67 MB/sec execute 155 sec latency 10.910 ms oleg303-client.virtnet: 1 25818 0.67 MB/sec execute 156 sec latency 27.123 ms oleg303-client.virtnet: 1 26540 0.69 MB/sec execute 157 sec latency 23.323 ms oleg303-client.virtnet: 1 26549 0.69 MB/sec execute 158 sec latency 988.178 ms oleg303-client.virtnet: 1 26549 0.69 MB/sec execute 159 sec latency 1988.393 ms oleg303-client.virtnet: 1 26549 0.68 MB/sec execute 160 sec latency 2988.595 ms oleg303-client.virtnet: 1 26549 0.68 MB/sec execute 161 sec latency 3988.912 ms oleg303-client.virtnet: 1 26549 0.67 MB/sec execute 162 sec latency 4989.141 ms oleg303-client.virtnet: 1 26549 0.67 MB/sec execute 163 sec latency 5989.349 ms oleg303-client.virtnet: 1 26549 0.66 MB/sec execute 164 sec latency 6989.508 ms oleg303-client.virtnet: 1 26549 0.66 MB/sec execute 165 sec latency 7989.679 ms oleg303-client.virtnet: 1 26549 0.66 MB/sec execute 166 sec latency 8989.887 ms oleg303-client.virtnet: 1 26549 0.65 MB/sec execute 167 sec latency 9990.094 ms oleg303-client.virtnet: 1 26549 0.65 MB/sec execute 168 sec latency 10990.291 ms oleg303-client.virtnet: 1 26549 0.65 MB/sec execute 169 sec latency 11990.462 ms oleg303-client.virtnet: 1 26549 0.64 MB/sec execute 170 sec latency 12990.597 ms oleg303-client.virtnet: 1 26549 0.64 MB/sec execute 171 sec latency 13990.711 ms oleg303-client.virtnet: 1 26549 0.63 MB/sec execute 172 sec latency 14990.837 ms oleg303-client.virtnet: 1 26549 0.63 MB/sec execute 173 sec latency 15991.018 ms oleg303-client.virtnet: 1 26549 0.63 MB/sec execute 174 sec latency 16991.199 ms oleg303-client.virtnet: 1 26549 0.62 MB/sec execute 175 sec latency 17991.386 ms oleg303-client.virtnet: 1 26549 0.62 MB/sec execute 176 sec latency 18991.512 ms oleg303-client.virtnet: 1 26549 0.62 MB/sec execute 177 sec latency 19991.681 ms oleg303-client.virtnet: 1 26549 0.61 MB/sec execute 178 sec latency 20991.893 ms oleg303-client.virtnet: 1 26549 0.61 MB/sec execute 179 sec latency 21992.073 ms oleg303-client.virtnet: 1 26549 0.61 MB/sec execute 180 sec latency 22992.231 ms oleg303-client.virtnet: 1 26549 0.60 MB/sec execute 181 sec latency 23992.392 ms oleg303-client.virtnet: 1 26549 0.60 MB/sec execute 182 sec latency 24992.555 ms oleg303-client.virtnet: 1 26549 0.60 MB/sec execute 183 sec latency 25992.697 ms oleg303-client.virtnet: 1 26549 0.59 MB/sec execute 184 sec latency 26992.842 ms oleg303-client.virtnet: 1 26549 0.59 MB/sec execute 185 sec latency 27992.978 ms oleg303-client.virtnet: 1 26549 0.59 MB/sec execute 186 sec latency 28993.098 ms oleg303-client.virtnet: 1 26549 0.58 MB/sec execute 187 sec latency 29993.258 ms oleg303-client.virtnet: 1 27112 0.58 MB/sec execute 188 sec latency 30215.331 ms oleg303-client.virtnet: 1 27858 0.62 MB/sec execute 189 sec latency 31.201 ms oleg303-client.virtnet: 1 28667 0.65 MB/sec execute 190 sec latency 30.965 ms oleg303-client.virtnet: 1 29151 0.65 MB/sec execute 191 sec latency 35.512 ms oleg303-client.virtnet: 1 29416 0.64 MB/sec execute 192 sec latency 40.744 ms oleg303-client.virtnet: 1 29905 0.66 MB/sec execute 193 sec latency 15.9test_70b fail mds1 8 times Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server reboot facets: mds1 Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3706880 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7459840 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server reboot facets: mds1 61 ms oleg303-client.virtnet: 1 30666 0.66 MB/sec execute 194 sec latency 18.621 ms oleg303-client.virtnet: 1 30993 0.67 MB/sec execute 195 sec latency 555.270 ms oleg303-client.virtnet: 1 30993 0.67 MB/sec execute 196 sec latency 1555.385 ms oleg303-client.virtnet: 1 30993 0.66 MB/sec execute 197 sec latency 2555.484 ms oleg303-client.virtnet: 1 30993 0.66 MB/sec execute 198 sec latency 3555.682 ms oleg303-client.virtnet: 1 30993 0.66 MB/sec execute 199 sec latency 4555.830 ms oleg303-client.virtnet: 1 30993 0.65 MB/sec execute 200 sec latency 5555.986 ms oleg303-client.virtnet: 1 30993 0.65 MB/sec execute 201 sec latency 6556.163 ms oleg303-client.virtnet: 1 30993 0.65 MB/sec execute 202 sec latency 7556.294 ms oleg303-client.virtnet: 1 30993 0.64 MB/sec execute 203 sec latency 8556.393 ms oleg303-client.virtnet: 1 30993 0.64 MB/sec execute 204 sec latency 9556.497 ms oleg303-client.virtnet: 1 30993 0.64 MB/sec execute 205 sec latency 10556.663 ms oleg303-client.virtnet: 1 30993 0.64 MB/sec execute 206 sec latency 11556.933 ms oleg303-client.virtnet: 1 30993 0.63 MB/sec execute 207 sec latency 12557.155 ms oleg303-client.virtnet: 1 30993 0.63 MB/sec execute 208 sec latency 13557.364 ms oleg303-client.virtnet: 1 30993 0.63 MB/sec execute 209 sec latency 14557.587 ms oleg303-client.virtnet: 1 30993 0.62 MB/sec execute 210 sec latency 15557.875 ms oleg303-client.virtnet: 1 30993 0.62 MB/sec execute 211 sec latency 16558.097 ms oleg303-client.virtnet: 1 30993 0.62 MB/sec execute 212 sec latency 17558.303 ms oleg303-client.virtnet: 1 30993 0.61 MB/sec execute 213 sec latency 18558.516 ms oleg303-client.virtnet: 1 30993 0.61 MB/sec execute 214 sec latency 19558.745 ms oleg303-client.virtnet: 1 30993 0.61 MB/sec execute 215 sec latency 20558.992 ms oleg303-client.virtnet: 1 30993 0.61 MB/sec execute 216 sec latency 21559.296 ms oleg303-client.virtnet: 1 30993 0.60 MB/sec execute 217 sec latency 22559.615 ms oleg303-client.virtnet: 1 30993 0.60 MB/sec execute 218 sec latency 23559.978 ms oleg303-client.virtnet: 1 30993 0.60 MB/sec execute 219 sec latency 24560.208 ms oleg303-client.virtnet: 1 30993 0.59 MB/sec execute 220 sec latency 25560.402 ms oleg303-client.virtnet: 1 30993 0.59 MB/sec execute 221 sec latency 26560.574 ms oleg303-client.virtnet: 1 30993 0.59 MB/sec execute 222 sec latency 27560.798 ms oleg303-client.virtnet: 1 30993 0.59 MB/sec execute 223 sec latency 28561.038 ms oleg303-client.virtnet: 1 30993 0.58 MB/sec execute 224 sec latency 29561.214 ms oleg303-client.virtnet: 1 31604 0.61 MB/sec execute 225 sec latency 29866.207 ms oleg303-client.virtnet: 1 32230 0.63 MB/sec execute 226 sec latency 19.916 ms oleg303-client.virtnet: 1 32631 0.63 MB/sec execute 227 sec latency 14.725 ms oleg303-client.virtnet: 1 33144 0.63 MB/sec execute 228 sec latency 32.876 ms oleg303-client.virtnet: 1 33854 0.64 MB/sec execute 229 sec latency 22.782 ms oleg303-client.virtnet: 1 34731 0.67 MB/sec execute 230 sec latency 20.644 ms oleg303-client.virtnet: 1 35726 0.70 MB/sec execute 231 sec latency 18.480 ms oleg303-client.virtnet: 1 35819 0.69 MB/sec execute 232 sec latency 701.188 ms oleg303-client.virtnet: 1 35819 0.69 MB/sec execute 233 sec latency 1701.593 ms oleg303-client.virtnet: 1 35819 0.69 MB/sec execute 234 sec latency 2701.907 ms oleg303-client.virtnet: 1 35819 0.68 MB/sec execute 235 sec latency 3702.125 ms oleg303-client.virtnet: 1 35819 0.68 MB/sec execute 236 sec latency 4702.280 ms oleg303-client.virtnet: 1 35819 0.68 MB/sec execute 237 sec latency 5702.Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 513 ms oleg303-client.virtnet: 1 35819 0.68 MB/sec execute 238 sec latency 6702.802 ms oleg303-client.virtnet: 1 35819 0.67 MB/sec execute 239 sec latency 7702.969 ms oleg303-client.virtnet: 1 35819 0.67 MB/sec execute 240 sec latency 8703.164 ms oleg303-client.virtnet: 1 35819 0.67 MB/sec execute 241 sec latency 9703.426 ms oleg303-client.virtnet: 1 35819 0.66 MB/sec execute 242 sec latency 10703.602 ms oleg303-client.virtnet: 1 35819 0.66 MB/sec execute 243 sec latency 11703.860 ms oleg303-client.virtnet: 1 35819 0.66 MB/sec execute 244 sec latency 12704.072 ms oleg303-client.virtnet: 1 35819 0.66 MB/sec execute 245 sec latency 13704.286 ms oleg303-client.virtnet: 1 35819 0.65 MB/sec execute 246 sec latency 14704.544 ms oleg303-client.virtnet: 1 35819 0.65 MB/sec execute 247 sec latency 15704.836 ms oleg303-client.virtnet: 1 35819 0.65 MB/sec execute 248 sec latency 16705.086 ms oleg303-client.virtnet: 1 35819 0.65 MB/sec execute 249 sec latency 17705.317 ms oleg303-client.virtnet: 1 35819 0.64 MB/sec execute 250 sec latency 18705.549 ms oleg303-client.virtnet: 1 35819 0.64 MB/sec execute 251 sec latency 19705.784 ms oleg303-client.virtnet: 1 35819 0.64 MB/sec execute 252 sec latency 20706.008 ms oleg303-client.virtnet: 1 35819 0.64 MB/sec execute 253 sec latency 21706.179 ms oleg303-client.virtnet: 1 35819 0.63 MB/sec execute 254 sec latency 22706.333 ms oleg303-client.virtnet: 1 35819 0.63 MB/sec execute 255 sec latency 23706.557 ms oleg303-client.virtnet: 1 35819 0.63 MB/sec execute 256 sec latency 24706.792 ms oleg303-client.virtnet: 1 35819 0.63 MB/sec execute 257 sec latency 25706.991 ms oleg303-client.virtnet: 1 35819 0.62 MB/sec execute 258 sec latency 26707.219 ms oleg303-client.virtnet: 1 35819 0.62 MB/sec execute 259 sec latency 27707.393 ms oleg303-client.virtnet: 1 35819 0.62 MB/sec execute 260 sec latency 28707.694 ms oleg303-client.virtnet: 1 35819 0.62 MB/sec execute 261 sec latency 29707.923 ms oleg303-client.virtnet: 1 36116 0.61 MB/sec execute 262 sec latency 30065.576 ms oleg303-client.virtnet: 1 36573 0.62 MB/sec execute 263 sec latency 32.855 ms oleg303-client.virtnet: 1 37167 0.63 MB/sec execute 264 sec latency 31.973 ms oleg303-client.virtnet: 1 37692 0.63 MB/sec execute 265 sec latency 20.055 ms oleg303-client.virtnet: 1 38111 0.64 MB/sec execute 266 sec latency 43.034 ms oleg303-client.virtnet: 1 39088 0.67 MB/sec execute 267 sec latency 21.066 ms oleg303-client.virtnet: 1 39459 0.67 MB/sec execute 268 sec latency 13.360 ms oleg303-client.virtnet: 1 39931 0.67 MB/sec execute 269 sec latency 33.062 ms oleg303-client.virtnet: 1 40473 0.69 MB/sec execute 270 sec latency 23.034 ms oleg303-client.virtnet: 1 40997 0.69 MB/sec execute 271 sec latency 44.397 ms oleg303-client.virtnet: 1 41928 0.70 MB/sec execute 272 sec latency 21.208 ms oleg303-client.virtnet: 1 42854 0.73 MB/sec execute 273 sec latency 19.502 ms oleg303-client.virtnet: 1 43389 0.73 MB/sec execute 274 sec latency 28.228 ms oleg303-client.virtnet: 1 44050 0.74 MB/sec execute 275 sec latency 20.271 ms oleg303-client.virtnet: 1 44617 0.74 MB/sec execute 276 sec latency 36.713 ms oleg303-client.virtnet: 1 45193 0.75 MB/sec execute 277 sec latency 25.660 ms oleg303-client.virtnet: 1 46284 0.78 MB/sec execute 278 sec latency 19.394 ms oleg303-client.virtnet: 1 46677 0.78 MB/sec execute 279 sec latency 12.053 ms oleg303-client.virtnet: 1 47143 0.78 MB/sec execute 280 sec latency 36.922 ms oleg303-client.virtnet: 1 47715 0.80 MB/sec execute 281 sec latency 49.324 ms oleg303-client.virtnet: 1 48298 0.80 MB/sec execute 282 sec latency 21.243 ms oleg303-client.virtnet: 1 49136 0.82 MB/sec execute 283 sec latency 21.415 ms oleg303-client.virtnet: 1 50028 0.84 MB/sec execute 284 sec latency 20.510 ms oleg303-client.virtnet: 1 50516 0.84 MB/sec execute 285 sec latency 29.883 ms oleg303-client.virtnet: 1 51081 0.85 MB/sec execute 286 sec latency 33.956 ms oleg303-client.virtnet: 1 51669 0.85 MB/sec execute 287 sec latency 31.723 ms oleg303-client.virtnet: 1 52188 0.85 MB/sec execute 288 sec latency 34.490 ms oleg303-client.virtnet: 1 53202 0.88 MB/sec execute 289 sec latency 30.440 ms oleg303-client.virtnet: 1 53605 0.89 MB/sec execute 290 sec latency 16.929 ms oleg303-client.virtnet: 1 53966 0.88 MB/sec execute 291 sec latency 38.643 ms oleg303-client.virtnet: 1 54429 0.89 MB/sec execute 292 sec latency 32.900 ms oleg303-client.virtnet: 1 55126 0.89 MB/sec execute 293 sec latency 27.565 ms oleg303-client.virtnet: 1 56004 0.91 MB/sec execute 294 sec latency 21.754 ms oleg303-client.virtnet: 1 57000 0.93 MB/sec execute 295 sec latency 22.548 ms oleg303-client.virtnet: 1 57513 0.93 MB/sec execute 296 sec latency 35.296 ms oleg303-client.virtnet: 1 58190 0.95 MB/sec execute 297 sec latency 19.434 ms oleg303-client.virtnet: 1 58880 0.94 MB/sec execute 298 sec latency 29.762 ms oleg303-client.virtnet: 1 59776 0.96 MB/sec execute 299 sec latency 24.834 ms oleg303-client.virtnet: 1 cleanup 300 sec oleg303-client.virtnet: 0 cleanup 300 sec oleg303-client.virtnet: oleg303-client.virtnet: Operation Count AvgLat MaxLat oleg303-client.virtnet: ---------------------------------------- oleg303-client.virtnet: NTCreateX 9376 20.762 30065.546 oleg303-client.virtnet: Close 6873 5.406 30215.316 oleg303-client.virtnet: Rename 399 81.740 29866.189 oleg303-client.virtnet: Unlink 1897 1.785 9.043 oleg303-client.virtnet: Qpathinfo 8480 4.838 29875.149 oleg303-client.virtnet: Qfileinfo 1483 0.118 2.314 oleg303-client.virtnet: Qfsinfo 1576 0.089 4.044 oleg303-client.virtnet: Sfileinfo 750 3.202 9.176 oleg303-client.virtnet: Find 3296 0.373 21.250 oleg303-client.virtnet: WriteX 4645 0.721 5.060 oleg303-client.virtnet: ReadX 14857 0.011 2.256 oleg303-client.virtnet: LockX 32 0.958 1.817 oleg303-client.virtnet: UnlockX 32 0.997 1.934 oleg303-client.virtnet: Flush 654 16.227 51.709 oleg303-client.virtnet: oleg303-client.virtnet: Throughput 0.964496 MB/sec 1 clients 1 procs max_latency=30215.331 ms oleg303-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg303-client.virtnet at Fri May 16 09:41:50 EDT 2025 with return code 0 oleg303-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg303-client.virtnet oleg303-client.virtnet: /mnt/lustre/d70b.replay-single/oleg303-client.virtnet /mnt/lustre/d70b.replay-single/oleg303-client.virtnet oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg303-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg303-client.virtnet: removed directory: 'clients/client0' oleg303-client.virtnet: removed directory: 'clients' oleg303-client.virtnet: removed 'client.txt' oleg303-client.virtnet: /mnt/lustre/d70b.replay-single/oleg303-client.virtnet oleg303-client.virtnet: dbench successfully finished