== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 19:58:48 (1746143928) Starting client oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Started clients oleg131-client.virtnet: 192.168.201.131@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg131-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found dbench: no process found Started rundbench load pid=14707 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2496 1285192 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26152 3556148 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7161596 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 19:59:02 (1746143942) shut down facet: mds1 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds1 to oleg131-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 19:59:20 (1746143960) targets are mounted 19:59:20 (1746143960) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: looking for dbench program oleg131-client.virtnet: /usr/bin/dbench oleg131-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg131-client.virtnet oleg131-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg131-client.virtnet' oleg131-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg131-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg131-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg131-client.virtnet at Thu May 1 19:58:50 EDT 2025 oleg131-client.virtnet: waiting for dbench pid 14738 oleg131-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg131-client.virtnet: oleg131-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg131-client.virtnet: failed to create barrier semaphore oleg131-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg131-client.virtnet: releasing clients oleg131-client.virtnet: 1 180 7.32 MB/sec warmup 1 sec latency 41.618 ms oleg131-client.virtnet: 1 298 5.48 MB/sec warmup 2 sec latency 59.489 ms oleg131-client.virtnet: 1 454 5.30 MB/sec warmup 3 sec latency 58.166 ms oleg131-client.virtnet: 1 689 5.29 MB/sec warmup 4 sec latency 52.817 ms oleg131-client.virtnet: 1 803 4.29 MB/sec warmup 5 sec latency 38.535 ms oleg131-client.virtnet: 1 904 3.61 MB/sec warmup 6 sec latency 73.732 ms oleg131-client.virtnet: 1 988 3.13 MB/sec warmup 7 sec latency 76.617 ms oleg131-client.virtnet: 1 1164 2.87 MB/sec warmup 8 sec latency 41.008 ms oleg131-client.virtnet: 1 1378 2.89 MB/sec warmup 9 sec latency 49.285 ms oleg131-client.virtnet: 1 1500 2.60 MB/sec warmup 10 sec latency 42.376 ms oleg131-client.virtnet: 1 1651 2.38 MB/sec warmup 11 sec latency 52.409 ms oleg131-client.virtnet: 1 1775 2.19 MB/sec warmup 12 sec latency 46.741 ms oleg131-client.virtnet: 1 1924 2.03 MB/sec warmup 13 sec latency 56.586 ms oleg131-client.virtnet: 1 2148 1.90 MB/sec warmup 14 sec latency 30.079 ms oleg131-client.virtnet: 1 2376 1.86 MB/sec warmup 15 sec latency 36.942 ms oleg131-client.virtnet: 1 2559 1.80 MB/sec warmup 16 sec latency 34.181 ms oleg131-client.virtnet: 1 2922 1.97 MB/sec warmup 17 sec latency 41.198 ms oleg131-client.virtnet: 1 3341 1.96 MB/sec warmup 18 sec latency 31.959 ms oleg131-client.virtnet: 1 3667 2.04 MB/sec warmup 19 sec latency 25.960 ms oleg131-client.virtnet: 1 3844 1.99 MB/sec warmup 20 sec latency 26.127 ms oleg131-client.virtnet: 1 3975 1.91 MB/sec warmup 21 sec latency 40.004 ms oleg131-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 252.740 ms oleg131-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 1252.936 ms oleg131-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 2254.631 ms oleg131-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 3256.705 ms oleg131-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 4258.255 ms oleg131-client.virtnet: 1 4083 1.48 MB/sec warmup 27 sec latency 5258.458 ms oleg131-client.virtnet: 1 4083 1.43 MB/sec warmup 28 sec latency 6258.688 ms oleg131-client.virtnet: 1 4083 1.38 MB/sec warmup 29 sec latency 7261.129 ms oleg131-client.virtnet: 1 4083 1.34 MB/sec warmup 30 sec latency 8262.660 ms oleg131-client.virtnet: 1 4083 1.29 MB/sec warmup 31 sec latency 9263.048 ms oleg131-client.virtnet: 1 4083 1.25 MB/sec warmup 32 sec latency 10263.433 ms oleg131-client.virtnet: 1 4180 1.21 MB/sec warmup 33 sec latency 10607.627 ms oleg131-client.virtnet: 1 4282 1.18 MB/sec warmup 34 sec latency 54.8mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12068 3594896 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37420 3566844 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7161740 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg131-server Stopping /mnt/lustre-mds2 (opts:) on oleg131-server 19:59:40 (1746143980) shut down facet: mds2 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds2 to oleg131-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 20:00:00 (1746144000) targets are mounted 20:00:00 (1746144000) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1285272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12200 3593252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37512 3569052 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49712 7162304 1% /mnt/lustre 43 ms oleg131-client.virtnet: 1 4350 1.16 MB/sec warmup 35 sec latency 129.027 ms oleg131-client.virtnet: 1 4448 1.13 MB/sec warmup 36 sec latency 91.522 ms oleg131-client.virtnet: 1 4564 1.11 MB/sec warmup 37 sec latency 40.527 ms oleg131-client.virtnet: 1 4723 1.11 MB/sec warmup 38 sec latency 47.829 ms oleg131-client.virtnet: 1 4840 1.08 MB/sec warmup 39 sec latency 58.262 ms oleg131-client.virtnet: 1 5007 1.13 MB/sec warmup 40 sec latency 62.880 ms oleg131-client.virtnet: 1 5075 1.10 MB/sec warmup 41 sec latency 68.293 ms oleg131-client.virtnet: 1 5156 1.08 MB/sec warmup 42 sec latency 88.156 ms oleg131-client.virtnet: 1 5243 1.05 MB/sec warmup 43 sec latency 68.090 ms oleg131-client.virtnet: 1 5342 1.03 MB/sec warmup 44 sec latency 59.508 ms oleg131-client.virtnet: 1 5436 1.01 MB/sec warmup 45 sec latency 44.699 ms oleg131-client.virtnet: 1 5581 0.99 MB/sec warmup 46 sec latency 49.673 ms oleg131-client.virtnet: 1 5693 0.97 MB/sec warmup 47 sec latency 87.861 ms oleg131-client.virtnet: 1 5730 0.95 MB/sec warmup 48 sec latency 447.073 ms oleg131-client.virtnet: 1 5730 0.93 MB/sec warmup 49 sec latency 1101.312 ms oleg131-client.virtnet: 1 5730 0.92 MB/sec warmup 50 sec latency 2102.129 ms oleg131-client.virtnet: 1 5730 0.90 MB/sec warmup 51 sec latency 3103.687 ms oleg131-client.virtnet: 1 5730 0.88 MB/sec warmup 52 sec latency 4106.477 ms oleg131-client.virtnet: 1 5730 0.86 MB/sec warmup 53 sec latency 5107.246 ms oleg131-client.virtnet: 1 5730 0.85 MB/sec warmup 54 sec latency 6112.904 ms oleg131-client.virtnet: 1 5730 0.83 MB/sec warmup 55 sec latency 7113.329 ms oleg131-client.virtnet: 1 5730 0.82 MB/sec warmup 56 sec latency 8114.494 ms oleg131-client.virtnet: 1 5730 0.80 MB/sec warmup 57 sec latency 9114.810 ms oleg131-client.virtnet: 1 5730 0.79 MB/sec warmup 58 sec latency 10117.858 ms oleg131-client.virtnet: 1 5730 0.78 MB/sec warmup 59 sec latency 11118.143 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 1 sec latency 13119.029 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 2 sec latency 14119.265 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 3 sec latency 15120.653 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 4 sec latency 16121.892 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 5 sec latency 17123.083 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 6 sec latency 18123.337 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 7 sec latency 19128.558 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 8 sec latency 20128.977 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 9 sec latency 21130.324 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 10 sec latency 22130.511 ms oleg131-client.virtnet: 1 5730 0.00 MB/sec execute 11 sec latency 23131.336 ms oleg131-client.virtnet: 1 5819 0.00 MB/sec execute 12 sec latency 23439.245 ms oleg131-client.virtnet: 1 5933 0.09 MB/sec execute 13 sec latency 64.278 ms oleg131-client.virtnet: 1 5996 0.10 MB/sec execute 14 sec latency 64.895 ms oleg131-client.virtnet: 1 6125 0.15 MB/sec execute 15 sec latency 57.184 ms oleg131-client.virtnet: 1 6449 0.38 MB/sec execute 16 sec latency 45.489 ms oleg131-client.virtnet: 1 6731 0.43 MB/sec execute 17 sec latency 33.334 ms oleg131-client.virtnet: 1 7046 0.64 MB/sec execute 18 sec latency 49.246 ms oleg131-client.virtnet: 1 7185 0.63 MB/sec execute 19 sec latency 46.451 ms oleg131-client.virtnet: 1 7274 0.61 Mtest_70b fail mds1 3 times Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 20:00:18 (1746144018) shut down facet: mds1 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds1 to oleg131-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 20:00:38 (1746144038) targets are mounted 20:00:38 (1746144038) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1285272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12220 3593972 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37420 3569176 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49640 7163148 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg131-server Stopping /mnt/lustre-mds2 (opts:) on oleg131-server 20:00:55 (1746144055) shut down facet: mds2 facet_host: oleg131-server facet_failover_host: oleg131-server B/sec execute 20 sec latency 71.762 ms oleg131-client.virtnet: 1 7363 0.62 MB/sec execute 21 sec latency 89.037 ms oleg131-client.virtnet: 1 7448 0.60 MB/sec execute 22 sec latency 58.950 ms oleg131-client.virtnet: 1 7539 0.58 MB/sec execute 23 sec latency 70.437 ms oleg131-client.virtnet: 1 7612 0.56 MB/sec execute 24 sec latency 100.491 ms oleg131-client.virtnet: 1 7713 0.53 MB/sec execute 25 sec latency 73.857 ms oleg131-client.virtnet: 1 7817 0.52 MB/sec execute 26 sec latency 57.879 ms oleg131-client.virtnet: 1 7875 0.51 MB/sec execute 27 sec latency 130.121 ms oleg131-client.virtnet: 1 7975 0.50 MB/sec execute 28 sec latency 59.577 ms oleg131-client.virtnet: 1 8081 0.49 MB/sec execute 29 sec latency 44.023 ms oleg131-client.virtnet: 1 8241 0.51 MB/sec execute 30 sec latency 49.488 ms oleg131-client.virtnet: 1 8370 0.50 MB/sec execute 31 sec latency 46.980 ms oleg131-client.virtnet: 1 8602 0.58 MB/sec execute 32 sec latency 39.280 ms oleg131-client.virtnet: 1 8723 0.56 MB/sec execute 33 sec latency 43.388 ms oleg131-client.virtnet: 1 8839 0.55 MB/sec execute 34 sec latency 56.885 ms oleg131-client.virtnet: 1 8993 0.54 MB/sec execute 35 sec latency 42.361 ms oleg131-client.virtnet: 1 9186 0.53 MB/sec execute 36 sec latency 36.880 ms oleg131-client.virtnet: 1 9343 0.52 MB/sec execute 37 sec latency 52.266 ms oleg131-client.virtnet: 1 9504 0.54 MB/sec execute 38 sec latency 64.687 ms oleg131-client.virtnet: 1 9581 0.54 MB/sec execute 39 sec latency 60.612 ms oleg131-client.virtnet: 1 9710 0.53 MB/sec execute 40 sec latency 66.641 ms oleg131-client.virtnet: 1 10016 0.63 MB/sec execute 41 sec latency 45.430 ms oleg131-client.virtnet: 1 10296 0.63 MB/sec execute 42 sec latency 37.466 ms oleg131-client.virtnet: 1 10508 0.66 MB/sec execute 43 sec latency 52.796 ms oleg131-client.virtnet: 1 10716 0.70 MB/sec execute 44 sec latency 48.062 ms oleg131-client.virtnet: 1 10866 0.71 MB/sec execute 45 sec latency 37.307 ms oleg131-client.virtnet: 1 10964 0.70 MB/sec execute 46 sec latency 63.934 ms oleg131-client.virtnet: 1 11059 0.69 MB/sec execute 47 sec latency 61.793 ms oleg131-client.virtnet: 1 11134 0.68 MB/sec execute 48 sec latency 63.960 ms oleg131-client.virtnet: 1 11322 0.67 MB/sec execute 49 sec latency 33.192 ms oleg131-client.virtnet: 1 11424 0.66 MB/sec execute 50 sec latency 79.612 ms oleg131-client.virtnet: 1 11507 0.65 MB/sec execute 51 sec latency 65.313 ms oleg131-client.virtnet: 1 11592 0.64 MB/sec execute 52 sec latency 52.145 ms oleg131-client.virtnet: 1 11708 0.65 MB/sec execute 53 sec latency 66.843 ms oleg131-client.virtnet: 1 11877 0.64 MB/sec execute 54 sec latency 37.256 ms oleg131-client.virtnet: 1 12054 0.68 MB/sec execute 55 sec latency 67.700 ms oleg131-client.virtnet: 1 12165 0.67 MB/sec execute 56 sec latency 52.205 ms oleg131-client.virtnet: 1 12280 0.66 MB/sec execute 57 sec latency 53.471 ms oleg131-client.virtnet: 1 12380 0.65 MB/sec execute 58 sec latency 48.083 ms oleg131-client.virtnet: 1 12464 0.64 MB/sec execute 59 sec latency 98.886 ms oleg131-client.virtnet: 1 12563 0.63 MB/sec execute 60 sec latency 117.046 ms oleg131-client.virtnet: 1 12712 0.62 MB/sec execute 61 sec latency 57.470 ms oleg131-client.virtnet: 1 12880 0.62 MB/sec execute 62 sec latency 61.442 ms oleg131-client.virtnet: 1 13043 0.63 MB/sec execute 63 sec latency 36.525 ms oleg131-client.virtnet: 1 13106 0.62 MB/sec execute 64 sec latency 442.445 ms oleg131-client.virtnet: 1 13106 0.6Failover mds2 to oleg131-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 20:01:14 (1746144074) targets are mounted 20:01:14 (1746144074) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1285272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12256 3593828 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37436 3566760 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49692 7160588 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 20:01:30 (1746144090) shut down facet: mds1 facet_host: oleg131-server facet_failover_host: oleg131-server 1 MB/sec execute 65 sec latency 1442.732 ms oleg131-client.virtnet: 1 13106 0.60 MB/sec execute 66 sec latency 2443.157 ms oleg131-client.virtnet: 1 13106 0.59 MB/sec execute 67 sec latency 3443.555 ms oleg131-client.virtnet: 1 13106 0.58 MB/sec execute 68 sec latency 4443.727 ms oleg131-client.virtnet: 1 13106 0.57 MB/sec execute 69 sec latency 5447.556 ms oleg131-client.virtnet: 1 13106 0.57 MB/sec execute 70 sec latency 6447.758 ms oleg131-client.virtnet: 1 13106 0.56 MB/sec execute 71 sec latency 7448.530 ms oleg131-client.virtnet: 1 13106 0.55 MB/sec execute 72 sec latency 8451.096 ms oleg131-client.virtnet: 1 13106 0.54 MB/sec execute 73 sec latency 9451.319 ms oleg131-client.virtnet: 1 13106 0.53 MB/sec execute 74 sec latency 10454.415 ms oleg131-client.virtnet: 1 13106 0.53 MB/sec execute 75 sec latency 11456.406 ms oleg131-client.virtnet: 1 13106 0.52 MB/sec execute 76 sec latency 12458.581 ms oleg131-client.virtnet: 1 13106 0.51 MB/sec execute 77 sec latency 13458.733 ms oleg131-client.virtnet: 1 13106 0.51 MB/sec execute 78 sec latency 14458.918 ms oleg131-client.virtnet: 1 13106 0.50 MB/sec execute 79 sec latency 15459.085 ms oleg131-client.virtnet: 1 13106 0.49 MB/sec execute 80 sec latency 16459.350 ms oleg131-client.virtnet: 1 13106 0.49 MB/sec execute 81 sec latency 17460.257 ms oleg131-client.virtnet: 1 13106 0.48 MB/sec execute 82 sec latency 18460.424 ms oleg131-client.virtnet: 1 13106 0.48 MB/sec execute 83 sec latency 19462.521 ms oleg131-client.virtnet: 1 13106 0.47 MB/sec execute 84 sec latency 20465.945 ms oleg131-client.virtnet: 1 13106 0.47 MB/sec execute 85 sec latency 21466.122 ms oleg131-client.virtnet: 1 13125 0.47 MB/sec execute 86 sec latency 22209.381 ms oleg131-client.virtnet: 1 13412 0.50 MB/sec execute 87 sec latency 65.013 ms oleg131-client.virtnet: 1 13643 0.51 MB/sec execute 88 sec latency 51.731 ms oleg131-client.virtnet: 1 14003 0.53 MB/sec execute 89 sec latency 77.699 ms oleg131-client.virtnet: 1 14280 0.56 MB/sec execute 90 sec latency 36.417 ms oleg131-client.virtnet: 1 14476 0.56 MB/sec execute 91 sec latency 43.677 ms oleg131-client.virtnet: 1 14597 0.56 MB/sec execute 92 sec latency 35.934 ms oleg131-client.virtnet: 1 14685 0.55 MB/sec execute 93 sec latency 54.288 ms oleg131-client.virtnet: 1 14820 0.55 MB/sec execute 94 sec latency 49.542 ms oleg131-client.virtnet: 1 14928 0.55 MB/sec execute 95 sec latency 70.563 ms oleg131-client.virtnet: 1 15053 0.54 MB/sec execute 96 sec latency 59.273 ms oleg131-client.virtnet: 1 15205 0.54 MB/sec execute 97 sec latency 41.533 ms oleg131-client.virtnet: 1 15363 0.55 MB/sec execute 98 sec latency 55.258 ms oleg131-client.virtnet: 1 15498 0.55 MB/sec execute 99 sec latency 41.827 ms oleg131-client.virtnet: 1 15711 0.57 MB/sec execute 100 sec latency 33.651 ms oleg131-client.virtnet: 1 15869 0.56 MB/sec execute 101 sec latency 54.247 ms oleg131-client.virtnet: 1 16009 0.56 MB/sec execute 102 sec latency 33.674 ms oleg131-client.virtnet: 1 16179 0.55 MB/sec execute 103 sec latency 52.231 ms oleg131-client.virtnet: 1 16337 0.55 MB/sec execute 104 sec latency 42.473 ms oleg131-client.virtnet: 1 16497 0.56 MB/sec execute 105 sec latency 41.570 ms oleg131-client.virtnet: 1 16643 0.55 MB/sec execute 106 sec latency 59.198 ms oleg131-client.virtnet: 1 16913 0.58 MB/sec execute 107 sec latency 37.016 ms oleg131-client.virtnet: 1 17182 0.59 MB/sec execute 108 sec latency 59.897 ms oleg131-client.virtnet: 1 17489 0.60 MB/sec execute 109 sec latency Failover mds1 to oleg131-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 20:01:48 (1746144108) targets are mounted 20:01:48 (1746144108) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1285272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11908 3594820 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37656 3568908 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7163728 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg131-server Stopping /mnt/lustre-mds2 (opts:) on oleg131-server 20:02:07 (1746144127) shut down facet: mds2 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds2 to oleg131-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 45.145 ms oleg131-client.virtnet: 1 17766 0.63 MB/sec execute 110 sec latency 35.480 ms oleg131-client.virtnet: 1 17872 0.62 MB/sec execute 111 sec latency 80.035 ms oleg131-client.virtnet: 1 17995 0.63 MB/sec execute 112 sec latency 52.336 ms oleg131-client.virtnet: 1 18103 0.62 MB/sec execute 113 sec latency 40.182 ms oleg131-client.virtnet: 1 18211 0.62 MB/sec execute 114 sec latency 51.497 ms oleg131-client.virtnet: 1 18266 0.61 MB/sec execute 115 sec latency 604.016 ms oleg131-client.virtnet: 1 18266 0.61 MB/sec execute 116 sec latency 1605.125 ms oleg131-client.virtnet: 1 18266 0.60 MB/sec execute 117 sec latency 2605.242 ms oleg131-client.virtnet: 1 18266 0.60 MB/sec execute 118 sec latency 3605.412 ms oleg131-client.virtnet: 1 18266 0.59 MB/sec execute 119 sec latency 4605.717 ms oleg131-client.virtnet: 1 18266 0.59 MB/sec execute 120 sec latency 5605.901 ms oleg131-client.virtnet: 1 18364 0.58 MB/sec execute 121 sec latency 6047.320 ms oleg131-client.virtnet: 1 18514 0.58 MB/sec execute 122 sec latency 46.092 ms oleg131-client.virtnet: 1 18628 0.58 MB/sec execute 123 sec latency 65.651 ms oleg131-client.virtnet: 1 18733 0.58 MB/sec execute 124 sec latency 69.617 ms oleg131-client.virtnet: 1 18907 0.58 MB/sec execute 125 sec latency 43.355 ms oleg131-client.virtnet: 1 19138 0.60 MB/sec execute 126 sec latency 52.446 ms oleg131-client.virtnet: 1 19248 0.60 MB/sec execute 127 sec latency 39.060 ms oleg131-client.virtnet: 1 19359 0.59 MB/sec execute 128 sec latency 56.663 ms oleg131-client.virtnet: 1 19469 0.59 MB/sec execute 129 sec latency 57.097 ms oleg131-client.virtnet: 1 19585 0.58 MB/sec execute 130 sec latency 58.188 ms oleg131-client.virtnet: 1 19790 0.58 MB/sec execute 131 sec latency 33.913 ms oleg131-client.virtnet: 1 19953 0.58 MB/sec execute 132 sec latency 46.256 ms oleg131-client.virtnet: 1 20130 0.58 MB/sec execute 133 sec latency 38.117 ms oleg131-client.virtnet: 1 20158 0.58 MB/sec execute 134 sec latency 767.786 ms oleg131-client.virtnet: 1 20158 0.58 MB/sec execute 135 sec latency 1769.610 ms oleg131-client.virtnet: 1 20158 0.57 MB/sec execute 136 sec latency 2775.501 ms oleg131-client.virtnet: 1 20158 0.57 MB/sec execute 137 sec latency 3775.985 ms oleg131-client.virtnet: 1 20158 0.56 MB/sec execute 138 sec latency 4777.291 ms oleg131-client.virtnet: 1 20158 0.56 MB/sec execute 139 sec latency 5779.851 ms oleg131-client.virtnet: 1 20158 0.55 MB/sec execute 140 sec latency 6780.270 ms oleg131-client.virtnet: 1 20158 0.55 MB/sec execute 141 sec latency 7782.006 ms oleg131-client.virtnet: 1 20158 0.55 MB/sec execute 142 sec latency 8782.170 ms oleg131-client.virtnet: 1 20158 0.54 MB/sec execute 143 sec latency 9782.347 ms oleg131-client.virtnet: 1 20158 0.54 MB/sec execute 144 sec latency 10782.612 ms oleg131-client.virtnet: 1 20158 0.54 MB/sec execute 145 sec latency 11782.760 ms oleg131-client.virtnet: 1 20158 0.53 MB/sec execute 146 sec latency 12783.388 ms oleg131-client.virtnet: 1 20158 0.53 MB/sec execute 147 sec latency 13784.756 ms oleg131-client.virtnet: 1 20158 0.52 MB/sec execute 148 sec latency 14785.157 ms oleg131-client.virtnet: 1 20158 0.52 MB/sec execute 149 sec latency 15785.309 ms oleg131-client.virtnet: 1 20158 0.52 MB/sec execute 150 sec latency 16786.296 ms oleg131-client.virtnet: 1 20158 0.51 MB/sec execute 151 sec latency 17786.422 ms oleg131-client.virtnet: 1 20158 0.51 MB/sec execute 152 sec latency 18786.717 ms oleg131-client.virtnet: 1 20158 0.51 MB/sec execute 153 sec latency 19786.903 ms oleg131-client.virtnoleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 20:02:26 (1746144146) targets are mounted 20:02:26 (1746144146) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1285272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12036 3594848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37644 3566804 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49680 7161652 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 20:02:43 (1746144163) shut down facet: mds1 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds1 to oleg131-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 20:03:03 (1746144183) targets are mounted 20:03:03 (1746144183) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec et: 1 20158 0.50 MB/sec execute 154 sec latency 20787.088 ms oleg131-client.virtnet: 1 20158 0.50 MB/sec execute 155 sec latency 21787.330 ms oleg131-client.virtnet: 1 20158 0.50 MB/sec execute 156 sec latency 22791.743 ms oleg131-client.virtnet: 1 20158 0.49 MB/sec execute 157 sec latency 23791.942 ms oleg131-client.virtnet: 1 20161 0.49 MB/sec execute 158 sec latency 24760.828 ms oleg131-client.virtnet: 1 20225 0.49 MB/sec execute 159 sec latency 67.644 ms oleg131-client.virtnet: 1 20369 0.49 MB/sec execute 160 sec latency 59.152 ms oleg131-client.virtnet: 1 20655 0.51 MB/sec execute 161 sec latency 77.628 ms oleg131-client.virtnet: 1 20790 0.51 MB/sec execute 162 sec latency 41.803 ms oleg131-client.virtnet: 1 21118 0.52 MB/sec execute 163 sec latency 83.034 ms oleg131-client.virtnet: 1 21346 0.54 MB/sec execute 164 sec latency 39.236 ms oleg131-client.virtnet: 1 21462 0.54 MB/sec execute 165 sec latency 46.508 ms oleg131-client.virtnet: 1 21601 0.54 MB/sec execute 166 sec latency 47.794 ms oleg131-client.virtnet: 1 21701 0.54 MB/sec execute 167 sec latency 51.416 ms oleg131-client.virtnet: 1 21783 0.53 MB/sec execute 168 sec latency 98.424 ms oleg131-client.virtnet: 1 21908 0.53 MB/sec execute 169 sec latency 60.660 ms oleg131-client.virtnet: 1 22064 0.53 MB/sec execute 170 sec latency 40.658 ms oleg131-client.virtnet: 1 22135 0.53 MB/sec execute 171 sec latency 70.664 ms oleg131-client.virtnet: 1 22264 0.53 MB/sec execute 172 sec latency 53.287 ms oleg131-client.virtnet: 1 22396 0.53 MB/sec execute 173 sec latency 53.683 ms oleg131-client.virtnet: 1 22527 0.53 MB/sec execute 174 sec latency 61.763 ms oleg131-client.virtnet: 1 22750 0.54 MB/sec execute 175 sec latency 40.996 ms oleg131-client.virtnet: 1 22890 0.54 MB/sec execute 176 sec latency 47.256 ms oleg131-client.virtnet: 1 23019 0.54 MB/sec execute 177 sec latency 37.531 ms oleg131-client.virtnet: 1 23135 0.53 MB/sec execute 178 sec latency 59.626 ms oleg131-client.virtnet: 1 23324 0.53 MB/sec execute 179 sec latency 55.332 ms oleg131-client.virtnet: 1 23495 0.53 MB/sec execute 180 sec latency 38.817 ms oleg131-client.virtnet: 1 23703 0.53 MB/sec execute 181 sec latency 43.804 ms oleg131-client.virtnet: 1 23895 0.54 MB/sec execute 182 sec latency 31.773 ms oleg131-client.virtnet: 1 24254 0.56 MB/sec execute 183 sec latency 28.152 ms oleg131-client.virtnet: 1 24520 0.56 MB/sec execute 184 sec latency 56.530 ms oleg131-client.virtnet: 1 24755 0.57 MB/sec execute 185 sec latency 49.787 ms oleg131-client.virtnet: 1 24944 0.58 MB/sec execute 186 sec latency 63.740 ms oleg131-client.virtnet: 1 25044 0.58 MB/sec execute 187 sec latency 59.795 ms oleg131-client.virtnet: 1 25167 0.58 MB/sec execute 188 sec latency 59.572 ms oleg131-client.virtnet: 1 25259 0.58 MB/sec execute 189 sec latency 63.396 ms oleg131-client.virtnet: 1 25355 0.57 MB/sec execute 190 sec latency 44.488 ms oleg131-client.virtnet: 1 25494 0.57 MB/sec execute 191 sec latency 63.075 ms oleg131-client.virtnet: 1 25607 0.57 MB/sec execute 192 sec latency 52.034 ms oleg131-client.virtnet: 1 25685 0.57 MB/sec execute 193 sec latency 82.286 ms oleg131-client.virtnet: 1 25789 0.57 MB/sec execute 194 sec latency 54.641 ms oleg131-client.virtnet: 1 25893 0.57 MB/sec execute 195 sec latency 68.431 ms oleg131-client.virtnet: 1 26042 0.57 MB/sec execute 196 sec latency 54.530 ms oleg131-client.virtnet: 1 26166 0.56 MB/sec execute 197 sec latency 72.002 ms oleg131-client.virtnet: 1 26375 0.58 MB/sec execute 198 sec latency 67.571 ms oleg13UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1285272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11916 3594332 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37584 3568812 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7163144 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg131-server Stopping /mnt/lustre-mds2 (opts:) on oleg131-server 20:03:20 (1746144200) shut down facet: mds2 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds2 to oleg131-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 20:03:38 (1746144218) targets are mounted 20:03:38 (1746144218) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1285272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12012 3594876 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37656 3567032 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49668 7161908 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 1-client.virtnet: 1 26490 0.57 MB/sec execute 199 sec latency 65.836 ms oleg131-client.virtnet: 1 26617 0.57 MB/sec execute 200 sec latency 39.264 ms oleg131-client.virtnet: 1 26713 0.57 MB/sec execute 201 sec latency 53.501 ms oleg131-client.virtnet: 1 26859 0.57 MB/sec execute 202 sec latency 57.252 ms oleg131-client.virtnet: 1 26999 0.56 MB/sec execute 203 sec latency 61.119 ms oleg131-client.virtnet: 1 27119 0.56 MB/sec execute 204 sec latency 48.654 ms oleg131-client.virtnet: 1 27288 0.57 MB/sec execute 205 sec latency 49.576 ms oleg131-client.virtnet: 1 27468 0.57 MB/sec execute 206 sec latency 39.764 ms oleg131-client.virtnet: 1 27772 0.58 MB/sec execute 207 sec latency 46.791 ms oleg131-client.virtnet: 1 27881 0.58 MB/sec execute 208 sec latency 290.540 ms oleg131-client.virtnet: 1 27881 0.58 MB/sec execute 209 sec latency 1290.688 ms oleg131-client.virtnet: 1 27881 0.58 MB/sec execute 210 sec latency 2290.913 ms oleg131-client.virtnet: 1 27881 0.58 MB/sec execute 211 sec latency 3292.302 ms oleg131-client.virtnet: 1 27881 0.57 MB/sec execute 212 sec latency 4292.518 ms oleg131-client.virtnet: 1 27881 0.57 MB/sec execute 213 sec latency 5292.728 ms oleg131-client.virtnet: 1 27881 0.57 MB/sec execute 214 sec latency 6293.417 ms oleg131-client.virtnet: 1 27881 0.56 MB/sec execute 215 sec latency 7293.841 ms oleg131-client.virtnet: 1 27881 0.56 MB/sec execute 216 sec latency 8294.703 ms oleg131-client.virtnet: 1 27881 0.56 MB/sec execute 217 sec latency 9295.229 ms oleg131-client.virtnet: 1 27881 0.56 MB/sec execute 218 sec latency 10295.369 ms oleg131-client.virtnet: 1 27881 0.55 MB/sec execute 219 sec latency 11295.820 ms oleg131-client.virtnet: 1 27881 0.55 MB/sec execute 220 sec latency 12296.473 ms oleg131-client.virtnet: 1 27881 0.55 MB/sec execute 221 sec latency 13296.673 ms oleg131-client.virtnet: 1 27881 0.55 MB/sec execute 222 sec latency 14297.710 ms oleg131-client.virtnet: 1 27881 0.54 MB/sec execute 223 sec latency 15302.850 ms oleg131-client.virtnet: 1 27881 0.54 MB/sec execute 224 sec latency 16303.077 ms oleg131-client.virtnet: 1 27881 0.54 MB/sec execute 225 sec latency 17303.487 ms oleg131-client.virtnet: 1 27881 0.54 MB/sec execute 226 sec latency 18305.466 ms oleg131-client.virtnet: 1 27881 0.54 MB/sec execute 227 sec latency 19305.616 ms oleg131-client.virtnet: 1 27881 0.53 MB/sec execute 228 sec latency 20307.078 ms oleg131-client.virtnet: 1 27881 0.53 MB/sec execute 229 sec latency 21307.313 ms oleg131-client.virtnet: 1 27881 0.53 MB/sec execute 230 sec latency 22308.287 ms oleg131-client.virtnet: 1 28067 0.53 MB/sec execute 231 sec latency 22842.799 ms oleg131-client.virtnet: 1 28374 0.54 MB/sec execute 232 sec latency 43.171 ms oleg131-client.virtnet: 1 28489 0.54 MB/sec execute 233 sec latency 41.805 ms oleg131-client.virtnet: 1 28643 0.55 MB/sec execute 234 sec latency 35.527 ms oleg131-client.virtnet: 1 28778 0.54 MB/sec execute 235 sec latency 72.504 ms oleg131-client.virtnet: 1 28898 0.54 MB/sec execute 236 sec latency 38.783 ms oleg131-client.virtnet: 1 28993 0.54 MB/sec execute 237 sec latency 68.401 ms oleg131-client.virtnet: 1 29138 0.54 MB/sec execute 238 sec latency 40.184 ms oleg131-client.virtnet: 1 29275 0.54 MB/sec execute 239 sec latency 37.857 ms oleg131-client.virtnet: 1 29381 0.54 MB/sec execute 240 sec latency 58.362 ms oleg131-client.virtnet: 1 29551 0.54 MB/sec execute 241 sec latency 64.956 ms oleg131-client.virtnet: 1 29691 0.54 MB/sec execute 242 sec latency 40.783 ms oleg131-client.virtnet: 1 298820:03:55 (1746144235) shut down facet: mds1 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds1 to oleg131-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 20:04:14 (1746144254) targets are mounted 20:04:14 (1746144254) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 8 0.55 MB/sec execute 243 sec latency 80.293 ms oleg131-client.virtnet: 1 30026 0.55 MB/sec execute 244 sec latency 49.976 ms oleg131-client.virtnet: 1 30181 0.54 MB/sec execute 245 sec latency 34.214 ms oleg131-client.virtnet: 1 30388 0.54 MB/sec execute 246 sec latency 29.915 ms oleg131-client.virtnet: 1 30581 0.54 MB/sec execute 247 sec latency 35.776 ms oleg131-client.virtnet: 1 30815 0.54 MB/sec execute 248 sec latency 28.828 ms oleg131-client.virtnet: 1 31012 0.55 MB/sec execute 249 sec latency 31.718 ms oleg131-client.virtnet: 1 31404 0.56 MB/sec execute 250 sec latency 33.424 ms oleg131-client.virtnet: 1 31813 0.57 MB/sec execute 251 sec latency 26.837 ms oleg131-client.virtnet: 1 32046 0.58 MB/sec execute 252 sec latency 119.559 ms oleg131-client.virtnet: 1 32213 0.58 MB/sec execute 253 sec latency 39.606 ms oleg131-client.virtnet: 1 32328 0.58 MB/sec execute 254 sec latency 42.663 ms oleg131-client.virtnet: 1 32424 0.58 MB/sec execute 255 sec latency 43.775 ms oleg131-client.virtnet: 1 32489 0.57 MB/sec execute 256 sec latency 417.048 ms oleg131-client.virtnet: 1 32489 0.57 MB/sec execute 257 sec latency 1417.240 ms oleg131-client.virtnet: 1 32489 0.57 MB/sec execute 258 sec latency 2418.042 ms oleg131-client.virtnet: 1 32489 0.57 MB/sec execute 259 sec latency 3419.743 ms oleg131-client.virtnet: 1 32489 0.57 MB/sec execute 260 sec latency 4420.128 ms oleg131-client.virtnet: 1 32489 0.56 MB/sec execute 261 sec latency 5420.295 ms oleg131-client.virtnet: 1 32489 0.56 MB/sec execute 262 sec latency 6423.341 ms oleg131-client.virtnet: 1 32489 0.56 MB/sec execute 263 sec latency 7423.652 ms oleg131-client.virtnet: 1 32489 0.56 MB/sec execute 264 sec latency 8425.082 ms oleg131-client.virtnet: 1 32489 0.56 MB/sec execute 265 sec latency 9425.314 ms oleg131-client.virtnet: 1 32502 0.55 MB/sec execute 266 sec latency 10247.567 ms oleg131-client.virtnet: 1 32651 0.55 MB/sec execute 267 sec latency 56.756 ms oleg131-client.virtnet: 1 32755 0.55 MB/sec execute 268 sec latency 59.218 ms oleg131-client.virtnet: 1 32889 0.55 MB/sec execute 269 sec latency 36.024 ms oleg131-client.virtnet: 1 33098 0.55 MB/sec execute 270 sec latency 29.587 ms oleg131-client.virtnet: 1 33256 0.55 MB/sec execute 271 sec latency 36.112 ms oleg131-client.virtnet: 1 33473 0.56 MB/sec execute 272 sec latency 32.596 ms oleg131-client.virtnet: 1 33651 0.56 MB/sec execute 273 sec latency 36.355 ms oleg131-client.virtnet: 1 33802 0.56 MB/sec execute 274 sec latency 39.177 ms oleg131-client.virtnet: 1 33986 0.56 MB/sec execute 275 sec latency 36.170 ms oleg131-client.virtnet: 1 34194 0.55 MB/sec execute 276 sec latency 25.776 ms oleg131-client.virtnet: 1 34413 0.56 MB/sec execute 277 sec latency 32.055 ms oleg131-client.virtnet: 1 34725 0.57 MB/sec execute 278 sec latency 38.011 ms oleg131-client.virtnet: 1 34955 0.57 MB/sec execute 279 sec latency 36.684 ms oleg131-client.virtnet: 1 35331 0.58 MB/sec execute 280 sec latency 41.352 ms oleg131-client.virtnet: 1 35571 0.59 MB/sec execute 281 sec latency 38.512 ms oleg131-client.virtnet: 1 35736 0.59 MB/sec execute 282 sec latency 32.175 ms oleg131-client.virtnet: 1 35866 0.59 MB/sec execute 283 sec latency 29.759 ms oleg131-client.virtnet: 1 35997 0.59 MB/sec execute 284 sec latency 31.930 ms oleg131-client.virtnet: 1 36186 0.58 MB/sec execute 285 sec latency 36.638 ms oleg131-client.virtnet: 1 36323 0.58 MB/sec execute 286 sec latency 43.562 ms oleg131-client.virtnet: 1 36490 0.58 MB/sec execute 287 sec latency 39.155 ms oleg131-client.virtnet: 1 36679 0.58 MB/sec execute 288 sec latency 37.500 ms oleg131-client.virtnet: 1 36933 0.59 MB/sec execute 289 sec latency 42.639 ms oleg131-client.virtnet: 1 37088 0.59 MB/sec execute 290 sec latency 37.682 ms oleg131-client.virtnet: 1 37227 0.59 MB/sec execute 291 sec latency 47.248 ms oleg131-client.virtnet: 1 37353 0.59 MB/sec execute 292 sec latency 37.837 ms oleg131-client.virtnet: 1 37536 0.59 MB/sec execute 293 sec latency 47.845 ms oleg131-client.virtnet: 1 37685 0.59 MB/sec execute 294 sec latency 49.830 ms oleg131-client.virtnet: 1 37909 0.59 MB/sec execute 295 sec latency 38.865 ms oleg131-client.virtnet: 1 38101 0.59 MB/sec execute 296 sec latency 34.004 ms oleg131-client.virtnet: 1 38435 0.60 MB/sec execute 297 sec latency 77.619 ms oleg131-client.virtnet: 1 38701 0.60 MB/sec execute 298 sec latency 60.804 ms oleg131-client.virtnet: 1 39012 0.61 MB/sec execute 299 sec latency 47.537 ms oleg131-client.virtnet: 1 cleanup 300 sec oleg131-client.virtnet: 0 cleanup 301 sec oleg131-client.virtnet: oleg131-client.virtnet: Operation Count AvgLat MaxLat oleg131-client.virtnet: ---------------------------------------- oleg131-client.virtnet: NTCreateX 5692 28.291 23439.231 oleg131-client.virtnet: Close 4199 4.037 36.907 oleg131-client.virtnet: Rename 247 29.220 104.948 oleg131-client.virtnet: Unlink 1146 26.225 22209.360 oleg131-client.virtnet: Qpathinfo 5179 7.919 10247.545 oleg131-client.virtnet: Qfileinfo 921 0.360 9.955 oleg131-client.virtnet: Qfsinfo 974 0.941 74.308 oleg131-client.virtnet: Sfileinfo 466 72.759 24760.818 oleg131-client.virtnet: Find 2037 1.064 49.607 oleg131-client.virtnet: WriteX 2857 2.900 56.764 oleg131-client.virtnet: ReadX 9259 0.067 45.975 oleg131-client.virtnet: LockX 20 5.814 15.527 oleg131-client.virtnet: UnlockX 20 4.412 12.611 oleg131-client.virtnet: Flush 400 22.699 130.102 oleg131-client.virtnet: oleg131-client.virtnet: Throughput 0.613976 MB/sec 1 clients 1 procs max_latency=24760.828 ms oleg131-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg131-client.virtnet at Thu May 1 20:04:51 EDT 2025 with return code 0 oleg131-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg131-client.virtnet oleg131-client.virtnet: /mnt/lustre/d70b.replay-single/oleg131-client.virtnet /mnt/lustre/d70b.replay-single/oleg131-client.virtnet oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg131-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg131-client.virtnet: removed directory: 'clients/client0' oleg131-client.virtnet: removed directory: 'clients' oleg131-client.virtnet: removed 'client.txt' oleg131-client.virtnet: /mnt/lustre/d70b.replay-single/oleg131-client.virtnet oleg131-client.virtnet: dbench successfully finished