== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 23:51:36 (1735102296) Starting client oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre Started clients oleg107-client.virtnet: 192.168.201.107@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg107-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13584 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2468 1285220 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3580864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1888 3592920 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28044 7173784 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg107-server Stopping /mnt/lustre-mds1 (opts:) on oleg107-server 23:51:43 (1735102303) shut down facet: mds1 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds1 to oleg107-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 23:51:57 (1735102317) targets are mounted 23:51:57 (1735102317) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26448 3569768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1736 3593060 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28184 7162828 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg107-server Stopping /mnt/lustre-mds2 (opts:) on oleg107-server 23:52:08 (1735102328) shut down facet: mds2 facet_host: oleg107-server facet_failover_host: oleg107-server oleg107-client.virtnet: looking for dbench program oleg107-client.virtnet: /usr/bin/dbench oleg107-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg107-client.virtnet oleg107-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg107-client.virtnet' oleg107-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg107-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg107-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg107-client.virtnet at Tue Dec 24 23:51:37 EST 2024 oleg107-client.virtnet: waiting for dbench pid 13608 oleg107-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg107-client.virtnet: oleg107-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg107-client.virtnet: failed to create barrier semaphore oleg107-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg107-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg107-client.virtnet: releasing clients oleg107-client.virtnet: 1 511 17.73 MB/sec warmup 1 sec latency 13.159 ms oleg107-client.virtnet: 1 1133 11.49 MB/sec warmup 2 sec latency 11.090 ms oleg107-client.virtnet: 1 1713 8.75 MB/sec warmup 3 sec latency 11.144 ms oleg107-client.virtnet: 1 2210 6.69 MB/sec warmup 4 sec latency 10.298 ms oleg107-client.virtnet: 1 2917 6.69 MB/sec warmup 5 sec latency 14.271 ms oleg107-client.virtnet: 1 3773 6.63 MB/sec warmup 6 sec latency 13.590 ms oleg107-client.virtnet: 1 4083 5.73 MB/sec warmup 7 sec latency 353.916 ms oleg107-client.virtnet: 1 4083 5.01 MB/sec warmup 8 sec latency 1354.089 ms oleg107-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 2354.234 ms oleg107-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 3354.365 ms oleg107-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 4354.511 ms oleg107-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 5354.669 ms oleg107-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 6354.843 ms oleg107-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 7354.992 ms oleg107-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 8355.338 ms oleg107-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 9355.491 ms oleg107-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 10355.688 ms oleg107-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 11355.845 ms oleg107-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 12356.058 ms oleg107-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 13356.196 ms oleg107-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 14356.321 ms oleg107-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 15356.467 ms oleg107-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 16356.620 ms oleg107-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 17356.753 ms oleg107-client.virtnet: 1 4714 1.68 MB/sec warmup 25 sec latency 17412.964 ms oleg107-client.virtnet: 1 5399 1.75 MB/sec warmup 26 sec latency 10.255 ms oleg107-client.virtnet: 1 6152 1.78 MB/sec warmup 27 sec latency 9.679 ms oleg107-client.virtnet: 1 7222 2.07 MB/sec warmup 28 sec latency 10.892 ms oleg107-client.virtnet: 1 7755 2.04 MB/sec warmup 29 sec latency 10.071 ms oleg107-client.virtnet: 1 8289 2.04 MB/sec warmup 30 sec latency 104.886 ms oleg107-client.virtnet: 1 8887 2.08 MB/sec warmup 31 sec latency 10.444 ms oleg107-client.virtnet: 1 9033 2.02 MB/sec warmup 32 sec latency 747.553 ms oleg107-client.virtnet: 1 9033 1.96 MB/sec warmup 33 sec latency 174Failover mds2 to oleg107-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 23:52:22 (1735102342) targets are mounted 23:52:22 (1735102342) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36568 3569752 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13204 3590860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49772 7160612 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg107-server Stopping /mnt/lustre-mds1 (opts:) on oleg107-server 23:52:34 (1735102354) shut down facet: mds1 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds1 to oleg107-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 23:52:48 (1735102368) targets are mounted 23:52:48 (1735102368) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36536 3569896 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13180 3590788 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49716 7160684 1% /mnt/lustre 7.754 ms oleg107-client.virtnet: 1 9033 1.90 MB/sec warmup 34 sec latency 2747.909 ms oleg107-client.virtnet: 1 9033 1.85 MB/sec warmup 35 sec latency 3748.056 ms oleg107-client.virtnet: 1 9033 1.80 MB/sec warmup 36 sec latency 4748.216 ms oleg107-client.virtnet: 1 9033 1.75 MB/sec warmup 37 sec latency 5748.365 ms oleg107-client.virtnet: 1 9033 1.70 MB/sec warmup 38 sec latency 6748.518 ms oleg107-client.virtnet: 1 9033 1.66 MB/sec warmup 39 sec latency 7748.665 ms oleg107-client.virtnet: 1 9033 1.62 MB/sec warmup 40 sec latency 8748.772 ms oleg107-client.virtnet: 1 9033 1.58 MB/sec warmup 41 sec latency 9748.878 ms oleg107-client.virtnet: 1 9033 1.54 MB/sec warmup 42 sec latency 10749.004 ms oleg107-client.virtnet: 1 9033 1.50 MB/sec warmup 43 sec latency 11749.130 ms oleg107-client.virtnet: 1 9033 1.47 MB/sec warmup 44 sec latency 12749.245 ms oleg107-client.virtnet: 1 9033 1.44 MB/sec warmup 45 sec latency 13749.345 ms oleg107-client.virtnet: 1 9033 1.41 MB/sec warmup 46 sec latency 14749.465 ms oleg107-client.virtnet: 1 9033 1.38 MB/sec warmup 47 sec latency 15749.588 ms oleg107-client.virtnet: 1 9033 1.35 MB/sec warmup 48 sec latency 16749.697 ms oleg107-client.virtnet: 1 9033 1.32 MB/sec warmup 49 sec latency 17749.808 ms oleg107-client.virtnet: 1 9260 1.30 MB/sec warmup 50 sec latency 18389.809 ms oleg107-client.virtnet: 1 9847 1.38 MB/sec warmup 51 sec latency 18.029 ms oleg107-client.virtnet: 1 10792 1.48 MB/sec warmup 52 sec latency 9.749 ms oleg107-client.virtnet: 1 11330 1.48 MB/sec warmup 53 sec latency 10.925 ms oleg107-client.virtnet: 1 11850 1.49 MB/sec warmup 54 sec latency 11.349 ms oleg107-client.virtnet: 1 12384 1.52 MB/sec warmup 55 sec latency 11.759 ms oleg107-client.virtnet: 1 12951 1.52 MB/sec warmup 56 sec latency 12.442 ms oleg107-client.virtnet: 1 13859 1.61 MB/sec warmup 57 sec latency 11.520 ms oleg107-client.virtnet: 1 14595 1.68 MB/sec warmup 58 sec latency 27.043 ms oleg107-client.virtnet: 1 15190 1.67 MB/sec warmup 59 sec latency 10.976 ms oleg107-client.virtnet: 1 16352 0.54 MB/sec execute 1 sec latency 11.287 ms oleg107-client.virtnet: 1 17187 3.68 MB/sec execute 2 sec latency 9.211 ms oleg107-client.virtnet: 1 18035 4.53 MB/sec execute 3 sec latency 11.039 ms oleg107-client.virtnet: 1 18566 3.61 MB/sec execute 4 sec latency 15.144 ms oleg107-client.virtnet: 1 19232 3.80 MB/sec execute 5 sec latency 8.637 ms oleg107-client.virtnet: 1 19771 3.25 MB/sec execute 6 sec latency 14.908 ms oleg107-client.virtnet: 1 20562 3.57 MB/sec execute 7 sec latency 8.215 ms oleg107-client.virtnet: 1 21600 4.10 MB/sec execute 8 sec latency 10.808 ms oleg107-client.virtnet: 1 22209 3.75 MB/sec execute 9 sec latency 9.353 ms oleg107-client.virtnet: 1 22944 3.83 MB/sec execute 10 sec latency 7.662 ms oleg107-client.virtnet: 1 23590 3.55 MB/sec execute 11 sec latency 8.862 ms oleg107-client.virtnet: 1 24712 3.97 MB/sec execute 12 sec latency 8.729 ms oleg107-client.virtnet: 1 25432 4.02 MB/sec execute 13 sec latency 7.506 ms oleg107-client.virtnet: 1 26126 3.88 MB/sec execute 14 sec latency 9.785 ms oleg107-client.virtnet: 1 26861 3.85 MB/sec execute 15 sec latency 10.122 ms oleg107-client.virtnet: 1 27507 3.79 MB/sec execute 16 sec latency 11.756 ms oleg107-client.virtnet: 1 28671 4.18 MB/sec execute 17 sec latency 9.107 ms oleg107-client.virtnet: 1 29209 3.99 MB/sec execute 18 sec latency 13.036 ms oleg107-client.virtnet: 1 29879 4.02 MB/sec execute 19 sec latetest_70b fail mds2 4 times Failing mds2 on oleg107-server Stopping /mnt/lustre-mds2 (opts:) on oleg107-server 23:52:59 (1735102379) shut down facet: mds2 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds2 to oleg107-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 23:53:13 (1735102393) targets are mounted 23:53:13 (1735102393) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36560 3569712 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12932 3593100 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7162812 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg107-server Stopping /mnt/lustre-mds1 (opts:) on oleg107-server 23:53:25 (1735102405) shut down facet: mds1 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds1 to oleg107-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 23:53:39 (1735102419) targets are mounted 23:53:39 (1735102419) facet_failover done ncy 9.170 ms oleg107-client.virtnet: 1 30307 3.84 MB/sec execute 20 sec latency 142.680 ms oleg107-client.virtnet: 1 30906 3.77 MB/sec execute 21 sec latency 247.567 ms oleg107-client.virtnet: 1 32041 4.05 MB/sec execute 22 sec latency 10.689 ms oleg107-client.virtnet: 1 32171 3.92 MB/sec execute 23 sec latency 805.843 ms oleg107-client.virtnet: 1 32171 3.76 MB/sec execute 24 sec latency 1805.939 ms oleg107-client.virtnet: 1 32171 3.61 MB/sec execute 25 sec latency 2806.095 ms oleg107-client.virtnet: 1 32171 3.47 MB/sec execute 26 sec latency 3806.210 ms oleg107-client.virtnet: 1 32171 3.34 MB/sec execute 27 sec latency 4806.363 ms oleg107-client.virtnet: 1 32171 3.22 MB/sec execute 28 sec latency 5806.510 ms oleg107-client.virtnet: 1 32171 3.11 MB/sec execute 29 sec latency 6806.663 ms oleg107-client.virtnet: 1 32171 3.00 MB/sec execute 30 sec latency 7806.798 ms oleg107-client.virtnet: 1 32171 2.91 MB/sec execute 31 sec latency 8806.948 ms oleg107-client.virtnet: 1 32171 2.82 MB/sec execute 32 sec latency 9807.055 ms oleg107-client.virtnet: 1 32171 2.73 MB/sec execute 33 sec latency 10807.184 ms oleg107-client.virtnet: 1 32171 2.65 MB/sec execute 34 sec latency 11807.308 ms oleg107-client.virtnet: 1 32171 2.58 MB/sec execute 35 sec latency 12807.430 ms oleg107-client.virtnet: 1 32171 2.50 MB/sec execute 36 sec latency 13807.654 ms oleg107-client.virtnet: 1 32171 2.44 MB/sec execute 37 sec latency 14807.835 ms oleg107-client.virtnet: 1 32171 2.37 MB/sec execute 38 sec latency 15807.960 ms oleg107-client.virtnet: 1 32171 2.31 MB/sec execute 39 sec latency 16808.126 ms oleg107-client.virtnet: 1 32171 2.25 MB/sec execute 40 sec latency 17808.274 ms oleg107-client.virtnet: 1 32365 2.21 MB/sec execute 41 sec latency 18346.044 ms oleg107-client.virtnet: 1 32867 2.17 MB/sec execute 42 sec latency 10.659 ms oleg107-client.virtnet: 1 33498 2.22 MB/sec execute 43 sec latency 13.922 ms oleg107-client.virtnet: 1 34112 2.19 MB/sec execute 44 sec latency 9.769 ms oleg107-client.virtnet: 1 34938 2.29 MB/sec execute 45 sec latency 10.443 ms oleg107-client.virtnet: 1 35802 2.37 MB/sec execute 46 sec latency 9.699 ms oleg107-client.virtnet: 1 36391 2.35 MB/sec execute 47 sec latency 11.165 ms oleg107-client.virtnet: 1 37024 2.39 MB/sec execute 48 sec latency 20.143 ms oleg107-client.virtnet: 1 37656 2.35 MB/sec execute 49 sec latency 10.097 ms oleg107-client.virtnet: 1 38506 2.44 MB/sec execute 50 sec latency 15.159 ms oleg107-client.virtnet: 1 39380 2.52 MB/sec execute 51 sec latency 9.300 ms oleg107-client.virtnet: 1 39936 2.49 MB/sec execute 52 sec latency 10.803 ms oleg107-client.virtnet: 1 40563 2.52 MB/sec execute 53 sec latency 10.981 ms oleg107-client.virtnet: 1 41146 2.49 MB/sec execute 54 sec latency 10.098 ms oleg107-client.virtnet: 1 41976 2.57 MB/sec execute 55 sec latency 10.473 ms oleg107-client.virtnet: 1 42863 2.63 MB/sec execute 56 sec latency 9.962 ms oleg107-client.virtnet: 1 43413 2.60 MB/sec execute 57 sec latency 10.285 ms oleg107-client.virtnet: 1 44091 2.64 MB/sec execute 58 sec latency 8.961 ms oleg107-client.virtnet: 1 44736 2.60 MB/sec execute 59 sec latency 9.336 ms oleg107-client.virtnet: 1 45595 2.67 MB/sec execute 60 sec latency 9.267 ms oleg107-client.virtnet: 1 46469 2.73 MB/sec execute 61 sec latency 9.273 ms oleg107-client.virtnet: 1 47031 2.70 MB/sec execute 62 sec latency 10.903 ms oleg107-client.virtnet: 1 47660 2.73 MB/sec execute 63 sec latency 19.141 ms oleg107-client.virtnet: 1 48302 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36712 3569656 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13052 3593052 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49764 7162708 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg107-server Stopping /mnt/lustre-mds2 (opts:) on oleg107-server 23:53:53 (1735102433) shut down facet: mds2 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds2 to oleg107-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 23:54:07 (1735102447) targets are mounted 23:54:07 (1735102447) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36652 3569480 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13064 3591064 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49716 7160544 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg107-server Stopping /mnt/lustre-mds1 (opts:) on oleg107-server 23:54:19 (1735102459) shut down facet: mds1 facet_host: oleg107-server facet_failover_host: oleg107-server 2.70 MB/sec execute 64 sec latency 9.149 ms oleg107-client.virtnet: 1 49143 2.76 MB/sec execute 65 sec latency 17.250 ms oleg107-client.virtnet: 1 50002 2.81 MB/sec execute 66 sec latency 9.372 ms oleg107-client.virtnet: 1 50560 2.79 MB/sec execute 67 sec latency 9.901 ms oleg107-client.virtnet: 1 51196 2.81 MB/sec execute 68 sec latency 11.171 ms oleg107-client.virtnet: 1 51699 2.78 MB/sec execute 69 sec latency 13.111 ms oleg107-client.virtnet: 1 52287 2.77 MB/sec execute 70 sec latency 11.544 ms oleg107-client.virtnet: 1 53330 2.87 MB/sec execute 71 sec latency 10.842 ms oleg107-client.virtnet: 1 53830 2.85 MB/sec execute 72 sec latency 9.920 ms oleg107-client.virtnet: 1 54451 2.84 MB/sec execute 73 sec latency 11.266 ms oleg107-client.virtnet: 1 54822 2.84 MB/sec execute 74 sec latency 481.644 ms oleg107-client.virtnet: 1 54822 2.80 MB/sec execute 75 sec latency 1481.815 ms oleg107-client.virtnet: 1 54822 2.77 MB/sec execute 76 sec latency 2481.961 ms oleg107-client.virtnet: 1 54822 2.73 MB/sec execute 77 sec latency 3482.108 ms oleg107-client.virtnet: 1 54822 2.70 MB/sec execute 78 sec latency 4482.244 ms oleg107-client.virtnet: 1 54822 2.66 MB/sec execute 79 sec latency 5482.404 ms oleg107-client.virtnet: 1 54822 2.63 MB/sec execute 80 sec latency 6482.648 ms oleg107-client.virtnet: 1 54822 2.60 MB/sec execute 81 sec latency 7482.849 ms oleg107-client.virtnet: 1 54822 2.57 MB/sec execute 82 sec latency 8482.978 ms oleg107-client.virtnet: 1 54822 2.53 MB/sec execute 83 sec latency 9483.090 ms oleg107-client.virtnet: 1 54822 2.50 MB/sec execute 84 sec latency 10483.211 ms oleg107-client.virtnet: 1 54822 2.47 MB/sec execute 85 sec latency 11483.406 ms oleg107-client.virtnet: 1 54822 2.45 MB/sec execute 86 sec latency 12483.589 ms oleg107-client.virtnet: 1 54822 2.42 MB/sec execute 87 sec latency 13483.732 ms oleg107-client.virtnet: 1 54822 2.39 MB/sec execute 88 sec latency 14483.867 ms oleg107-client.virtnet: 1 54822 2.36 MB/sec execute 89 sec latency 15483.982 ms oleg107-client.virtnet: 1 54822 2.34 MB/sec execute 90 sec latency 16484.130 ms oleg107-client.virtnet: 1 54822 2.31 MB/sec execute 91 sec latency 17484.302 ms oleg107-client.virtnet: 1 54822 2.29 MB/sec execute 92 sec latency 18484.431 ms oleg107-client.virtnet: 1 54822 2.26 MB/sec execute 93 sec latency 19484.628 ms oleg107-client.virtnet: 1 54822 2.24 MB/sec execute 94 sec latency 20484.778 ms oleg107-client.virtnet: 1 55304 2.22 MB/sec execute 95 sec latency 20574.837 ms oleg107-client.virtnet: 1 56156 2.27 MB/sec execute 96 sec latency 10.814 ms oleg107-client.virtnet: 1 57036 2.31 MB/sec execute 97 sec latency 10.764 ms oleg107-client.virtnet: 1 57554 2.29 MB/sec execute 98 sec latency 12.325 ms oleg107-client.virtnet: 1 58214 2.32 MB/sec execute 99 sec latency 9.485 ms oleg107-client.virtnet: 1 58760 2.30 MB/sec execute 100 sec latency 10.740 ms oleg107-client.virtnet: 1 59412 2.30 MB/sec execute 101 sec latency 9.782 ms oleg107-client.virtnet: 1 60470 2.37 MB/sec execute 102 sec latency 8.738 ms oleg107-client.virtnet: 1 61035 2.37 MB/sec execute 103 sec latency 11.419 ms oleg107-client.virtnet: 1 61592 2.36 MB/sec execute 104 sec latency 10.170 ms oleg107-client.virtnet: 1 62221 2.37 MB/sec execute 105 sec latency 10.667 ms oleg107-client.virtnet: 1 62931 2.37 MB/sec execute 106 sec latency 11.731 ms oleg107-client.virtnet: 1 63998 2.44 MB/sec execute 107 sec latency 10.771 ms oleg107-client.virtnet: 1 64524 2.43 MB/sec execute 108 sec latency 9.Failover mds1 to oleg107-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 23:54:33 (1735102473) targets are mounted 23:54:33 (1735102473) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3569736 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13096 3590628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7160364 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg107-server Stopping /mnt/lustre-mds2 (opts:) on oleg107-server 23:54:45 (1735102485) shut down facet: mds2 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds2 to oleg107-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 23:54:58 (1735102498) targets are mounted 23:54:58 (1735102498) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36400 3570004 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13096 3592936 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7162940 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg107-server Stopping /mnt/lustre-mds1 (opts:) on oleg107-server 413 ms oleg107-client.virtnet: 1 65066 2.43 MB/sec execute 109 sec latency 17.532 ms oleg107-client.virtnet: 1 65696 2.44 MB/sec execute 110 sec latency 10.622 ms oleg107-client.virtnet: 1 66355 2.43 MB/sec execute 111 sec latency 9.451 ms oleg107-client.virtnet: 1 67493 2.50 MB/sec execute 112 sec latency 9.202 ms oleg107-client.virtnet: 1 67985 2.49 MB/sec execute 113 sec latency 10.036 ms oleg107-client.virtnet: 1 68563 2.49 MB/sec execute 114 sec latency 10.467 ms oleg107-client.virtnet: 1 69149 2.50 MB/sec execute 115 sec latency 12.974 ms oleg107-client.virtnet: 1 69795 2.49 MB/sec execute 116 sec latency 7.497 ms oleg107-client.virtnet: 1 70745 2.53 MB/sec execute 117 sec latency 10.866 ms oleg107-client.virtnet: 1 71325 2.55 MB/sec execute 118 sec latency 10.335 ms oleg107-client.virtnet: 1 71793 2.53 MB/sec execute 119 sec latency 14.161 ms oleg107-client.virtnet: 1 72429 2.55 MB/sec execute 120 sec latency 10.917 ms oleg107-client.virtnet: 1 72988 2.53 MB/sec execute 121 sec latency 9.937 ms oleg107-client.virtnet: 1 73755 2.56 MB/sec execute 122 sec latency 8.934 ms oleg107-client.virtnet: 1 74722 2.59 MB/sec execute 123 sec latency 8.800 ms oleg107-client.virtnet: 1 75268 2.58 MB/sec execute 124 sec latency 9.325 ms oleg107-client.virtnet: 1 75934 2.60 MB/sec execute 125 sec latency 9.615 ms oleg107-client.virtnet: 1 76514 2.58 MB/sec execute 126 sec latency 9.586 ms oleg107-client.virtnet: 1 77150 2.58 MB/sec execute 127 sec latency 11.046 ms oleg107-client.virtnet: 1 77518 2.60 MB/sec execute 128 sec latency 730.050 ms oleg107-client.virtnet: 1 77518 2.58 MB/sec execute 129 sec latency 1730.263 ms oleg107-client.virtnet: 1 77518 2.56 MB/sec execute 130 sec latency 2730.405 ms oleg107-client.virtnet: 1 77518 2.54 MB/sec execute 131 sec latency 3730.599 ms oleg107-client.virtnet: 1 77518 2.52 MB/sec execute 132 sec latency 4730.758 ms oleg107-client.virtnet: 1 77518 2.50 MB/sec execute 133 sec latency 5730.924 ms oleg107-client.virtnet: 1 77518 2.48 MB/sec execute 134 sec latency 6731.244 ms oleg107-client.virtnet: 1 77518 2.46 MB/sec execute 135 sec latency 7731.434 ms oleg107-client.virtnet: 1 77518 2.44 MB/sec execute 136 sec latency 8731.598 ms oleg107-client.virtnet: 1 77518 2.43 MB/sec execute 137 sec latency 9731.744 ms oleg107-client.virtnet: 1 77518 2.41 MB/sec execute 138 sec latency 10731.887 ms oleg107-client.virtnet: 1 77518 2.39 MB/sec execute 139 sec latency 11732.050 ms oleg107-client.virtnet: 1 77518 2.37 MB/sec execute 140 sec latency 12732.201 ms oleg107-client.virtnet: 1 77518 2.36 MB/sec execute 141 sec latency 13732.402 ms oleg107-client.virtnet: 1 77518 2.34 MB/sec execute 142 sec latency 14732.550 ms oleg107-client.virtnet: 1 77518 2.32 MB/sec execute 143 sec latency 15732.689 ms oleg107-client.virtnet: 1 77518 2.31 MB/sec execute 144 sec latency 16732.872 ms oleg107-client.virtnet: 1 77518 2.29 MB/sec execute 145 sec latency 17733.044 ms oleg107-client.virtnet: 1 77779 2.28 MB/sec execute 146 sec latency 18479.352 ms oleg107-client.virtnet: 1 78431 2.31 MB/sec execute 147 sec latency 10.453 ms oleg107-client.virtnet: 1 78881 2.29 MB/sec execute 148 sec latency 15.324 ms oleg107-client.virtnet: 1 79471 2.31 MB/sec execute 149 sec latency 12.245 ms oleg107-client.virtnet: 1 80008 2.30 MB/sec execute 150 sec latency 11.957 ms oleg107-client.virtnet: 1 80719 2.30 MB/sec execute 151 sec latency 9.304 ms oleg107-client.virtnet: 1 81827 2.35 MB/sec execute 152 sec latency 9.094 ms oleg107-client.virtnet: 1 82346 2.34 MB23:55:10 (1735102510) shut down facet: mds1 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds1 to oleg107-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 23:55:24 (1735102524) targets are mounted 23:55:24 (1735102524) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3570104 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13180 3591952 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49584 7162056 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg107-server Stopping /mnt/lustre-mds2 (opts:) on oleg107-server 23:55:36 (1735102536) shut down facet: mds2 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds2 to oleg107-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 23:55:50 (1735102550) targets are mounted 23:55:50 (1735102550) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid /sec execute 153 sec latency 11.453 ms oleg107-client.virtnet: 1 82886 2.34 MB/sec execute 154 sec latency 9.740 ms oleg107-client.virtnet: 1 83489 2.35 MB/sec execute 155 sec latency 9.618 ms oleg107-client.virtnet: 1 84198 2.35 MB/sec execute 156 sec latency 9.640 ms oleg107-client.virtnet: 1 85286 2.39 MB/sec execute 157 sec latency 8.974 ms oleg107-client.virtnet: 1 85843 2.39 MB/sec execute 158 sec latency 8.243 ms oleg107-client.virtnet: 1 86401 2.39 MB/sec execute 159 sec latency 10.430 ms oleg107-client.virtnet: 1 87020 2.39 MB/sec execute 160 sec latency 10.303 ms oleg107-client.virtnet: 1 87667 2.39 MB/sec execute 161 sec latency 9.687 ms oleg107-client.virtnet: 1 88792 2.44 MB/sec execute 162 sec latency 11.097 ms oleg107-client.virtnet: 1 89251 2.43 MB/sec execute 163 sec latency 10.312 ms oleg107-client.virtnet: 1 89837 2.43 MB/sec execute 164 sec latency 11.334 ms oleg107-client.virtnet: 1 90441 2.43 MB/sec execute 165 sec latency 11.300 ms oleg107-client.virtnet: 1 91117 2.43 MB/sec execute 166 sec latency 8.468 ms oleg107-client.virtnet: 1 92117 2.46 MB/sec execute 167 sec latency 12.472 ms oleg107-client.virtnet: 1 92725 2.47 MB/sec execute 168 sec latency 10.171 ms oleg107-client.virtnet: 1 93332 2.47 MB/sec execute 169 sec latency 9.882 ms oleg107-client.virtnet: 1 93967 2.48 MB/sec execute 170 sec latency 10.646 ms oleg107-client.virtnet: 1 94624 2.47 MB/sec execute 171 sec latency 8.351 ms oleg107-client.virtnet: 1 95540 2.50 MB/sec execute 172 sec latency 9.963 ms oleg107-client.virtnet: 1 96178 2.51 MB/sec execute 173 sec latency 13.306 ms oleg107-client.virtnet: 1 96736 2.50 MB/sec execute 174 sec latency 12.053 ms oleg107-client.virtnet: 1 97390 2.51 MB/sec execute 175 sec latency 10.635 ms oleg107-client.virtnet: 1 97997 2.50 MB/sec execute 176 sec latency 9.579 ms oleg107-client.virtnet: 1 98381 2.50 MB/sec execute 177 sec latency 466.490 ms oleg107-client.virtnet: 1 99460 2.54 MB/sec execute 178 sec latency 8.862 ms oleg107-client.virtnet: 1 99899 2.54 MB/sec execute 179 sec latency 96.088 ms oleg107-client.virtnet: 1 99899 2.52 MB/sec execute 180 sec latency 1096.269 ms oleg107-client.virtnet: 1 99899 2.51 MB/sec execute 181 sec latency 2096.452 ms oleg107-client.virtnet: 1 99899 2.49 MB/sec execute 182 sec latency 3096.668 ms oleg107-client.virtnet: 1 99899 2.48 MB/sec execute 183 sec latency 4096.837 ms oleg107-client.virtnet: 1 99899 2.47 MB/sec execute 184 sec latency 5097.011 ms oleg107-client.virtnet: 1 99899 2.45 MB/sec execute 185 sec latency 6097.167 ms oleg107-client.virtnet: 1 99899 2.44 MB/sec execute 186 sec latency 7097.337 ms oleg107-client.virtnet: 1 99899 2.43 MB/sec execute 187 sec latency 8097.528 ms oleg107-client.virtnet: 1 99899 2.41 MB/sec execute 188 sec latency 9097.731 ms oleg107-client.virtnet: 1 99899 2.40 MB/sec execute 189 sec latency 10097.931 ms oleg107-client.virtnet: 1 99899 2.39 MB/sec execute 190 sec latency 11098.156 ms oleg107-client.virtnet: 1 99899 2.38 MB/sec execute 191 sec latency 12098.325 ms oleg107-client.virtnet: 1 99899 2.36 MB/sec execute 192 sec latency 13098.584 ms oleg107-client.virtnet: 1 99899 2.35 MB/sec execute 193 sec latency 14098.766 ms oleg107-client.virtnet: 1 99899 2.34 MB/sec execute 194 sec latency 15098.936 ms oleg107-client.virtnet: 1 99899 2.33 MB/sec execute 195 sec latency 16099.102 ms oleg107-client.virtnet: 1 99899 2.32 MB/sec execute 196 sec latency 17099.274 ms oleg107-client.virtnet: 1 99899 2.30 MB/sec execute 197 sec latency 18099.479 ms oleg107-client.vimdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36540 3569880 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13156 3592100 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49696 7161980 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg107-server Stopping /mnt/lustre-mds1 (opts:) on oleg107-server 23:56:02 (1735102562) shut down facet: mds1 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds1 to oleg107-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 23:56:16 (1735102576) targets are mounted 23:56:16 (1735102576) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36408 3570044 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13092 3592872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162916 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg107-server Stopping /mnt/lustre-mds2 (opts:) on oleg107-server 23:56:28 (1735102588) shut down facet: mds2 facet_host: oleg107-server facet_failover_host: oleg107-server Failover mds2 to oleg107-server mount facets: mds2 rtnet: 1 100162 2.29 MB/sec execute 198 sec latency 18545.080 ms oleg107-client.virtnet: 1 100686 2.30 MB/sec execute 199 sec latency 12.659 ms oleg107-client.virtnet: 1 101328 2.30 MB/sec execute 200 sec latency 11.502 ms oleg107-client.virtnet: 1 101946 2.30 MB/sec execute 201 sec latency 11.223 ms oleg107-client.virtnet: 1 102999 2.33 MB/sec execute 202 sec latency 11.165 ms oleg107-client.virtnet: 1 103538 2.33 MB/sec execute 203 sec latency 9.831 ms oleg107-client.virtnet: 1 104130 2.33 MB/sec execute 204 sec latency 12.934 ms oleg107-client.virtnet: 1 104748 2.33 MB/sec execute 205 sec latency 9.573 ms oleg107-client.virtnet: 1 105408 2.33 MB/sec execute 206 sec latency 9.805 ms oleg107-client.virtnet: 1 106542 2.37 MB/sec execute 207 sec latency 9.423 ms oleg107-client.virtnet: 1 107053 2.37 MB/sec execute 208 sec latency 10.745 ms oleg107-client.virtnet: 1 107655 2.36 MB/sec execute 209 sec latency 10.401 ms oleg107-client.virtnet: 1 108305 2.37 MB/sec execute 210 sec latency 9.677 ms oleg107-client.virtnet: 1 109027 2.37 MB/sec execute 211 sec latency 9.445 ms oleg107-client.virtnet: 1 110115 2.40 MB/sec execute 212 sec latency 10.786 ms oleg107-client.virtnet: 1 110662 2.40 MB/sec execute 213 sec latency 9.258 ms oleg107-client.virtnet: 1 111238 2.40 MB/sec execute 214 sec latency 11.493 ms oleg107-client.virtnet: 1 111849 2.40 MB/sec execute 215 sec latency 10.620 ms oleg107-client.virtnet: 1 112565 2.40 MB/sec execute 216 sec latency 9.037 ms oleg107-client.virtnet: 1 113631 2.44 MB/sec execute 217 sec latency 10.494 ms oleg107-client.virtnet: 1 114108 2.43 MB/sec execute 218 sec latency 11.287 ms oleg107-client.virtnet: 1 114739 2.43 MB/sec execute 219 sec latency 10.977 ms oleg107-client.virtnet: 1 115382 2.43 MB/sec execute 220 sec latency 9.639 ms oleg107-client.virtnet: 1 116034 2.43 MB/sec execute 221 sec latency 10.639 ms oleg107-client.virtnet: 1 117120 2.47 MB/sec execute 222 sec latency 10.920 ms oleg107-client.virtnet: 1 117607 2.46 MB/sec execute 223 sec latency 9.870 ms oleg107-client.virtnet: 1 118180 2.46 MB/sec execute 224 sec latency 12.881 ms oleg107-client.virtnet: 1 118783 2.46 MB/sec execute 225 sec latency 10.334 ms oleg107-client.virtnet: 1 119468 2.46 MB/sec execute 226 sec latency 8.699 ms oleg107-client.virtnet: 1 120519 2.49 MB/sec execute 227 sec latency 10.244 ms oleg107-client.virtnet: 1 121100 2.49 MB/sec execute 228 sec latency 9.413 ms oleg107-client.virtnet: 1 121557 2.49 MB/sec execute 229 sec latency 172.984 ms oleg107-client.virtnet: 1 122197 2.49 MB/sec execute 230 sec latency 9.723 ms oleg107-client.virtnet: 1 122506 2.48 MB/sec execute 231 sec latency 285.091 ms oleg107-client.virtnet: 1 122506 2.47 MB/sec execute 232 sec latency 1285.323 ms oleg107-client.virtnet: 1 122506 2.46 MB/sec execute 233 sec latency 2285.556 ms oleg107-client.virtnet: 1 122506 2.45 MB/sec execute 234 sec latency 3285.743 ms oleg107-client.virtnet: 1 122506 2.44 MB/sec execute 235 sec latency 4285.909 ms oleg107-client.virtnet: 1 122506 2.43 MB/sec execute 236 sec latency 5286.086 ms oleg107-client.virtnet: 1 122506 2.42 MB/sec execute 237 sec latency 6286.331 ms oleg107-client.virtnet: 1 122506 2.41 MB/sec execute 238 sec latency 7286.533 ms oleg107-client.virtnet: 1 122506 2.40 MB/sec execute 239 sec latency 8286.706 ms oleg107-client.virtnet: 1 122506 2.39 MB/sec execute 240 sec latency 9286.856 ms oleg107-client.virtnet: 1 122506 2.38 MB/sec execute 241 sec latency 10287.024 ms oleg107-client.virtnet: 1 122506 2.37 MB/sec execute 242 sec latency 11287.149 ms olStarting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 23:56:42 (1735102602) targets are mounted 23:56:42 (1735102602) facet_failover done oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec eg107-client.virtnet: 1 122506 2.36 MB/sec execute 243 sec latency 12287.350 ms oleg107-client.virtnet: 1 122506 2.35 MB/sec execute 244 sec latency 13287.543 ms oleg107-client.virtnet: 1 122506 2.34 MB/sec execute 245 sec latency 14287.678 ms oleg107-client.virtnet: 1 122506 2.33 MB/sec execute 246 sec latency 15287.863 ms oleg107-client.virtnet: 1 122506 2.32 MB/sec execute 247 sec latency 16288.022 ms oleg107-client.virtnet: 1 122506 2.31 MB/sec execute 248 sec latency 17288.177 ms oleg107-client.virtnet: 1 122506 2.30 MB/sec execute 249 sec latency 18288.298 ms oleg107-client.virtnet: 1 122922 2.30 MB/sec execute 250 sec latency 18567.215 ms oleg107-client.virtnet: 1 123840 2.32 MB/sec execute 251 sec latency 11.271 ms oleg107-client.virtnet: 1 124531 2.33 MB/sec execute 252 sec latency 9.920 ms oleg107-client.virtnet: 1 125062 2.33 MB/sec execute 253 sec latency 12.460 ms oleg107-client.virtnet: 1 125648 2.33 MB/sec execute 254 sec latency 12.033 ms oleg107-client.virtnet: 1 126207 2.33 MB/sec execute 255 sec latency 12.153 ms oleg107-client.virtnet: 1 127077 2.34 MB/sec execute 256 sec latency 9.493 ms oleg107-client.virtnet: 1 127978 2.36 MB/sec execute 257 sec latency 9.623 ms oleg107-client.virtnet: 1 128476 2.35 MB/sec execute 258 sec latency 11.517 ms oleg107-client.virtnet: 1 129030 2.35 MB/sec execute 259 sec latency 10.650 ms oleg107-client.virtnet: 1 129604 2.35 MB/sec execute 260 sec latency 10.181 ms oleg107-client.virtnet: 1 130225 2.35 MB/sec execute 261 sec latency 9.565 ms oleg107-client.virtnet: 1 131336 2.38 MB/sec execute 262 sec latency 11.043 ms oleg107-client.virtnet: 1 131829 2.38 MB/sec execute 263 sec latency 11.661 ms oleg107-client.virtnet: 1 132361 2.38 MB/sec execute 264 sec latency 13.899 ms oleg107-client.virtnet: 1 132993 2.38 MB/sec execute 265 sec latency 11.064 ms oleg107-client.virtnet: 1 133668 2.38 MB/sec execute 266 sec latency 8.801 ms oleg107-client.virtnet: 1 134764 2.41 MB/sec execute 267 sec latency 10.869 ms oleg107-client.virtnet: 1 135293 2.41 MB/sec execute 268 sec latency 9.027 ms oleg107-client.virtnet: 1 135870 2.40 MB/sec execute 269 sec latency 12.951 ms oleg107-client.virtnet: 1 136503 2.41 MB/sec execute 270 sec latency 9.979 ms oleg107-client.virtnet: 1 137169 2.41 MB/sec execute 271 sec latency 13.183 ms oleg107-client.virtnet: 1 138160 2.42 MB/sec execute 272 sec latency 17.205 ms oleg107-client.virtnet: 1 138788 2.43 MB/sec execute 273 sec latency 11.502 ms oleg107-client.virtnet: 1 139332 2.43 MB/sec execute 274 sec latency 19.939 ms oleg107-client.virtnet: 1 139982 2.43 MB/sec execute 275 sec latency 18.900 ms oleg107-client.virtnet: 1 140665 2.43 MB/sec execute 276 sec latency 10.308 ms oleg107-client.virtnet: 1 141704 2.45 MB/sec execute 277 sec latency 9.190 ms oleg107-client.virtnet: 1 142333 2.46 MB/sec execute 278 sec latency 8.909 ms oleg107-client.virtnet: 1 142907 2.45 MB/sec execute 279 sec latency 11.157 ms oleg107-client.virtnet: 1 143579 2.46 MB/sec execute 280 sec latency 14.010 ms oleg107-client.virtnet: 1 144270 2.46 MB/sec execute 281 sec latency 9.465 ms oleg107-client.virtnet: 1 145277 2.47 MB/sec execute 282 sec latency 11.376 ms oleg107-client.virtnet: 1 145891 2.48 MB/sec execute 283 sec latency 8.556 ms oleg107-client.virtnet: 1 146460 2.48 MB/sec execute 284 sec latency 11.605 ms oleg107-client.virtnet: 1 147130 2.48 MB/sec execute 285 sec latency 8.809 ms oleg107-client.virtnet: 1 147820 2.48 MB/sec execute 286 sec latency 9.239 ms oleg107-client.virtnet: 1 148887 2.50 MB/sec execute 287 sec latency 8.871 ms oleg107-client.virtnet: 1 149482 2.50 MB/sec execute 288 sec latency 9.541 ms oleg107-client.virtnet: 1 150089 2.50 MB/sec execute 289 sec latency 12.977 ms oleg107-client.virtnet: 1 150715 2.51 MB/sec execute 290 sec latency 10.613 ms oleg107-client.virtnet: 1 151341 2.50 MB/sec execute 291 sec latency 9.512 ms oleg107-client.virtnet: 1 152206 2.52 MB/sec execute 292 sec latency 10.477 ms oleg107-client.virtnet: 1 152893 2.53 MB/sec execute 293 sec latency 9.314 ms oleg107-client.virtnet: 1 153443 2.52 MB/sec execute 294 sec latency 12.687 ms oleg107-client.virtnet: 1 154109 2.53 MB/sec execute 295 sec latency 10.236 ms oleg107-client.virtnet: 1 154705 2.52 MB/sec execute 296 sec latency 10.376 ms oleg107-client.virtnet: 1 155539 2.54 MB/sec execute 297 sec latency 9.097 ms oleg107-client.virtnet: 1 156335 2.55 MB/sec execute 298 sec latency 12.631 ms oleg107-client.virtnet: 1 156871 2.54 MB/sec execute 299 sec latency 13.216 ms oleg107-client.virtnet: 1 cleanup 300 sec oleg107-client.virtnet: 0 cleanup 300 sec oleg107-client.virtnet: oleg107-client.virtnet: Operation Count AvgLat MaxLat oleg107-client.virtnet: ---------------------------------------- oleg107-client.virtnet: NTCreateX 24495 6.220 20574.824 oleg107-client.virtnet: Close 17995 1.872 18346.025 oleg107-client.virtnet: Rename 1040 7.129 14.847 oleg107-client.virtnet: Unlink 4956 5.433 18545.063 oleg107-client.virtnet: Qpathinfo 22220 2.115 18479.336 oleg107-client.virtnet: Qfileinfo 3873 0.138 1.695 oleg107-client.virtnet: Qfsinfo 4080 0.205 15.139 oleg107-client.virtnet: Sfileinfo 2000 4.121 12.015 oleg107-client.virtnet: Find 8590 0.366 10.726 oleg107-client.virtnet: WriteX 12111 0.761 6.822 oleg107-client.virtnet: ReadX 38480 0.020 3.549 oleg107-client.virtnet: LockX 80 0.864 1.255 oleg107-client.virtnet: UnlockX 80 0.860 1.656 oleg107-client.virtnet: Flush 1719 5.031 466.478 oleg107-client.virtnet: oleg107-client.virtnet: Throughput 2.54207 MB/sec 1 clients 1 procs max_latency=20574.837 ms oleg107-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg107-client.virtnet at Tue Dec 24 23:57:37 EST 2024 with return code 0 oleg107-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg107-client.virtnet oleg107-client.virtnet: /mnt/lustre/d70b.replay-single/oleg107-client.virtnet /mnt/lustre/d70b.replay-single/oleg107-client.virtnet oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg107-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg107-client.virtnet: removed directory: 'clients/client0' oleg107-client.virtnet: removed directory: 'clients' oleg107-client.virtnet: removed 'client.txt' oleg107-client.virtnet: /mnt/lustre/d70b.replay-single/oleg107-client.virtnet oleg107-client.virtnet: dbench successfully finished