== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 23:23:17 (1746069797) Starting client oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre Started clients oleg223-client.virtnet: 192.168.202.123@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname dbench: no process found + DIR=/mnt/lustre/d70b.replay-single/oleg223-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 Started rundbench load pid=6355 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3840 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3722240 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7458816 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:23:27 (1746069807) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:23:41 (1746069821) targets are mounted 23:23:41 (1746069821) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 32768 3678208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 11264 3719168 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7397376 1% /mnt/lustre oleg223-client.virtnet: looking for dbench program oleg223-client.virtnet: /usr/bin/dbench oleg223-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg223-client.virtnet oleg223-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg223-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg223-client.virtnet' oleg223-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg223-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg223-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg223-client.virtnet at Wed Apr 30 23:23:19 EDT 2025 oleg223-client.virtnet: waiting for dbench pid 6379 oleg223-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg223-client.virtnet: oleg223-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg223-client.virtnet: failed to create barrier semaphore oleg223-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg223-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg223-client.virtnet: releasing clients oleg223-client.virtnet: 1 129 5.76 MB/sec warmup 1 sec latency 104.568 ms oleg223-client.virtnet: 1 304 5.61 MB/sec warmup 2 sec latency 40.913 ms oleg223-client.virtnet: 1 422 4.95 MB/sec warmup 3 sec latency 47.851 ms oleg223-client.virtnet: 1 652 5.26 MB/sec warmup 4 sec latency 54.600 ms oleg223-client.virtnet: 1 730 4.26 MB/sec warmup 5 sec latency 172.439 ms oleg223-client.virtnet: 1 811 3.58 MB/sec warmup 6 sec latency 133.125 ms oleg223-client.virtnet: 1 913 3.11 MB/sec warmup 7 sec latency 160.237 ms oleg223-client.virtnet: 1 913 2.72 MB/sec warmup 8 sec latency 1160.401 ms oleg223-client.virtnet: 1 913 2.42 MB/sec warmup 9 sec latency 2160.589 ms oleg223-client.virtnet: 1 913 2.18 MB/sec warmup 10 sec latency 3160.889 ms oleg223-client.virtnet: 1 913 1.98 MB/sec warmup 11 sec latency 4161.705 ms oleg223-client.virtnet: 1 913 1.81 MB/sec warmup 12 sec latency 5162.299 ms oleg223-client.virtnet: 1 913 1.67 MB/sec warmup 13 sec latency 6162.465 ms oleg223-client.virtnet: 1 913 1.55 MB/sec warmup 14 sec latency 7162.793 ms oleg223-client.virtnet: 1 913 1.45 MB/sec warmup 15 sec latency 8163.109 ms oleg223-client.virtnet: 1 913 1.36 MB/sec warmup 16 sec latency 9164.894 ms oleg223-client.virtnet: 1 913 1.28 MB/sec warmup 17 sec latency 10166.743 ms oleg223-client.virtnet: 1 913 1.21 MB/sec warmup 18 sec latency 11166.919 ms oleg223-client.virtnet: 1 913 1.15 MB/sec warmup 19 sec latency 12167.411 ms oleg223-client.virtnet: 1 913 1.09 MB/sec warmup 20 sec latency 13169.880 ms oleg223-client.virtnet: 1 913 1.04 MB/sec warmup 21 sec latency 14170.708 ms oleg223-client.virtnet: 1 913 0.99 MB/sec warmup 22 sec latency 15171.010 ms oleg223-client.virtnet: 1 913 0.95 MB/sec warmup 23 sec latency 16171.227 ms oleg223-client.virtnet: 1 913 0.91 MB/sec warmup 24 sec latency 17171.504 ms oleg223-client.virtnet: 1 913 0.87 MB/sec warmup 25 sec latency 18171.713 ms oleg223-client.virtnet: 1 981 0.84 MB/sec warmup 26 sec latency 18605.335 ms oleg223-client.virtnet: 1 1172 0.85 MB/sec warmup 27 sec latency 58.945 ms oleg223-client.virtnet: 1 1426 0.93 MB/sec warmup 28 sec latency 38.135 ms oleg223-client.virtnet: 1 1625 0.90 MB/sec warmup 29 sec latency 87.580 ms oleg223-client.virtnet: 1 1849 0.88 MB/sec warmup 30 sec latency 58.296 ms oleg223-client.virtnet: 1 2012 0.86 MB/sec warmup 31 sec latency 95.386 ms oleg223-client.virtnet: 1 2198 0.84 MB/sec warmup 32 sec latency 39.489 test_70b fail mds1 2 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:23:54 (1746069834) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:24:09 (1746069849) targets are mounted 23:24:09 (1746069849) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3710976 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3747840 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7458816 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:24:21 (1746069861) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:24:36 (1746069876) targets are mounted 23:24:36 (1746069876) facet_failover done ms oleg223-client.virtnet: 1 2416 0.85 MB/sec warmup 33 sec latency 74.055 ms oleg223-client.virtnet: 1 2582 0.85 MB/sec warmup 34 sec latency 109.323 ms oleg223-client.virtnet: 1 2596 0.83 MB/sec warmup 35 sec latency 893.763 ms oleg223-client.virtnet: 1 2596 0.81 MB/sec warmup 36 sec latency 1894.960 ms oleg223-client.virtnet: 1 2596 0.78 MB/sec warmup 37 sec latency 2895.123 ms oleg223-client.virtnet: 1 2596 0.76 MB/sec warmup 38 sec latency 3896.348 ms oleg223-client.virtnet: 1 2596 0.74 MB/sec warmup 39 sec latency 4896.512 ms oleg223-client.virtnet: 1 2596 0.72 MB/sec warmup 40 sec latency 5897.349 ms oleg223-client.virtnet: 1 2596 0.71 MB/sec warmup 41 sec latency 6897.528 ms oleg223-client.virtnet: 1 2596 0.69 MB/sec warmup 42 sec latency 7897.692 ms oleg223-client.virtnet: 1 2596 0.67 MB/sec warmup 43 sec latency 8897.881 ms oleg223-client.virtnet: 1 2596 0.66 MB/sec warmup 44 sec latency 9900.570 ms oleg223-client.virtnet: 1 2596 0.64 MB/sec warmup 45 sec latency 10900.701 ms oleg223-client.virtnet: 1 2596 0.63 MB/sec warmup 46 sec latency 11901.285 ms oleg223-client.virtnet: 1 2596 0.62 MB/sec warmup 47 sec latency 12902.497 ms oleg223-client.virtnet: 1 2596 0.60 MB/sec warmup 48 sec latency 13902.758 ms oleg223-client.virtnet: 1 2596 0.59 MB/sec warmup 49 sec latency 14905.212 ms oleg223-client.virtnet: 1 2596 0.58 MB/sec warmup 50 sec latency 15905.385 ms oleg223-client.virtnet: 1 2764 0.63 MB/sec warmup 51 sec latency 16530.081 ms oleg223-client.virtnet: 1 2982 0.64 MB/sec warmup 52 sec latency 128.463 ms oleg223-client.virtnet: 1 3209 0.65 MB/sec warmup 53 sec latency 113.688 ms oleg223-client.virtnet: 1 3493 0.71 MB/sec warmup 54 sec latency 50.952 ms oleg223-client.virtnet: 1 3678 0.70 MB/sec warmup 55 sec latency 32.946 ms oleg223-client.virtnet: 1 3844 0.71 MB/sec warmup 56 sec latency 35.949 ms oleg223-client.virtnet: 1 3978 0.70 MB/sec warmup 57 sec latency 36.709 ms oleg223-client.virtnet: 1 4055 0.69 MB/sec warmup 58 sec latency 196.566 ms oleg223-client.virtnet: 1 4236 0.68 MB/sec warmup 59 sec latency 80.821 ms oleg223-client.virtnet: 1 4395 0.17 MB/sec execute 1 sec latency 108.738 ms oleg223-client.virtnet: 1 4434 0.13 MB/sec execute 2 sec latency 820.278 ms oleg223-client.virtnet: 1 4434 0.09 MB/sec execute 3 sec latency 1821.034 ms oleg223-client.virtnet: 1 4434 0.06 MB/sec execute 4 sec latency 2821.423 ms oleg223-client.virtnet: 1 4434 0.05 MB/sec execute 5 sec latency 3821.586 ms oleg223-client.virtnet: 1 4434 0.04 MB/sec execute 6 sec latency 4822.077 ms oleg223-client.virtnet: 1 4434 0.04 MB/sec execute 7 sec latency 5822.239 ms oleg223-client.virtnet: 1 4434 0.03 MB/sec execute 8 sec latency 6822.405 ms oleg223-client.virtnet: 1 4434 0.03 MB/sec execute 9 sec latency 7822.573 ms oleg223-client.virtnet: 1 4434 0.03 MB/sec execute 10 sec latency 8823.861 ms oleg223-client.virtnet: 1 4434 0.02 MB/sec execute 11 sec latency 9824.044 ms oleg223-client.virtnet: 1 4434 0.02 MB/sec execute 12 sec latency 10824.207 ms oleg223-client.virtnet: 1 4434 0.02 MB/sec execute 13 sec latency 11824.835 ms oleg223-client.virtnet: 1 4434 0.02 MB/sec execute 14 sec latency 12825.471 ms oleg223-client.virtnet: 1 4434 0.02 MB/sec execute 15 sec latency 13825.619 ms oleg223-client.virtnet: 1 4434 0.02 MB/sec execute 16 sec latency 14828.003 ms oleg223-client.virtnet: 1 4434 0.02 MB/sec execute 17 sec latency 15829.427 ms oleg223-client.virtnet: 1 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3724288 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3747840 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7472128 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:24:50 (1746069890) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:25:05 (1746069905) targets are mounted 23:25:05 (1746069905) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3710976 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3725312 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7436288 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:25:20 (1746069920) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server 4434 0.01 MB/sec execute 18 sec latency 16829.741 ms oleg223-client.virtnet: 1 4434 0.01 MB/sec execute 19 sec latency 17831.977 ms oleg223-client.virtnet: 1 4434 0.01 MB/sec execute 20 sec latency 18833.266 ms oleg223-client.virtnet: 1 4443 0.02 MB/sec execute 21 sec latency 19633.612 ms oleg223-client.virtnet: 1 4503 0.02 MB/sec execute 22 sec latency 126.758 ms oleg223-client.virtnet: 1 4578 0.04 MB/sec execute 23 sec latency 214.289 ms oleg223-client.virtnet: 1 4746 0.07 MB/sec execute 24 sec latency 60.849 ms oleg223-client.virtnet: 1 4969 0.19 MB/sec execute 25 sec latency 37.276 ms oleg223-client.virtnet: 1 5131 0.18 MB/sec execute 26 sec latency 109.392 ms oleg223-client.virtnet: 1 5249 0.18 MB/sec execute 27 sec latency 82.386 ms oleg223-client.virtnet: 1 5379 0.18 MB/sec execute 28 sec latency 32.348 ms oleg223-client.virtnet: 1 5571 0.18 MB/sec execute 29 sec latency 72.267 ms oleg223-client.virtnet: 1 5732 0.18 MB/sec execute 30 sec latency 39.333 ms oleg223-client.virtnet: 1 5855 0.21 MB/sec execute 31 sec latency 551.410 ms oleg223-client.virtnet: 1 5855 0.20 MB/sec execute 32 sec latency 1551.616 ms oleg223-client.virtnet: 1 5855 0.20 MB/sec execute 33 sec latency 2551.786 ms oleg223-client.virtnet: 1 5855 0.19 MB/sec execute 34 sec latency 3552.176 ms oleg223-client.virtnet: 1 5855 0.18 MB/sec execute 35 sec latency 4552.470 ms oleg223-client.virtnet: 1 5855 0.18 MB/sec execute 36 sec latency 5552.821 ms oleg223-client.virtnet: 1 5855 0.17 MB/sec execute 37 sec latency 6553.005 ms oleg223-client.virtnet: 1 5855 0.17 MB/sec execute 38 sec latency 7553.238 ms oleg223-client.virtnet: 1 5855 0.17 MB/sec execute 39 sec latency 8554.029 ms oleg223-client.virtnet: 1 5855 0.16 MB/sec execute 40 sec latency 9554.208 ms oleg223-client.virtnet: 1 5855 0.16 MB/sec execute 41 sec latency 10554.556 ms oleg223-client.virtnet: 1 5855 0.15 MB/sec execute 42 sec latency 11555.023 ms oleg223-client.virtnet: 1 5855 0.15 MB/sec execute 43 sec latency 12555.334 ms oleg223-client.virtnet: 1 5855 0.15 MB/sec execute 44 sec latency 13555.554 ms oleg223-client.virtnet: 1 5855 0.14 MB/sec execute 45 sec latency 14555.787 ms oleg223-client.virtnet: 1 5855 0.14 MB/sec execute 46 sec latency 15555.955 ms oleg223-client.virtnet: 1 5855 0.14 MB/sec execute 47 sec latency 16556.772 ms oleg223-client.virtnet: 1 5855 0.13 MB/sec execute 48 sec latency 17561.891 ms oleg223-client.virtnet: 1 5855 0.13 MB/sec execute 49 sec latency 18562.045 ms oleg223-client.virtnet: 1 5855 0.13 MB/sec execute 50 sec latency 19562.324 ms oleg223-client.virtnet: 1 5936 0.13 MB/sec execute 51 sec latency 20002.669 ms oleg223-client.virtnet: 1 6026 0.14 MB/sec execute 52 sec latency 112.295 ms oleg223-client.virtnet: 1 6180 0.15 MB/sec execute 53 sec latency 119.328 ms oleg223-client.virtnet: 1 6544 0.23 MB/sec execute 54 sec latency 93.171 ms oleg223-client.virtnet: 1 6894 0.26 MB/sec execute 55 sec latency 96.151 ms oleg223-client.virtnet: 1 7122 0.31 MB/sec execute 56 sec latency 55.983 ms oleg223-client.virtnet: 1 7269 0.31 MB/sec execute 57 sec latency 44.353 ms oleg223-client.virtnet: 1 7441 0.32 MB/sec execute 58 sec latency 36.806 ms oleg223-client.virtnet: 1 7547 0.32 MB/sec execute 59 sec latency 46.999 ms oleg223-client.virtnet: 1 7642 0.31 MB/sec execute 60 sec latency 102.121 ms oleg223-client.virtnet: 1 7642 0.31 MB/sec execute 61 sec latency 1102.849 ms oleg223-client.virtnet: 1 7642 0.30 MB/secFailover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:25:36 (1746069936) targets are mounted 23:25:36 (1746069936) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3716096 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7460864 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:25:50 (1746069950) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 execute 62 sec latency 2103.447 ms oleg223-client.virtnet: 1 7642 0.30 MB/sec execute 63 sec latency 3103.676 ms oleg223-client.virtnet: 1 7642 0.29 MB/sec execute 64 sec latency 4104.449 ms oleg223-client.virtnet: 1 7642 0.29 MB/sec execute 65 sec latency 5105.212 ms oleg223-client.virtnet: 1 7642 0.28 MB/sec execute 66 sec latency 6105.558 ms oleg223-client.virtnet: 1 7642 0.28 MB/sec execute 67 sec latency 7105.722 ms oleg223-client.virtnet: 1 7642 0.28 MB/sec execute 68 sec latency 8106.325 ms oleg223-client.virtnet: 1 7642 0.27 MB/sec execute 69 sec latency 9106.859 ms oleg223-client.virtnet: 1 7642 0.27 MB/sec execute 70 sec latency 10107.029 ms oleg223-client.virtnet: 1 7642 0.26 MB/sec execute 71 sec latency 11107.531 ms oleg223-client.virtnet: 1 7642 0.26 MB/sec execute 72 sec latency 12107.689 ms oleg223-client.virtnet: 1 7642 0.26 MB/sec execute 73 sec latency 13107.910 ms oleg223-client.virtnet: 1 7642 0.25 MB/sec execute 74 sec latency 14108.084 ms oleg223-client.virtnet: 1 7642 0.25 MB/sec execute 75 sec latency 15108.679 ms oleg223-client.virtnet: 1 7642 0.25 MB/sec execute 76 sec latency 16108.991 ms oleg223-client.virtnet: 1 7704 0.24 MB/sec execute 77 sec latency 16488.096 ms oleg223-client.virtnet: 1 7800 0.24 MB/sec execute 78 sec latency 200.629 ms oleg223-client.virtnet: 1 7902 0.24 MB/sec execute 79 sec latency 107.997 ms oleg223-client.virtnet: 1 7973 0.24 MB/sec execute 80 sec latency 92.555 ms oleg223-client.virtnet: 1 8025 0.24 MB/sec execute 81 sec latency 159.875 ms oleg223-client.virtnet: 1 8101 0.24 MB/sec execute 82 sec latency 201.258 ms oleg223-client.virtnet: 1 8265 0.25 MB/sec execute 83 sec latency 52.316 ms oleg223-client.virtnet: 1 8405 0.25 MB/sec execute 84 sec latency 45.505 ms oleg223-client.virtnet: 1 8591 0.28 MB/sec execute 85 sec latency 104.795 ms oleg223-client.virtnet: 1 8686 0.28 MB/sec execute 86 sec latency 53.322 ms oleg223-client.virtnet: 1 8799 0.28 MB/sec execute 87 sec latency 85.761 ms oleg223-client.virtnet: 1 8899 0.27 MB/sec execute 88 sec latency 60.067 ms oleg223-client.virtnet: 1 8993 0.27 MB/sec execute 89 sec latency 56.591 ms oleg223-client.virtnet: 1 9017 0.27 MB/sec execute 90 sec latency 628.556 ms oleg223-client.virtnet: 1 9017 0.27 MB/sec execute 91 sec latency 1628.869 ms oleg223-client.virtnet: 1 9017 0.26 MB/sec execute 92 sec latency 2629.013 ms oleg223-client.virtnet: 1 9017 0.26 MB/sec execute 93 sec latency 3629.457 ms oleg223-client.virtnet: 1 9017 0.26 MB/sec execute 94 sec latency 4630.082 ms oleg223-client.virtnet: 1 9017 0.26 MB/sec execute 95 sec latency 5635.706 ms oleg223-client.virtnet: 1 9017 0.25 MB/sec execute 96 sec latency 6636.075 ms oleg223-client.virtnet: 1 9017 0.25 MB/sec execute 97 sec latency 7637.186 ms oleg223-client.virtnet: 1 9017 0.25 MB/sec execute 98 sec latency 8637.348 ms oleg223-client.virtnet: 1 9017 0.24 MB/sec execute 99 sec latency 9637.507 ms oleg223-client.virtnet: 1 9017 0.24 MB/sec execute 100 sec latency 10638.509 ms oleg223-client.virtnet: 1 9017 0.24 MB/sec execute 101 sec latency 11638.817 ms oleg223-client.virtnet: 1 9017 0.24 MB/sec execute 102 sec latency 12639.178 ms oleg223-client.virtnet: 1 9017 0.24 MB/sec execute 103 sec latency 13641.127 ms oleg223-client.virtnet: 1 9017 0.23 MB/sec execute 104 sec latency 14641.905 ms oleg223-client.virtnet: 1 9017 0.23 MB/sec execute 105 sec latency 15642.355 ms oleg223-client.virtnet: 1 9049 0.23 MB/sec execute Started lustre-MDT0000 23:26:05 (1746069965) targets are mounted 23:26:05 (1746069965) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3705856 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3729408 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7435264 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:26:18 (1746069978) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:26:33 (1746069993) targets are mounted 23:26:33 (1746069993) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3719168 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7471104 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:26:46 (1746070006) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server 106 sec latency 16431.258 ms oleg223-client.virtnet: 1 9204 0.23 MB/sec execute 107 sec latency 40.366 ms oleg223-client.virtnet: 1 9401 0.24 MB/sec execute 108 sec latency 35.016 ms oleg223-client.virtnet: 1 9520 0.24 MB/sec execute 109 sec latency 122.838 ms oleg223-client.virtnet: 1 9661 0.24 MB/sec execute 110 sec latency 127.493 ms oleg223-client.virtnet: 1 9873 0.27 MB/sec execute 111 sec latency 133.969 ms oleg223-client.virtnet: 1 10095 0.28 MB/sec execute 112 sec latency 101.861 ms oleg223-client.virtnet: 1 10404 0.29 MB/sec execute 113 sec latency 115.419 ms oleg223-client.virtnet: 1 10659 0.32 MB/sec execute 114 sec latency 60.128 ms oleg223-client.virtnet: 1 10806 0.32 MB/sec execute 115 sec latency 42.361 ms oleg223-client.virtnet: 1 10958 0.32 MB/sec execute 116 sec latency 53.580 ms oleg223-client.virtnet: 1 11050 0.32 MB/sec execute 117 sec latency 72.330 ms oleg223-client.virtnet: 1 11133 0.32 MB/sec execute 118 sec latency 74.249 ms oleg223-client.virtnet: 1 11139 0.32 MB/sec execute 119 sec latency 916.313 ms oleg223-client.virtnet: 1 11139 0.32 MB/sec execute 120 sec latency 1916.485 ms oleg223-client.virtnet: 1 11139 0.31 MB/sec execute 121 sec latency 2917.077 ms oleg223-client.virtnet: 1 11139 0.31 MB/sec execute 122 sec latency 3917.970 ms oleg223-client.virtnet: 1 11139 0.31 MB/sec execute 123 sec latency 4918.733 ms oleg223-client.virtnet: 1 11139 0.31 MB/sec execute 124 sec latency 5919.091 ms oleg223-client.virtnet: 1 11139 0.30 MB/sec execute 125 sec latency 6919.249 ms oleg223-client.virtnet: 1 11139 0.30 MB/sec execute 126 sec latency 7919.730 ms oleg223-client.virtnet: 1 11139 0.30 MB/sec execute 127 sec latency 8919.912 ms oleg223-client.virtnet: 1 11139 0.30 MB/sec execute 128 sec latency 9921.339 ms oleg223-client.virtnet: 1 11139 0.29 MB/sec execute 129 sec latency 10921.496 ms oleg223-client.virtnet: 1 11139 0.29 MB/sec execute 130 sec latency 11923.238 ms oleg223-client.virtnet: 1 11139 0.29 MB/sec execute 131 sec latency 12923.517 ms oleg223-client.virtnet: 1 11139 0.29 MB/sec execute 132 sec latency 13924.211 ms oleg223-client.virtnet: 1 11139 0.28 MB/sec execute 133 sec latency 14924.954 ms oleg223-client.virtnet: 1 11139 0.28 MB/sec execute 134 sec latency 15925.177 ms oleg223-client.virtnet: 1 11139 0.28 MB/sec execute 135 sec latency 16925.305 ms oleg223-client.virtnet: 1 11149 0.28 MB/sec execute 136 sec latency 17794.445 ms oleg223-client.virtnet: 1 11307 0.28 MB/sec execute 137 sec latency 34.673 ms oleg223-client.virtnet: 1 11390 0.28 MB/sec execute 138 sec latency 110.449 ms oleg223-client.virtnet: 1 11508 0.28 MB/sec execute 139 sec latency 71.191 ms oleg223-client.virtnet: 1 11594 0.28 MB/sec execute 140 sec latency 87.896 ms oleg223-client.virtnet: 1 11758 0.28 MB/sec execute 141 sec latency 66.167 ms oleg223-client.virtnet: 1 11879 0.28 MB/sec execute 142 sec latency 54.868 ms oleg223-client.virtnet: 1 12082 0.30 MB/sec execute 143 sec latency 46.670 ms oleg223-client.virtnet: 1 12219 0.30 MB/sec execute 144 sec latency 97.910 ms oleg223-client.virtnet: 1 12308 0.30 MB/sec execute 145 sec latency 137.047 ms oleg223-client.virtnet: 1 12382 0.30 MB/sec execute 146 sec latency 368.630 ms oleg223-client.virtnet: 1 12382 0.29 MB/sec execute 147 sec latency 1368.832 ms oleg223-client.virtnet: 1 12382 0.29 MB/sec execute 148 sec latency 2369.004 ms oleg223-client.virtnet: 1 12382 0.29 MB/sec execute 149 sec latency 3369.195 ms oleg223-client.virtnet: 1 12382 0.29 MB/sec execute 150 sec latency 4369.460 ms oleFailover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:27:01 (1746070021) targets are mounted 23:27:01 (1746070021) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3718144 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7470080 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:27:14 (1746070034) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:27:29 (1746070049) targets are mounted 23:27:29 (1746070049) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec g223-client.virtnet: 1 12382 0.29 MB/sec execute 151 sec latency 5371.361 ms oleg223-client.virtnet: 1 12382 0.28 MB/sec execute 152 sec latency 6374.180 ms oleg223-client.virtnet: 1 12382 0.28 MB/sec execute 153 sec latency 7374.737 ms oleg223-client.virtnet: 1 12382 0.28 MB/sec execute 154 sec latency 8375.416 ms oleg223-client.virtnet: 1 12382 0.28 MB/sec execute 155 sec latency 9375.632 ms oleg223-client.virtnet: 1 12382 0.28 MB/sec execute 156 sec latency 10375.848 ms oleg223-client.virtnet: 1 12382 0.27 MB/sec execute 157 sec latency 11376.263 ms oleg223-client.virtnet: 1 12382 0.27 MB/sec execute 158 sec latency 12376.792 ms oleg223-client.virtnet: 1 12382 0.27 MB/sec execute 159 sec latency 13377.074 ms oleg223-client.virtnet: 1 12382 0.27 MB/sec execute 160 sec latency 14377.296 ms oleg223-client.virtnet: 1 12382 0.27 MB/sec execute 161 sec latency 15378.139 ms oleg223-client.virtnet: 1 12382 0.27 MB/sec execute 162 sec latency 16378.701 ms oleg223-client.virtnet: 1 12382 0.26 MB/sec execute 163 sec latency 17379.665 ms oleg223-client.virtnet: 1 12382 0.26 MB/sec execute 164 sec latency 18381.393 ms oleg223-client.virtnet: 1 12382 0.26 MB/sec execute 165 sec latency 19382.530 ms oleg223-client.virtnet: 1 12430 0.26 MB/sec execute 166 sec latency 20051.990 ms oleg223-client.virtnet: 1 12542 0.26 MB/sec execute 167 sec latency 51.733 ms oleg223-client.virtnet: 1 12666 0.26 MB/sec execute 168 sec latency 117.647 ms oleg223-client.virtnet: 1 12863 0.26 MB/sec execute 169 sec latency 43.568 ms oleg223-client.virtnet: 1 13070 0.26 MB/sec execute 170 sec latency 69.951 ms oleg223-client.virtnet: 1 13285 0.27 MB/sec execute 171 sec latency 66.036 ms oleg223-client.virtnet: 1 13643 0.29 MB/sec execute 172 sec latency 84.820 ms oleg223-client.virtnet: 1 13995 0.30 MB/sec execute 173 sec latency 78.171 ms oleg223-client.virtnet: 1 14229 0.32 MB/sec execute 174 sec latency 57.410 ms oleg223-client.virtnet: 1 14347 0.32 MB/sec execute 175 sec latency 392.097 ms oleg223-client.virtnet: 1 14347 0.32 MB/sec execute 176 sec latency 1393.960 ms oleg223-client.virtnet: 1 14347 0.31 MB/sec execute 177 sec latency 2394.128 ms oleg223-client.virtnet: 1 14347 0.31 MB/sec execute 178 sec latency 3394.612 ms oleg223-client.virtnet: 1 14347 0.31 MB/sec execute 179 sec latency 4394.771 ms oleg223-client.virtnet: 1 14347 0.31 MB/sec execute 180 sec latency 5395.063 ms oleg223-client.virtnet: 1 14347 0.31 MB/sec execute 181 sec latency 6395.271 ms oleg223-client.virtnet: 1 14347 0.31 MB/sec execute 182 sec latency 7399.070 ms oleg223-client.virtnet: 1 14347 0.30 MB/sec execute 183 sec latency 8399.536 ms oleg223-client.virtnet: 1 14347 0.30 MB/sec execute 184 sec latency 9399.695 ms oleg223-client.virtnet: 1 14347 0.30 MB/sec execute 185 sec latency 10400.536 ms oleg223-client.virtnet: 1 14347 0.30 MB/sec execute 186 sec latency 11402.572 ms oleg223-client.virtnet: 1 14347 0.30 MB/sec execute 187 sec latency 12402.900 ms oleg223-client.virtnet: 1 14347 0.30 MB/sec execute 188 sec latency 13403.118 ms oleg223-client.virtnet: 1 14347 0.29 MB/sec execute 189 sec latency 14403.667 ms oleg223-client.virtnet: 1 14347 0.29 MB/sec execute 190 sec latency 15405.024 ms oleg223-client.virtnet: 1 14401 0.30 MB/sec execute 191 sec latency 16139.015 ms oleg223-client.virtnet: 1 14507 0.30 MB/sec execute 192 sec latency 46.123 ms oleg223-client.virtnet: 1 14617 0.30 MB/sec execute 193 sec latency 53.329 ms oleg223-client.virtnet: 1 14730 0.29 MB/sec execute 194 sec latency 43.302 ms oleg223-cliUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3717120 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7461888 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:27:47 (1746070067) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:28:02 (1746070082) targets are mounted 23:28:02 (1746070082) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3722240 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7469056 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg223-server ent.virtnet: 1 14884 0.29 MB/sec execute 195 sec latency 131.978 ms oleg223-client.virtnet: 1 14993 0.29 MB/sec execute 196 sec latency 103.134 ms oleg223-client.virtnet: 1 15082 0.29 MB/sec execute 197 sec latency 140.011 ms oleg223-client.virtnet: 1 15164 0.29 MB/sec execute 198 sec latency 94.764 ms oleg223-client.virtnet: 1 15322 0.30 MB/sec execute 199 sec latency 124.145 ms oleg223-client.virtnet: 1 15456 0.30 MB/sec execute 200 sec latency 58.846 ms oleg223-client.virtnet: 1 15659 0.31 MB/sec execute 201 sec latency 76.841 ms oleg223-client.virtnet: 1 15676 0.31 MB/sec execute 202 sec latency 880.199 ms oleg223-client.virtnet: 1 15676 0.31 MB/sec execute 203 sec latency 1881.957 ms oleg223-client.virtnet: 1 15676 0.30 MB/sec execute 204 sec latency 2882.184 ms oleg223-client.virtnet: 1 15676 0.30 MB/sec execute 205 sec latency 3882.391 ms oleg223-client.virtnet: 1 15676 0.30 MB/sec execute 206 sec latency 4883.992 ms oleg223-client.virtnet: 1 15676 0.30 MB/sec execute 207 sec latency 5884.152 ms oleg223-client.virtnet: 1 15676 0.30 MB/sec execute 208 sec latency 6884.708 ms oleg223-client.virtnet: 1 15676 0.30 MB/sec execute 209 sec latency 7885.005 ms oleg223-client.virtnet: 1 15676 0.30 MB/sec execute 210 sec latency 8885.268 ms oleg223-client.virtnet: 1 15676 0.29 MB/sec execute 211 sec latency 9885.523 ms oleg223-client.virtnet: 1 15676 0.29 MB/sec execute 212 sec latency 10885.735 ms oleg223-client.virtnet: 1 15676 0.29 MB/sec execute 213 sec latency 11886.243 ms oleg223-client.virtnet: 1 15676 0.29 MB/sec execute 214 sec latency 12886.736 ms oleg223-client.virtnet: 1 15676 0.29 MB/sec execute 215 sec latency 13886.896 ms oleg223-client.virtnet: 1 15676 0.29 MB/sec execute 216 sec latency 14888.151 ms oleg223-client.virtnet: 1 15676 0.29 MB/sec execute 217 sec latency 15888.409 ms oleg223-client.virtnet: 1 15676 0.28 MB/sec execute 218 sec latency 16889.534 ms oleg223-client.virtnet: 1 15676 0.28 MB/sec execute 219 sec latency 17890.543 ms oleg223-client.virtnet: 1 15676 0.28 MB/sec execute 220 sec latency 18890.675 ms oleg223-client.virtnet: 1 15676 0.28 MB/sec execute 221 sec latency 19890.988 ms oleg223-client.virtnet: 1 15676 0.28 MB/sec execute 222 sec latency 20891.764 ms oleg223-client.virtnet: 1 15676 0.28 MB/sec execute 223 sec latency 21891.931 ms oleg223-client.virtnet: 1 15676 0.28 MB/sec execute 224 sec latency 22892.303 ms oleg223-client.virtnet: 1 15676 0.28 MB/sec execute 225 sec latency 23892.581 ms oleg223-client.virtnet: 1 15676 0.27 MB/sec execute 226 sec latency 24893.555 ms oleg223-client.virtnet: 1 15676 0.27 MB/sec execute 227 sec latency 25893.834 ms oleg223-client.virtnet: 1 15676 0.27 MB/sec execute 228 sec latency 26893.999 ms oleg223-client.virtnet: 1 15676 0.27 MB/sec execute 229 sec latency 27894.159 ms oleg223-client.virtnet: 1 15676 0.27 MB/sec execute 230 sec latency 28897.112 ms oleg223-client.virtnet: 1 15701 0.27 MB/sec execute 231 sec latency 29674.739 ms oleg223-client.virtnet: 1 15844 0.27 MB/sec execute 232 sec latency 106.581 ms oleg223-client.virtnet: 1 16022 0.27 MB/sec execute 233 sec latency 56.324 ms oleg223-client.virtnet: 1 16239 0.27 MB/sec execute 234 sec latency 63.354 ms oleg223-client.virtnet: 1 16448 0.27 MB/sec execute 235 sec latency 41.044 ms oleg223-client.virtnet: 1 16612 0.27 MB/sec execute 236 sec latency 125.900 ms oleg223-client.virtnet: 1 16781 0.27 MB/sec execute 237 sec latency 146.821 ms oleg223-client.virtnet: 1 17170 0.29 MB/sec execute 238 sec latency 83.745 ms oleg223-client.viStopping /mnt/lustre-mds1 (opts:) on oleg223-server 23:28:20 (1746070100) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 23:28:34 (1746070114) targets are mounted 23:28:34 (1746070114) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec rtnet: 1 17533 0.30 MB/sec execute 239 sec latency 64.220 ms oleg223-client.virtnet: 1 17760 0.31 MB/sec execute 240 sec latency 302.233 ms oleg223-client.virtnet: 1 17760 0.31 MB/sec execute 241 sec latency 1302.743 ms oleg223-client.virtnet: 1 17760 0.31 MB/sec execute 242 sec latency 2302.938 ms oleg223-client.virtnet: 1 17760 0.31 MB/sec execute 243 sec latency 3304.569 ms oleg223-client.virtnet: 1 17760 0.31 MB/sec execute 244 sec latency 4304.704 ms oleg223-client.virtnet: 1 17760 0.30 MB/sec execute 245 sec latency 5304.976 ms oleg223-client.virtnet: 1 17760 0.30 MB/sec execute 246 sec latency 6305.143 ms oleg223-client.virtnet: 1 17760 0.30 MB/sec execute 247 sec latency 7305.595 ms oleg223-client.virtnet: 1 17760 0.30 MB/sec execute 248 sec latency 8306.090 ms oleg223-client.virtnet: 1 17760 0.30 MB/sec execute 249 sec latency 9306.998 ms oleg223-client.virtnet: 1 17760 0.30 MB/sec execute 250 sec latency 10307.172 ms oleg223-client.virtnet: 1 17760 0.30 MB/sec execute 251 sec latency 11309.010 ms oleg223-client.virtnet: 1 17760 0.30 MB/sec execute 252 sec latency 12309.188 ms oleg223-client.virtnet: 1 17760 0.29 MB/sec execute 253 sec latency 13309.339 ms oleg223-client.virtnet: 1 17760 0.29 MB/sec execute 254 sec latency 14309.874 ms oleg223-client.virtnet: 1 17760 0.29 MB/sec execute 255 sec latency 15312.971 ms oleg223-client.virtnet: 1 17766 0.29 MB/sec execute 256 sec latency 16239.246 ms oleg223-client.virtnet: 1 17907 0.29 MB/sec execute 257 sec latency 46.132 ms oleg223-client.virtnet: 1 18019 0.29 MB/sec execute 258 sec latency 76.412 ms oleg223-client.virtnet: 1 18138 0.29 MB/sec execute 259 sec latency 53.226 ms oleg223-client.virtnet: 1 18273 0.29 MB/sec execute 260 sec latency 32.210 ms oleg223-client.virtnet: 1 18425 0.29 MB/sec execute 261 sec latency 108.670 ms oleg223-client.virtnet: 1 18517 0.29 MB/sec execute 262 sec latency 79.386 ms oleg223-client.virtnet: 1 18646 0.29 MB/sec execute 263 sec latency 85.428 ms oleg223-client.virtnet: 1 18794 0.30 MB/sec execute 264 sec latency 83.483 ms oleg223-client.virtnet: 1 18985 0.30 MB/sec execute 265 sec latency 29.923 ms oleg223-client.virtnet: 1 19232 0.31 MB/sec execute 266 sec latency 55.070 ms oleg223-client.virtnet: 1 19382 0.30 MB/sec execute 267 sec latency 76.447 ms oleg223-client.virtnet: 1 19541 0.30 MB/sec execute 268 sec latency 107.980 ms oleg223-client.virtnet: 1 19763 0.30 MB/sec execute 269 sec latency 65.154 ms oleg223-client.virtnet: 1 19968 0.30 MB/sec execute 270 sec latency 27.505 ms oleg223-client.virtnet: 1 20174 0.31 MB/sec execute 271 sec latency 90.924 ms oleg223-client.virtnet: 1 20346 0.31 MB/sec execute 272 sec latency 114.132 ms oleg223-client.virtnet: 1 20702 0.32 MB/sec execute 273 sec latency 119.962 ms oleg223-client.virtnet: 1 20990 0.33 MB/sec execute 274 sec latency 122.651 ms oleg223-client.virtnet: 1 21316 0.34 MB/sec execute 275 sec latency 58.123 ms oleg223-client.virtnet: 1 21537 0.34 MB/sec execute 276 sec latency 30.995 ms oleg223-client.virtnet: 1 21689 0.34 MB/sec execute 277 sec latency 30.828 ms oleg223-client.virtnet: 1 21831 0.34 MB/sec execute 278 sec latency 34.631 ms oleg223-client.virtnet: 1 22006 0.34 MB/sec execute 279 sec latency 159.450 ms oleg223-client.virtnet: 1 22098 0.34 MB/sec execute 280 sec latency 123.581 ms oleg223-client.virtnet: 1 22213 0.34 MB/sec execute 281 sec latency 92.560 ms oleg223-client.virtnet: 1 22317 0.34 MB/sec execute 282 sec latency 90.605 ms oleg223-client.virtnet: 1 22516 0.34 MB/sec execute 283 sec latency 44.171 ms oleg223-client.virtnet: 1 22759 0.35 MB/sec execute 284 sec latency 36.611 ms oleg223-client.virtnet: 1 22910 0.35 MB/sec execute 285 sec latency 79.874 ms oleg223-client.virtnet: 1 23059 0.35 MB/sec execute 286 sec latency 95.227 ms oleg223-client.virtnet: 1 23214 0.35 MB/sec execute 287 sec latency 105.076 ms oleg223-client.virtnet: 1 23398 0.35 MB/sec execute 288 sec latency 137.010 ms oleg223-client.virtnet: 1 23621 0.35 MB/sec execute 289 sec latency 37.229 ms oleg223-client.virtnet: 1 23774 0.35 MB/sec execute 290 sec latency 94.585 ms oleg223-client.virtnet: 1 23929 0.35 MB/sec execute 291 sec latency 98.707 ms oleg223-client.virtnet: 1 24246 0.37 MB/sec execute 292 sec latency 148.108 ms oleg223-client.virtnet: 1 24560 0.37 MB/sec execute 293 sec latency 81.825 ms oleg223-client.virtnet: 1 24872 0.38 MB/sec execute 294 sec latency 55.334 ms oleg223-client.virtnet: 1 25071 0.39 MB/sec execute 295 sec latency 25.269 ms oleg223-client.virtnet: 1 25237 0.39 MB/sec execute 296 sec latency 34.027 ms oleg223-client.virtnet: 1 25385 0.38 MB/sec execute 297 sec latency 25.478 ms oleg223-client.virtnet: 1 25575 0.38 MB/sec execute 298 sec latency 85.729 ms oleg223-client.virtnet: 1 25680 0.38 MB/sec execute 299 sec latency 105.243 ms oleg223-client.virtnet: 1 cleanup 300 sec oleg223-client.virtnet: 0 cleanup 301 sec oleg223-client.virtnet: oleg223-client.virtnet: Operation Count AvgLat MaxLat oleg223-client.virtnet: ---------------------------------------- oleg223-client.virtnet: NTCreateX 3732 44.796 29674.729 oleg223-client.virtnet: Close 2748 3.784 40.257 oleg223-client.virtnet: Rename 158 126.983 16138.993 oleg223-client.virtnet: Unlink 750 28.028 16488.071 oleg223-client.virtnet: Qpathinfo 3353 15.315 20002.608 oleg223-client.virtnet: Qfileinfo 586 0.263 5.995 oleg223-client.virtnet: Qfsinfo 620 0.527 29.672 oleg223-client.virtnet: Sfileinfo 309 16.037 52.305 oleg223-client.virtnet: Find 1305 1.149 96.398 oleg223-client.virtnet: WriteX 1841 2.244 60.836 oleg223-client.virtnet: ReadX 5835 0.055 19.066 oleg223-client.virtnet: LockX 12 3.880 6.296 oleg223-client.virtnet: UnlockX 12 3.154 6.087 oleg223-client.virtnet: Flush 268 68.118 214.276 oleg223-client.virtnet: oleg223-client.virtnet: Throughput 0.384112 MB/sec 1 clients 1 procs max_latency=29674.739 ms oleg223-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg223-client.virtnet at Wed Apr 30 23:29:19 EDT 2025 with return code 0 oleg223-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg223-client.virtnet oleg223-client.virtnet: /mnt/lustre/d70b.replay-single/oleg223-client.virtnet /mnt/lustre/d70b.replay-single/oleg223-client.virtnet oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg223-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg223-client.virtnet: removed directory: 'clients/client0' oleg223-client.virtnet: removed directory: 'clients' oleg223-client.virtnet: removed 'client.txt' oleg223-client.virtnet: /mnt/lustre/d70b.replay-single/oleg223-client.virtnet oleg223-client.virtnet: dbench successfully finished