== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 08:35:28 (1742906128) Starting client oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre Started clients oleg410-client.virtnet: 192.168.204.110@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg410-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13519 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1740 3592564 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26152 3556148 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27892 7148712 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 08:35:36 (1742906136) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 08:35:50 (1742906150) targets are mounted 08:35:50 (1742906150) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1736 3593032 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26416 3569856 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28152 7162888 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 08:36:01 (1742906161) shut down facet: mds2 facet_host: oleg410-server facet_failover_host: oleg410-server oleg410-client.virtnet: looking for dbench program oleg410-client.virtnet: /usr/bin/dbench oleg410-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg410-client.virtnet oleg410-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg410-client.virtnet' oleg410-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg410-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg410-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg410-client.virtnet at Tue Mar 25 08:35:29 EDT 2025 oleg410-client.virtnet: waiting for dbench pid 13543 oleg410-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg410-client.virtnet: oleg410-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg410-client.virtnet: failed to create barrier semaphore oleg410-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg410-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg410-client.virtnet: releasing clients oleg410-client.virtnet: 1 522 18.05 MB/sec warmup 1 sec latency 12.424 ms oleg410-client.virtnet: 1 1209 11.53 MB/sec warmup 2 sec latency 10.047 ms oleg410-client.virtnet: 1 1855 8.79 MB/sec warmup 3 sec latency 12.449 ms oleg410-client.virtnet: 1 2490 7.18 MB/sec warmup 4 sec latency 57.862 ms oleg410-client.virtnet: 1 3688 7.75 MB/sec warmup 5 sec latency 9.532 ms oleg410-client.virtnet: 1 4123 6.68 MB/sec warmup 6 sec latency 214.338 ms oleg410-client.virtnet: 1 4123 5.73 MB/sec warmup 7 sec latency 1214.470 ms oleg410-client.virtnet: 1 4123 5.01 MB/sec warmup 8 sec latency 2214.678 ms oleg410-client.virtnet: 1 4123 4.45 MB/sec warmup 9 sec latency 3214.879 ms oleg410-client.virtnet: 1 4123 4.01 MB/sec warmup 10 sec latency 4215.072 ms oleg410-client.virtnet: 1 4123 3.64 MB/sec warmup 11 sec latency 5215.245 ms oleg410-client.virtnet: 1 4123 3.34 MB/sec warmup 12 sec latency 6215.418 ms oleg410-client.virtnet: 1 4123 3.08 MB/sec warmup 13 sec latency 7215.592 ms oleg410-client.virtnet: 1 4123 2.86 MB/sec warmup 14 sec latency 8215.770 ms oleg410-client.virtnet: 1 4123 2.67 MB/sec warmup 15 sec latency 9215.929 ms oleg410-client.virtnet: 1 4123 2.51 MB/sec warmup 16 sec latency 10216.115 ms oleg410-client.virtnet: 1 4123 2.36 MB/sec warmup 17 sec latency 11216.360 ms oleg410-client.virtnet: 1 4123 2.23 MB/sec warmup 18 sec latency 12216.518 ms oleg410-client.virtnet: 1 4123 2.11 MB/sec warmup 19 sec latency 13216.712 ms oleg410-client.virtnet: 1 4123 2.00 MB/sec warmup 20 sec latency 14216.825 ms oleg410-client.virtnet: 1 4123 1.91 MB/sec warmup 21 sec latency 15216.992 ms oleg410-client.virtnet: 1 4123 1.82 MB/sec warmup 22 sec latency 16217.211 ms oleg410-client.virtnet: 1 4123 1.74 MB/sec warmup 23 sec latency 17217.414 ms oleg410-client.virtnet: 1 4123 1.67 MB/sec warmup 24 sec latency 18217.526 ms oleg410-client.virtnet: 1 4554 1.64 MB/sec warmup 25 sec latency 18423.464 ms oleg410-client.virtnet: 1 5251 1.74 MB/sec warmup 26 sec latency 9.083 ms oleg410-client.virtnet: 1 5967 1.75 MB/sec warmup 27 sec latency 9.534 ms oleg410-client.virtnet: 1 7174 2.06 MB/sec warmup 28 sec latency 8.252 ms oleg410-client.virtnet: 1 7778 2.05 MB/sec warmup 29 sec latency 8.912 ms oleg410-client.virtnet: 1 8382 2.05 MB/sec warmup 30 sec latency 12.373 ms oleg410-client.virtnet: 1 9091 2.09 MB/sec warmup 31 sec latency 9.418 ms oleg410-client.virtnet: 1 9981 2.20 MB/sec warmup 32 sec latency 8.882 ms oleg410-client.virtnet: 1 10943 2.36 MB/sec warmup 33 sec latency 8.988 msFailover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 08:36:15 (1742906175) targets are mounted 08:36:15 (1742906175) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13116 3591888 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36532 3569900 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7161788 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 08:36:26 (1742906186) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 08:36:40 (1742906200) targets are mounted 08:36:40 (1742906200) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13160 3592124 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36456 3569948 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7162072 1% /mnt/lustre oleg410-client.virtnet: 1 11562 2.32 MB/sec warmup 34 sec latency 9.193 ms oleg410-client.virtnet: 1 12271 2.39 MB/sec warmup 35 sec latency 17.763 ms oleg410-client.virtnet: 1 13030 2.37 MB/sec warmup 36 sec latency 8.973 ms oleg410-client.virtnet: 1 14176 2.59 MB/sec warmup 37 sec latency 8.428 ms oleg410-client.virtnet: 1 14766 2.56 MB/sec warmup 38 sec latency 8.042 ms oleg410-client.virtnet: 1 15426 2.55 MB/sec warmup 39 sec latency 10.978 ms oleg410-client.virtnet: 1 16142 2.57 MB/sec warmup 40 sec latency 8.483 ms oleg410-client.virtnet: 1 17068 2.65 MB/sec warmup 41 sec latency 8.211 ms oleg410-client.virtnet: 1 18021 2.77 MB/sec warmup 42 sec latency 8.464 ms oleg410-client.virtnet: 1 18618 2.72 MB/sec warmup 43 sec latency 8.764 ms oleg410-client.virtnet: 1 19320 2.77 MB/sec warmup 44 sec latency 9.216 ms oleg410-client.virtnet: 1 19961 2.72 MB/sec warmup 45 sec latency 8.633 ms oleg410-client.virtnet: 1 20705 2.81 MB/sec warmup 46 sec latency 204.089 ms oleg410-client.virtnet: 1 20705 2.75 MB/sec warmup 47 sec latency 1204.223 ms oleg410-client.virtnet: 1 20705 2.69 MB/sec warmup 48 sec latency 2204.378 ms oleg410-client.virtnet: 1 20705 2.64 MB/sec warmup 49 sec latency 3204.581 ms oleg410-client.virtnet: 1 21434 2.69 MB/sec warmup 50 sec latency 3539.747 ms oleg410-client.virtnet: 1 22031 2.67 MB/sec warmup 51 sec latency 12.355 ms oleg410-client.virtnet: 1 22753 2.71 MB/sec warmup 52 sec latency 8.878 ms oleg410-client.virtnet: 1 23444 2.67 MB/sec warmup 53 sec latency 8.110 ms oleg410-client.virtnet: 1 24323 2.75 MB/sec warmup 54 sec latency 64.258 ms oleg410-client.virtnet: 1 25039 2.79 MB/sec warmup 55 sec latency 430.117 ms oleg410-client.virtnet: 1 25648 2.77 MB/sec warmup 56 sec latency 9.538 ms oleg410-client.virtnet: 1 25933 2.75 MB/sec warmup 57 sec latency 534.564 ms oleg410-client.virtnet: 1 25933 2.70 MB/sec warmup 58 sec latency 1534.755 ms oleg410-client.virtnet: 1 25933 2.66 MB/sec warmup 59 sec latency 2534.972 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 1 sec latency 4535.332 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 2 sec latency 5535.583 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 3 sec latency 6535.770 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 4 sec latency 7536.003 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 5 sec latency 8536.222 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 6 sec latency 9536.418 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 7 sec latency 10536.637 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 8 sec latency 11536.821 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 9 sec latency 12536.966 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 10 sec latency 13537.081 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 11 sec latency 14537.242 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 12 sec latency 15537.348 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 13 sec latency 16537.479 ms oleg410-client.virtnet: 1 25933 0.00 MB/sec execute 14 sec latency 17537.664 ms oleg410-client.virtnet: 1 26006 0.01 MB/sec execute 15 sec latency 18417.757 ms oleg410-client.virtnet: 1 26660 0.21 MB/sec execute 16 sec latency 9.702 ms oleg410-client.virtnet: 1 27343 0.34 MB/sec execute 17 sec latency 9.109 ms oleg410-client.virtnet: 1 28490 0.87 MB/sec execute 18 sec latency 8.696 ms oleg410-client.virtnet: 1 29019 0.91 MB/sec execute 19 sec latenctest_70b fail mds2 4 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 08:36:52 (1742906212) shut down facet: mds2 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 08:37:06 (1742906226) targets are mounted 08:37:06 (1742906226) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13104 3592836 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36552 3569732 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49656 7162568 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 08:37:17 (1742906237) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 08:37:31 (1742906251) targets are mounted 08:37:31 (1742906251) facet_failover done y 8.309 ms oleg410-client.virtnet: 1 29601 0.96 MB/sec execute 20 sec latency 10.280 ms oleg410-client.virtnet: 1 30200 1.07 MB/sec execute 21 sec latency 12.480 ms oleg410-client.virtnet: 1 30853 1.11 MB/sec execute 22 sec latency 10.380 ms oleg410-client.virtnet: 1 31952 1.50 MB/sec execute 23 sec latency 10.374 ms oleg410-client.virtnet: 1 32490 1.51 MB/sec execute 24 sec latency 8.186 ms oleg410-client.virtnet: 1 33119 1.53 MB/sec execute 25 sec latency 9.763 ms oleg410-client.virtnet: 1 33753 1.60 MB/sec execute 26 sec latency 9.895 ms oleg410-client.virtnet: 1 34453 1.63 MB/sec execute 27 sec latency 8.884 ms oleg410-client.virtnet: 1 35626 1.93 MB/sec execute 28 sec latency 8.198 ms oleg410-client.virtnet: 1 36218 1.92 MB/sec execute 29 sec latency 10.074 ms oleg410-client.virtnet: 1 36933 2.02 MB/sec execute 30 sec latency 8.543 ms oleg410-client.virtnet: 1 37569 1.97 MB/sec execute 31 sec latency 8.634 ms oleg410-client.virtnet: 1 38493 2.12 MB/sec execute 32 sec latency 9.217 ms oleg410-client.virtnet: 1 39410 2.26 MB/sec execute 33 sec latency 8.282 ms oleg410-client.virtnet: 1 40017 2.22 MB/sec execute 34 sec latency 8.858 ms oleg410-client.virtnet: 1 40715 2.28 MB/sec execute 35 sec latency 10.049 ms oleg410-client.virtnet: 1 41350 2.26 MB/sec execute 36 sec latency 116.999 ms oleg410-client.virtnet: 1 41350 2.20 MB/sec execute 37 sec latency 1117.183 ms oleg410-client.virtnet: 1 41350 2.14 MB/sec execute 38 sec latency 2117.337 ms oleg410-client.virtnet: 1 41350 2.09 MB/sec execute 39 sec latency 3117.554 ms oleg410-client.virtnet: 1 41350 2.03 MB/sec execute 40 sec latency 4117.728 ms oleg410-client.virtnet: 1 41986 2.12 MB/sec execute 41 sec latency 4421.013 ms oleg410-client.virtnet: 1 42890 2.22 MB/sec execute 42 sec latency 10.401 ms oleg410-client.virtnet: 1 43473 2.19 MB/sec execute 43 sec latency 12.585 ms oleg410-client.virtnet: 1 44163 2.25 MB/sec execute 44 sec latency 11.598 ms oleg410-client.virtnet: 1 44766 2.21 MB/sec execute 45 sec latency 8.804 ms oleg410-client.virtnet: 1 45520 2.31 MB/sec execute 46 sec latency 251.987 ms oleg410-client.virtnet: 1 46456 2.39 MB/sec execute 47 sec latency 8.472 ms oleg410-client.virtnet: 1 46729 2.35 MB/sec execute 48 sec latency 494.905 ms oleg410-client.virtnet: 1 46729 2.30 MB/sec execute 49 sec latency 1495.067 ms oleg410-client.virtnet: 1 46729 2.26 MB/sec execute 50 sec latency 2495.216 ms oleg410-client.virtnet: 1 46729 2.21 MB/sec execute 51 sec latency 3495.423 ms oleg410-client.virtnet: 1 46729 2.17 MB/sec execute 52 sec latency 4495.617 ms oleg410-client.virtnet: 1 46729 2.13 MB/sec execute 53 sec latency 5495.778 ms oleg410-client.virtnet: 1 46729 2.09 MB/sec execute 54 sec latency 6495.943 ms oleg410-client.virtnet: 1 46729 2.05 MB/sec execute 55 sec latency 7496.107 ms oleg410-client.virtnet: 1 46729 2.01 MB/sec execute 56 sec latency 8496.298 ms oleg410-client.virtnet: 1 46729 1.98 MB/sec execute 57 sec latency 9496.432 ms oleg410-client.virtnet: 1 46729 1.94 MB/sec execute 58 sec latency 10496.579 ms oleg410-client.virtnet: 1 46729 1.91 MB/sec execute 59 sec latency 11496.687 ms oleg410-client.virtnet: 1 46729 1.88 MB/sec execute 60 sec latency 12496.918 ms oleg410-client.virtnet: 1 46729 1.85 MB/sec execute 61 sec latency 13497.087 ms oleg410-client.virtnet: 1 46729 1.82 MB/sec execute 62 sec latency 14497.274 ms oleg410-client.virtnet: 1 46729 1.79 MB/sec execute 63 sec latency 15497.389 ms oleg410-client.virtnet: 1 46729 1oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13172 3592908 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36404 3570164 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49576 7163072 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 08:37:43 (1742906263) shut down facet: mds2 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 08:37:57 (1742906277) targets are mounted 08:37:57 (1742906277) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13104 3592808 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36520 3569912 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7162720 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 08:38:09 (1742906289) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server .76 MB/sec execute 64 sec latency 16497.587 ms oleg410-client.virtnet: 1 46729 1.74 MB/sec execute 65 sec latency 17497.761 ms oleg410-client.virtnet: 1 46730 1.71 MB/sec execute 66 sec latency 18493.139 ms oleg410-client.virtnet: 1 47247 1.71 MB/sec execute 67 sec latency 74.080 ms oleg410-client.virtnet: 1 47914 1.74 MB/sec execute 68 sec latency 9.618 ms oleg410-client.virtnet: 1 48615 1.75 MB/sec execute 69 sec latency 8.050 ms oleg410-client.virtnet: 1 49798 1.86 MB/sec execute 70 sec latency 8.490 ms oleg410-client.virtnet: 1 50386 1.86 MB/sec execute 71 sec latency 11.262 ms oleg410-client.virtnet: 1 50964 1.86 MB/sec execute 72 sec latency 9.069 ms oleg410-client.virtnet: 1 51568 1.88 MB/sec execute 73 sec latency 11.348 ms oleg410-client.virtnet: 1 52267 1.89 MB/sec execute 74 sec latency 9.400 ms oleg410-client.virtnet: 1 53374 2.00 MB/sec execute 75 sec latency 9.754 ms oleg410-client.virtnet: 1 53940 1.99 MB/sec execute 76 sec latency 13.760 ms oleg410-client.virtnet: 1 54528 1.99 MB/sec execute 77 sec latency 9.752 ms oleg410-client.virtnet: 1 55228 2.01 MB/sec execute 78 sec latency 9.713 ms oleg410-client.virtnet: 1 56131 2.06 MB/sec execute 79 sec latency 8.998 ms oleg410-client.virtnet: 1 57029 2.12 MB/sec execute 80 sec latency 10.447 ms oleg410-client.virtnet: 1 57586 2.11 MB/sec execute 81 sec latency 10.531 ms oleg410-client.virtnet: 1 58268 2.14 MB/sec execute 82 sec latency 8.873 ms oleg410-client.virtnet: 1 58846 2.12 MB/sec execute 83 sec latency 10.166 ms oleg410-client.virtnet: 1 59695 2.17 MB/sec execute 84 sec latency 11.267 ms oleg410-client.virtnet: 1 60606 2.22 MB/sec execute 85 sec latency 9.487 ms oleg410-client.virtnet: 1 61144 2.21 MB/sec execute 86 sec latency 12.555 ms oleg410-client.virtnet: 1 61825 2.23 MB/sec execute 87 sec latency 9.197 ms oleg410-client.virtnet: 1 62478 2.22 MB/sec execute 88 sec latency 8.733 ms oleg410-client.virtnet: 1 62681 2.20 MB/sec execute 89 sec latency 749.129 ms oleg410-client.virtnet: 1 62681 2.18 MB/sec execute 90 sec latency 1749.273 ms oleg410-client.virtnet: 1 62681 2.16 MB/sec execute 91 sec latency 2749.483 ms oleg410-client.virtnet: 1 63227 2.19 MB/sec execute 92 sec latency 3187.464 ms oleg410-client.virtnet: 1 64142 2.24 MB/sec execute 93 sec latency 8.958 ms oleg410-client.virtnet: 1 64725 2.22 MB/sec execute 94 sec latency 11.428 ms oleg410-client.virtnet: 1 65411 2.25 MB/sec execute 95 sec latency 10.457 ms oleg410-client.virtnet: 1 66054 2.23 MB/sec execute 96 sec latency 9.288 ms oleg410-client.virtnet: 1 66645 2.26 MB/sec execute 97 sec latency 314.420 ms oleg410-client.virtnet: 1 67668 2.32 MB/sec execute 98 sec latency 10.257 ms oleg410-client.virtnet: 1 67934 2.30 MB/sec execute 99 sec latency 453.069 ms oleg410-client.virtnet: 1 67934 2.27 MB/sec execute 100 sec latency 1453.292 ms oleg410-client.virtnet: 1 67934 2.25 MB/sec execute 101 sec latency 2453.501 ms oleg410-client.virtnet: 1 67934 2.23 MB/sec execute 102 sec latency 3453.651 ms oleg410-client.virtnet: 1 67934 2.21 MB/sec execute 103 sec latency 4453.799 ms oleg410-client.virtnet: 1 67934 2.19 MB/sec execute 104 sec latency 5453.979 ms oleg410-client.virtnet: 1 67934 2.17 MB/sec execute 105 sec latency 6454.132 ms oleg410-client.virtnet: 1 67934 2.15 MB/sec execute 106 sec latency 7454.284 ms oleg410-client.virtnet: 1 67934 2.13 MB/sec execute 107 sec latency 8454.512 ms oleg410-client.virtnet: 1 67934 2.11 MB/sec execute 108 sec latency 9454.663 ms oleg410-client.virtnet: Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 08:38:22 (1742906302) targets are mounted 08:38:22 (1742906302) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12940 3593136 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36636 3569704 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49576 7162840 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 08:38:34 (1742906314) shut down facet: mds2 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 08:38:48 (1742906328) targets are mounted 08:38:48 (1742906328) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13068 3592960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36580 3568928 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7161888 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 1 67934 2.09 MB/sec execute 109 sec latency 10454.819 ms oleg410-client.virtnet: 1 67934 2.07 MB/sec execute 110 sec latency 11454.920 ms oleg410-client.virtnet: 1 67934 2.05 MB/sec execute 111 sec latency 12455.101 ms oleg410-client.virtnet: 1 67934 2.03 MB/sec execute 112 sec latency 13455.259 ms oleg410-client.virtnet: 1 67934 2.01 MB/sec execute 113 sec latency 14455.411 ms oleg410-client.virtnet: 1 67934 2.00 MB/sec execute 114 sec latency 15455.530 ms oleg410-client.virtnet: 1 67934 1.98 MB/sec execute 115 sec latency 16455.729 ms oleg410-client.virtnet: 1 67934 1.96 MB/sec execute 116 sec latency 17455.920 ms oleg410-client.virtnet: 1 67946 1.94 MB/sec execute 117 sec latency 18411.540 ms oleg410-client.virtnet: 1 68503 1.94 MB/sec execute 118 sec latency 10.564 ms oleg410-client.virtnet: 1 69124 1.96 MB/sec execute 119 sec latency 9.961 ms oleg410-client.virtnet: 1 69810 1.95 MB/sec execute 120 sec latency 8.846 ms oleg410-client.virtnet: 1 70918 2.02 MB/sec execute 121 sec latency 9.862 ms oleg410-client.virtnet: 1 71421 2.02 MB/sec execute 122 sec latency 9.821 ms oleg410-client.virtnet: 1 71981 2.02 MB/sec execute 123 sec latency 10.629 ms oleg410-client.virtnet: 1 72624 2.03 MB/sec execute 124 sec latency 10.043 ms oleg410-client.virtnet: 1 73290 2.03 MB/sec execute 125 sec latency 10.774 ms oleg410-client.virtnet: 1 74371 2.07 MB/sec execute 126 sec latency 8.768 ms oleg410-client.virtnet: 1 75054 2.09 MB/sec execute 127 sec latency 7.087 ms oleg410-client.virtnet: 1 75705 2.09 MB/sec execute 128 sec latency 10.671 ms oleg410-client.virtnet: 1 76386 2.10 MB/sec execute 129 sec latency 8.913 ms oleg410-client.virtnet: 1 77157 2.11 MB/sec execute 130 sec latency 8.331 ms oleg410-client.virtnet: 1 78318 2.17 MB/sec execute 131 sec latency 8.268 ms oleg410-client.virtnet: 1 78902 2.16 MB/sec execute 132 sec latency 10.782 ms oleg410-client.virtnet: 1 79625 2.18 MB/sec execute 133 sec latency 8.491 ms oleg410-client.virtnet: 1 80319 2.17 MB/sec execute 134 sec latency 8.689 ms oleg410-client.virtnet: 1 81436 2.22 MB/sec execute 135 sec latency 8.586 ms oleg410-client.virtnet: 1 82108 2.24 MB/sec execute 136 sec latency 7.726 ms oleg410-client.virtnet: 1 82781 2.23 MB/sec execute 137 sec latency 9.578 ms oleg410-client.virtnet: 1 83462 2.24 MB/sec execute 138 sec latency 9.813 ms oleg410-client.virtnet: 1 84231 2.25 MB/sec execute 139 sec latency 8.428 ms oleg410-client.virtnet: 1 84250 2.23 MB/sec execute 140 sec latency 962.944 ms oleg410-client.virtnet: 1 84250 2.21 MB/sec execute 141 sec latency 1963.118 ms oleg410-client.virtnet: 1 84250 2.20 MB/sec execute 142 sec latency 2963.271 ms oleg410-client.virtnet: 1 85052 2.23 MB/sec execute 143 sec latency 3364.386 ms oleg410-client.virtnet: 1 85659 2.24 MB/sec execute 144 sec latency 8.234 ms oleg410-client.virtnet: 1 86287 2.24 MB/sec execute 145 sec latency 12.193 ms oleg410-client.virtnet: 1 86973 2.25 MB/sec execute 146 sec latency 10.178 ms oleg410-client.virtnet: 1 87655 2.25 MB/sec execute 147 sec latency 9.225 ms oleg410-client.virtnet: 1 88547 2.28 MB/sec execute 148 sec latency 297.343 ms oleg410-client.virtnet: 1 89189 2.30 MB/sec execute 149 sec latency 8.130 ms oleg410-client.virtnet: 1 89561 2.29 MB/sec execute 150 sec latency 319.979 ms oleg410-client.virtnet: 1 89561 2.27 MB/sec execute 151 sec latency 1320.127 ms oleg410-client.virtnet: 1 89561 2.26 MB/sec execute 152 sec latency 2320.273 ms oleg410-client.virtnet: 1 89561 2.24 MB/sec execute 153 sec latency 3320.442 ms ol08:39:06 (1742906346) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 08:39:20 (1742906360) targets are mounted 08:39:20 (1742906360) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13016 3592848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36496 3570020 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49512 7162868 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 08:39:31 (1742906371) shut down facet: mds2 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 08:39:45 (1742906385) targets are mounted 08:39:45 (1742906385) facet_failover done eg410-client.virtnet: 1 89561 2.23 MB/sec execute 154 sec latency 4320.600 ms oleg410-client.virtnet: 1 89561 2.21 MB/sec execute 155 sec latency 5320.751 ms oleg410-client.virtnet: 1 89561 2.20 MB/sec execute 156 sec latency 6320.942 ms oleg410-client.virtnet: 1 89561 2.18 MB/sec execute 157 sec latency 7321.115 ms oleg410-client.virtnet: 1 89561 2.17 MB/sec execute 158 sec latency 8321.281 ms oleg410-client.virtnet: 1 89561 2.16 MB/sec execute 159 sec latency 9321.450 ms oleg410-client.virtnet: 1 89561 2.14 MB/sec execute 160 sec latency 10321.636 ms oleg410-client.virtnet: 1 89561 2.13 MB/sec execute 161 sec latency 11321.922 ms oleg410-client.virtnet: 1 89561 2.12 MB/sec execute 162 sec latency 12322.195 ms oleg410-client.virtnet: 1 89561 2.10 MB/sec execute 163 sec latency 13322.413 ms oleg410-client.virtnet: 1 89561 2.09 MB/sec execute 164 sec latency 14322.599 ms oleg410-client.virtnet: 1 89561 2.08 MB/sec execute 165 sec latency 15322.745 ms oleg410-client.virtnet: 1 89561 2.06 MB/sec execute 166 sec latency 16322.910 ms oleg410-client.virtnet: 1 89561 2.05 MB/sec execute 167 sec latency 17323.125 ms oleg410-client.virtnet: 1 89561 2.04 MB/sec execute 168 sec latency 18323.299 ms oleg410-client.virtnet: 1 89561 2.03 MB/sec execute 169 sec latency 19323.457 ms oleg410-client.virtnet: 1 89561 2.02 MB/sec execute 170 sec latency 20323.605 ms oleg410-client.virtnet: 1 89561 2.00 MB/sec execute 171 sec latency 21323.775 ms oleg410-client.virtnet: 1 89561 1.99 MB/sec execute 172 sec latency 22323.932 ms oleg410-client.virtnet: 1 89561 1.98 MB/sec execute 173 sec latency 23324.105 ms oleg410-client.virtnet: 1 89561 1.97 MB/sec execute 174 sec latency 24324.200 ms oleg410-client.virtnet: 1 90136 1.98 MB/sec execute 175 sec latency 24511.084 ms oleg410-client.virtnet: 1 90740 1.98 MB/sec execute 176 sec latency 10.725 ms oleg410-client.virtnet: 1 91652 2.00 MB/sec execute 177 sec latency 8.259 ms oleg410-client.virtnet: 1 92578 2.03 MB/sec execute 178 sec latency 8.425 ms oleg410-client.virtnet: 1 93087 2.02 MB/sec execute 179 sec latency 11.513 ms oleg410-client.virtnet: 1 93781 2.04 MB/sec execute 180 sec latency 8.664 ms oleg410-client.virtnet: 1 94453 2.03 MB/sec execute 181 sec latency 9.234 ms oleg410-client.virtnet: 1 95499 2.06 MB/sec execute 182 sec latency 8.600 ms oleg410-client.virtnet: 1 96225 2.08 MB/sec execute 183 sec latency 9.044 ms oleg410-client.virtnet: 1 96866 2.08 MB/sec execute 184 sec latency 8.604 ms oleg410-client.virtnet: 1 97576 2.09 MB/sec execute 185 sec latency 9.266 ms oleg410-client.virtnet: 1 98295 2.09 MB/sec execute 186 sec latency 8.423 ms oleg410-client.virtnet: 1 99448 2.13 MB/sec execute 187 sec latency 9.120 ms oleg410-client.virtnet: 1 100034 2.13 MB/sec execute 188 sec latency 8.634 ms oleg410-client.virtnet: 1 100636 2.13 MB/sec execute 189 sec latency 11.997 ms oleg410-client.virtnet: 1 101334 2.13 MB/sec execute 190 sec latency 9.645 ms oleg410-client.virtnet: 1 102133 2.15 MB/sec execute 191 sec latency 8.319 ms oleg410-client.virtnet: 1 103168 2.18 MB/sec execute 192 sec latency 9.277 ms oleg410-client.virtnet: 1 103783 2.17 MB/sec execute 193 sec latency 9.339 ms oleg410-client.virtnet: 1 104416 2.18 MB/sec execute 194 sec latency 9.771 ms oleg410-client.virtnet: 1 105055 2.18 MB/sec execute 195 sec latency 9.664 ms oleg410-client.virtnet: 1 105493 2.18 MB/sec execute 196 sec latency 366.397 ms oleg410-client.virtnet: 1 105493 2.17 MB/sec execute 197 sec latency 1366.588 ms oleg410-client.virtnet: 1 105493 2.16 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13168 3592964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36396 3570112 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7163076 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 08:39:57 (1742906397) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 08:40:11 (1742906411) targets are mounted 08:40:11 (1742906411) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12940 3593252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36636 3569820 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49576 7163072 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 08:40:23 (1742906423) shut down facet: mds2 facet_host: oleg410-server facet_failover_host: oleg410-server MB/sec execute 198 sec latency 2366.735 ms oleg410-client.virtnet: 1 105493 2.14 MB/sec execute 199 sec latency 3366.880 ms oleg410-client.virtnet: 1 106114 2.16 MB/sec execute 200 sec latency 3870.215 ms oleg410-client.virtnet: 1 106775 2.18 MB/sec execute 201 sec latency 10.821 ms oleg410-client.virtnet: 1 107243 2.17 MB/sec execute 202 sec latency 14.926 ms oleg410-client.virtnet: 1 107775 2.17 MB/sec execute 203 sec latency 12.730 ms oleg410-client.virtnet: 1 108344 2.17 MB/sec execute 204 sec latency 9.572 ms oleg410-client.virtnet: 1 108892 2.17 MB/sec execute 205 sec latency 256.744 ms oleg410-client.virtnet: 1 110059 2.21 MB/sec execute 206 sec latency 9.070 ms oleg410-client.virtnet: 1 110520 2.21 MB/sec execute 207 sec latency 81.044 ms oleg410-client.virtnet: 1 110520 2.20 MB/sec execute 208 sec latency 1081.205 ms oleg410-client.virtnet: 1 110520 2.19 MB/sec execute 209 sec latency 2081.369 ms oleg410-client.virtnet: 1 110520 2.18 MB/sec execute 210 sec latency 3081.583 ms oleg410-client.virtnet: 1 110520 2.17 MB/sec execute 211 sec latency 4081.763 ms oleg410-client.virtnet: 1 110520 2.16 MB/sec execute 212 sec latency 5081.919 ms oleg410-client.virtnet: 1 110520 2.15 MB/sec execute 213 sec latency 6082.119 ms oleg410-client.virtnet: 1 110520 2.14 MB/sec execute 214 sec latency 7082.272 ms oleg410-client.virtnet: 1 110520 2.13 MB/sec execute 215 sec latency 8082.422 ms oleg410-client.virtnet: 1 110520 2.12 MB/sec execute 216 sec latency 9082.640 ms oleg410-client.virtnet: 1 110520 2.11 MB/sec execute 217 sec latency 10082.892 ms oleg410-client.virtnet: 1 110520 2.10 MB/sec execute 218 sec latency 11083.108 ms oleg410-client.virtnet: 1 110520 2.09 MB/sec execute 219 sec latency 12083.232 ms oleg410-client.virtnet: 1 110520 2.08 MB/sec execute 220 sec latency 13083.434 ms oleg410-client.virtnet: 1 110520 2.07 MB/sec execute 221 sec latency 14083.561 ms oleg410-client.virtnet: 1 110520 2.06 MB/sec execute 222 sec latency 15083.719 ms oleg410-client.virtnet: 1 110520 2.05 MB/sec execute 223 sec latency 16083.886 ms oleg410-client.virtnet: 1 110520 2.04 MB/sec execute 224 sec latency 17084.092 ms oleg410-client.virtnet: 1 110520 2.03 MB/sec execute 225 sec latency 18084.290 ms oleg410-client.virtnet: 1 110884 2.03 MB/sec execute 226 sec latency 18420.131 ms oleg410-client.virtnet: 1 111523 2.04 MB/sec execute 227 sec latency 10.340 ms oleg410-client.virtnet: 1 112159 2.03 MB/sec execute 228 sec latency 10.157 ms oleg410-client.virtnet: 1 112935 2.05 MB/sec execute 229 sec latency 11.178 ms oleg410-client.virtnet: 1 113820 2.07 MB/sec execute 230 sec latency 13.087 ms oleg410-client.virtnet: 1 114333 2.06 MB/sec execute 231 sec latency 11.218 ms oleg410-client.virtnet: 1 114971 2.07 MB/sec execute 232 sec latency 11.225 ms oleg410-client.virtnet: 1 115459 2.07 MB/sec execute 233 sec latency 12.128 ms oleg410-client.virtnet: 1 116165 2.07 MB/sec execute 234 sec latency 8.957 ms oleg410-client.virtnet: 1 117299 2.11 MB/sec execute 235 sec latency 9.129 ms oleg410-client.virtnet: 1 117812 2.10 MB/sec execute 236 sec latency 13.655 ms oleg410-client.virtnet: 1 118347 2.10 MB/sec execute 237 sec latency 9.424 ms oleg410-client.virtnet: 1 119061 2.10 MB/sec execute 238 sec latency 8.766 ms oleg410-client.virtnet: 1 119847 2.12 MB/sec execute 239 sec latency 10.875 ms oleg410-client.virtnet: 1 120799 2.14 MB/sec execute 240 sec latency 9.655 ms oleg410-client.virtnet: 1 121385 2.14 MB/sec execute 241 sec latency 10.135 ms oleg410-client.virtnet: 1 122091 2.15 MB/sec execute 242 sec latency 8.796 ms olFailover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 08:40:37 (1742906437) targets are mounted 08:40:37 (1742906437) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec eg410-client.virtnet: 1 122743 2.14 MB/sec execute 243 sec latency 8.617 ms oleg410-client.virtnet: 1 123642 2.16 MB/sec execute 244 sec latency 9.233 ms oleg410-client.virtnet: 1 124524 2.18 MB/sec execute 245 sec latency 9.843 ms oleg410-client.virtnet: 1 125093 2.17 MB/sec execute 246 sec latency 10.206 ms oleg410-client.virtnet: 1 125187 2.16 MB/sec execute 247 sec latency 812.898 ms oleg410-client.virtnet: 1 125187 2.15 MB/sec execute 248 sec latency 1813.054 ms oleg410-client.virtnet: 1 125187 2.15 MB/sec execute 249 sec latency 2813.251 ms oleg410-client.virtnet: 1 125187 2.14 MB/sec execute 250 sec latency 3813.449 ms oleg410-client.virtnet: 1 125210 2.13 MB/sec execute 251 sec latency 4796.536 ms oleg410-client.virtnet: 1 125845 2.14 MB/sec execute 252 sec latency 10.680 ms oleg410-client.virtnet: 1 126390 2.13 MB/sec execute 253 sec latency 9.778 ms oleg410-client.virtnet: 1 127380 2.15 MB/sec execute 254 sec latency 10.319 ms oleg410-client.virtnet: 1 128106 2.17 MB/sec execute 255 sec latency 9.582 ms oleg410-client.virtnet: 1 128602 2.16 MB/sec execute 256 sec latency 14.607 ms oleg410-client.virtnet: 1 129197 2.17 MB/sec execute 257 sec latency 10.943 ms oleg410-client.virtnet: 1 129735 2.16 MB/sec execute 258 sec latency 10.112 ms oleg410-client.virtnet: 1 130488 2.18 MB/sec execute 259 sec latency 10.040 ms oleg410-client.virtnet: 1 131477 2.20 MB/sec execute 260 sec latency 9.261 ms oleg410-client.virtnet: 1 131963 2.19 MB/sec execute 261 sec latency 14.759 ms oleg410-client.virtnet: 1 132443 2.19 MB/sec execute 262 sec latency 12.129 ms oleg410-client.virtnet: 1 133052 2.19 MB/sec execute 263 sec latency 10.332 ms oleg410-client.virtnet: 1 133899 2.20 MB/sec execute 264 sec latency 6.973 ms oleg410-client.virtnet: 1 135041 2.23 MB/sec execute 265 sec latency 8.346 ms oleg410-client.virtnet: 1 135621 2.22 MB/sec execute 266 sec latency 11.452 ms oleg410-client.virtnet: 1 136328 2.23 MB/sec execute 267 sec latency 8.843 ms oleg410-client.virtnet: 1 136988 2.23 MB/sec execute 268 sec latency 9.180 ms oleg410-client.virtnet: 1 138015 2.25 MB/sec execute 269 sec latency 8.833 ms oleg410-client.virtnet: 1 138757 2.26 MB/sec execute 270 sec latency 9.322 ms oleg410-client.virtnet: 1 139354 2.25 MB/sec execute 271 sec latency 12.035 ms oleg410-client.virtnet: 1 140065 2.26 MB/sec execute 272 sec latency 9.606 ms oleg410-client.virtnet: 1 140777 2.26 MB/sec execute 273 sec latency 8.267 ms oleg410-client.virtnet: 1 141930 2.29 MB/sec execute 274 sec latency 10.772 ms oleg410-client.virtnet: 1 142485 2.29 MB/sec execute 275 sec latency 8.420 ms oleg410-client.virtnet: 1 143133 2.29 MB/sec execute 276 sec latency 12.058 ms oleg410-client.virtnet: 1 143803 2.29 MB/sec execute 277 sec latency 9.803 ms oleg410-client.virtnet: 1 144663 2.30 MB/sec execute 278 sec latency 8.629 ms oleg410-client.virtnet: 1 145682 2.32 MB/sec execute 279 sec latency 9.874 ms oleg410-client.virtnet: 1 146265 2.32 MB/sec execute 280 sec latency 12.306 ms oleg410-client.virtnet: 1 146971 2.32 MB/sec execute 281 sec latency 8.865 ms oleg410-client.virtnet: 1 147648 2.32 MB/sec execute 282 sec latency 9.431 ms oleg410-client.virtnet: 1 148703 2.34 MB/sec execute 283 sec latency 11.073 ms oleg410-client.virtnet: 1 149447 2.35 MB/sec execute 284 sec latency 7.692 ms oleg410-client.virtnet: 1 150060 2.35 MB/sec execute 285 sec latency 13.368 ms oleg410-client.virtnet: 1 150712 2.35 MB/sec execute 286 sec latency 8.970 ms oleg410-client.virtnet: 1 151420 2.35 MB/sec execute 287 sec latency 8.326 ms oleg410-client.virtnet: 1 152574 2.38 MB/sec execute 288 sec latency 8.940 ms oleg410-client.virtnet: 1 153142 2.38 MB/sec execute 289 sec latency 8.242 ms oleg410-client.virtnet: 1 153706 2.37 MB/sec execute 290 sec latency 12.641 ms oleg410-client.virtnet: 1 154335 2.38 MB/sec execute 291 sec latency 10.815 ms oleg410-client.virtnet: 1 155040 2.38 MB/sec execute 292 sec latency 8.075 ms oleg410-client.virtnet: 1 156189 2.40 MB/sec execute 293 sec latency 9.331 ms oleg410-client.virtnet: 1 156712 2.40 MB/sec execute 294 sec latency 10.892 ms oleg410-client.virtnet: 1 157281 2.40 MB/sec execute 295 sec latency 11.919 ms oleg410-client.virtnet: 1 157942 2.40 MB/sec execute 296 sec latency 9.722 ms oleg410-client.virtnet: 1 158585 2.40 MB/sec execute 297 sec latency 9.832 ms oleg410-client.virtnet: 1 159735 2.43 MB/sec execute 298 sec latency 9.118 ms oleg410-client.virtnet: 1 160328 2.42 MB/sec execute 299 sec latency 12.258 ms oleg410-client.virtnet: 1 cleanup 300 sec oleg410-client.virtnet: 0 cleanup 300 sec oleg410-client.virtnet: oleg410-client.virtnet: Operation Count AvgLat MaxLat oleg410-client.virtnet: ---------------------------------------- oleg410-client.virtnet: NTCreateX 23346 7.378 24511.073 oleg410-client.virtnet: Close 17152 0.813 5.503 oleg410-client.virtnet: Rename 994 6.770 12.461 oleg410-client.virtnet: Unlink 4719 5.483 18493.121 oleg410-client.virtnet: Qpathinfo 21197 2.081 18417.725 oleg410-client.virtnet: Qfileinfo 3692 0.132 1.797 oleg410-client.virtnet: Qfsinfo 3886 1.333 4420.995 oleg410-client.virtnet: Sfileinfo 1916 5.679 3187.454 oleg410-client.virtnet: Find 8187 0.705 3364.378 oleg410-client.virtnet: WriteX 11535 0.734 5.596 oleg410-client.virtnet: ReadX 36693 0.017 3.336 oleg410-client.virtnet: LockX 76 0.818 1.151 oleg410-client.virtnet: UnlockX 76 0.827 0.990 oleg410-client.virtnet: Flush 1634 4.997 314.408 oleg410-client.virtnet: oleg410-client.virtnet: Throughput 2.42433 MB/sec 1 clients 1 procs max_latency=24511.084 ms oleg410-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg410-client.virtnet at Tue Mar 25 08:41:30 EDT 2025 with return code 0 oleg410-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg410-client.virtnet oleg410-client.virtnet: /mnt/lustre/d70b.replay-single/oleg410-client.virtnet /mnt/lustre/d70b.replay-single/oleg410-client.virtnet oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg410-client.virtnet: removed directory: 'clients/client0' oleg410-client.virtnet: removed directory: 'clients' oleg410-client.virtnet: removed 'client.txt' oleg410-client.virtnet: /mnt/lustre/d70b.replay-single/oleg410-client.virtnet oleg410-client.virtnet: dbench successfully finished