== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 08:43:13 (1746794593) Starting client oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre Started clients oleg348-client.virtnet: 192.168.203.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg348-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13396 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2168 1285520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3592864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26148 3556152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7149016 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg348-server Stopping /mnt/lustre-mds1 (opts:) on oleg348-server 08:43:22 (1746794602) shut down facet: mds1 facet_host: oleg348-server facet_failover_host: oleg348-server Failover mds1 to oleg348-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 08:43:38 (1746794618) targets are mounted 08:43:38 (1746794618) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2136 1285552 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3593124 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26408 3569948 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28220 7163072 1% /mnt/lustre oleg348-client.virtnet: looking for dbench program oleg348-client.virtnet: /usr/bin/dbench oleg348-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg348-client.virtnet oleg348-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg348-client.virtnet' oleg348-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg348-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg348-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg348-client.virtnet at Fri May 9 08:43:14 EDT 2025 oleg348-client.virtnet: waiting for dbench pid 13420 oleg348-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg348-client.virtnet: oleg348-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg348-client.virtnet: failed to create barrier semaphore oleg348-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg348-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg348-client.virtnet: releasing clients oleg348-client.virtnet: 1 297 10.96 MB/sec warmup 1 sec latency 26.713 ms oleg348-client.virtnet: 1 664 10.58 MB/sec warmup 2 sec latency 17.332 ms oleg348-client.virtnet: 1 972 7.27 MB/sec warmup 3 sec latency 16.933 ms oleg348-client.virtnet: 1 1378 6.50 MB/sec warmup 4 sec latency 18.758 ms oleg348-client.virtnet: 1 1689 5.24 MB/sec warmup 5 sec latency 19.223 ms oleg348-client.virtnet: 1 2060 4.44 MB/sec warmup 6 sec latency 14.389 ms oleg348-client.virtnet: 1 2490 4.10 MB/sec warmup 7 sec latency 17.073 ms oleg348-client.virtnet: 1 3232 4.29 MB/sec warmup 8 sec latency 17.703 ms oleg348-client.virtnet: 1 3723 4.31 MB/sec warmup 9 sec latency 12.789 ms oleg348-client.virtnet: 1 4065 4.01 MB/sec warmup 10 sec latency 15.445 ms oleg348-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 966.323 ms oleg348-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 1966.598 ms oleg348-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 2966.960 ms oleg348-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 3967.157 ms oleg348-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 4967.344 ms oleg348-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 5967.550 ms oleg348-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 6967.787 ms oleg348-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 7968.081 ms oleg348-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 8968.258 ms oleg348-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 9968.431 ms oleg348-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 10968.609 ms oleg348-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 11968.868 ms oleg348-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 12969.039 ms oleg348-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 13969.247 ms oleg348-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 14969.509 ms oleg348-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 15969.934 ms oleg348-client.virtnet: 1 4302 1.50 MB/sec warmup 27 sec latency 16375.380 ms oleg348-client.virtnet: 1 4564 1.47 MB/sec warmup 28 sec latency 21.006 ms oleg348-client.virtnet: 1 4975 1.56 MB/sec warmup 29 sec latency 17.838 ms oleg348-client.virtnet: 1 5252 1.51 MB/sec warmup 30 sec latency 19.571 ms oleg348-client.virtnet: 1 5560 1.47 MB/sec warmup 31 sec latency 16.465 ms oleg348-client.virtnet: 1 5954 1.47 MB/sec warmup 32 sec latency 18.199 ms oleg348-client.virtnet: 1 6324 1.55 MB/sec warmup 33 sec latency 383.294 mstest_70b fail mds2 2 times Failing mds2 on oleg348-server Stopping /mnt/lustre-mds2 (opts:) on oleg348-server 08:43:51 (1746794631) shut down facet: mds2 facet_host: oleg348-server facet_failover_host: oleg348-server Failover mds2 to oleg348-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0001 08:44:06 (1746794646) targets are mounted 08:44:06 (1746794646) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12936 3591776 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36800 3569908 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49736 7161684 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg348-server Stopping /mnt/lustre-mds1 (opts:) on oleg348-server 08:44:20 (1746794660) shut down facet: mds1 facet_host: oleg348-server facet_failover_host: oleg348-server Failover mds1 to oleg348-server mount facets: mds1 oleg348-client.virtnet: 1 6921 1.60 MB/sec warmup 34 sec latency 400.504 ms oleg348-client.virtnet: 1 7336 1.68 MB/sec warmup 35 sec latency 15.257 ms oleg348-client.virtnet: 1 7503 1.64 MB/sec warmup 36 sec latency 287.410 ms oleg348-client.virtnet: 1 7503 1.60 MB/sec warmup 37 sec latency 1287.766 ms oleg348-client.virtnet: 1 7503 1.56 MB/sec warmup 38 sec latency 2288.008 ms oleg348-client.virtnet: 1 7503 1.52 MB/sec warmup 39 sec latency 3288.307 ms oleg348-client.virtnet: 1 7503 1.48 MB/sec warmup 40 sec latency 4288.689 ms oleg348-client.virtnet: 1 7503 1.44 MB/sec warmup 41 sec latency 5288.986 ms oleg348-client.virtnet: 1 7503 1.41 MB/sec warmup 42 sec latency 6289.327 ms oleg348-client.virtnet: 1 7503 1.38 MB/sec warmup 43 sec latency 7289.558 ms oleg348-client.virtnet: 1 7503 1.34 MB/sec warmup 44 sec latency 8289.920 ms oleg348-client.virtnet: 1 7503 1.31 MB/sec warmup 45 sec latency 9290.263 ms oleg348-client.virtnet: 1 7503 1.29 MB/sec warmup 46 sec latency 10290.618 ms oleg348-client.virtnet: 1 7503 1.26 MB/sec warmup 47 sec latency 11290.880 ms oleg348-client.virtnet: 1 7503 1.23 MB/sec warmup 48 sec latency 12291.016 ms oleg348-client.virtnet: 1 7503 1.21 MB/sec warmup 49 sec latency 13291.167 ms oleg348-client.virtnet: 1 7503 1.18 MB/sec warmup 50 sec latency 14291.405 ms oleg348-client.virtnet: 1 7503 1.16 MB/sec warmup 51 sec latency 15291.666 ms oleg348-client.virtnet: 1 7503 1.14 MB/sec warmup 52 sec latency 16291.849 ms oleg348-client.virtnet: 1 7503 1.12 MB/sec warmup 53 sec latency 17292.084 ms oleg348-client.virtnet: 1 7503 1.10 MB/sec warmup 54 sec latency 18292.309 ms oleg348-client.virtnet: 1 7503 1.08 MB/sec warmup 55 sec latency 19292.552 ms oleg348-client.virtnet: 1 7596 1.06 MB/sec warmup 56 sec latency 19854.097 ms oleg348-client.virtnet: 1 7882 1.05 MB/sec warmup 57 sec latency 19.840 ms oleg348-client.virtnet: 1 8194 1.05 MB/sec warmup 58 sec latency 17.510 ms oleg348-client.virtnet: 1 8568 1.09 MB/sec warmup 59 sec latency 17.993 ms oleg348-client.virtnet: 1 9175 0.34 MB/sec execute 1 sec latency 21.467 ms oleg348-client.virtnet: 1 9527 0.91 MB/sec execute 2 sec latency 17.168 ms oleg348-client.virtnet: 1 10030 2.40 MB/sec execute 3 sec latency 18.477 ms oleg348-client.virtnet: 1 10639 3.03 MB/sec execute 4 sec latency 19.972 ms oleg348-client.virtnet: 1 10929 2.70 MB/sec execute 5 sec latency 16.892 ms oleg348-client.virtnet: 1 11158 2.30 MB/sec execute 6 sec latency 16.750 ms oleg348-client.virtnet: 1 11471 2.05 MB/sec execute 7 sec latency 21.466 ms oleg348-client.virtnet: 1 11789 1.98 MB/sec execute 8 sec latency 17.542 ms oleg348-client.virtnet: 1 12148 2.10 MB/sec execute 9 sec latency 20.598 ms oleg348-client.virtnet: 1 12429 1.91 MB/sec execute 10 sec latency 22.390 ms oleg348-client.virtnet: 1 12733 1.77 MB/sec execute 11 sec latency 12.425 ms oleg348-client.virtnet: 1 13074 1.75 MB/sec execute 12 sec latency 17.737 ms oleg348-client.virtnet: 1 13581 2.02 MB/sec execute 13 sec latency 19.743 ms oleg348-client.virtnet: 1 14194 2.23 MB/sec execute 14 sec latency 18.132 ms oleg348-client.virtnet: 1 14475 2.17 MB/sec execute 15 sec latency 16.431 ms oleg348-client.virtnet: 1 14712 2.06 MB/sec execute 16 sec latency 15.775 ms oleg348-client.virtnet: 1 15012 1.97 MB/sec execute 17 sec latency 19.858 ms oleg348-client.virtnet: 1 15329 1.94 MB/sec execute 18 sec latency 19.425 ms oleg348-client.virtnet: 1 15683 2.00 MB/sec execute Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 08:44:36 (1746794676) targets are mounted 08:44:36 (1746794676) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12936 3593092 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36720 3569128 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49656 7162220 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg348-server Stopping /mnt/lustre-mds2 (opts:) on oleg348-server 08:44:49 (1746794689) shut down facet: mds2 facet_host: oleg348-server facet_failover_host: oleg348-server Failover mds2 to oleg348-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0001 08:45:05 (1746794705) targets are mounted 08:45:05 (1746794705) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12936 3593144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38928 3567380 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51864 7160524 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg348-server Stopping /mnt/lustre-mds1 (opts:) on oleg348-server 08:45:18 (1746794718) shut down 19 sec latency 17.975 ms oleg348-client.virtnet: 1 15982 1.91 MB/sec execute 20 sec latency 20.010 ms oleg348-client.virtnet: 1 16321 1.84 MB/sec execute 21 sec latency 17.243 ms oleg348-client.virtnet: 1 16643 1.82 MB/sec execute 22 sec latency 17.754 ms oleg348-client.virtnet: 1 17181 1.98 MB/sec execute 23 sec latency 17.438 ms oleg348-client.virtnet: 1 17772 2.10 MB/sec execute 24 sec latency 18.567 ms oleg348-client.virtnet: 1 18088 2.08 MB/sec execute 25 sec latency 15.608 ms oleg348-client.virtnet: 1 18372 2.00 MB/sec execute 26 sec latency 16.522 ms oleg348-client.virtnet: 1 18658 1.95 MB/sec execute 27 sec latency 19.533 ms oleg348-client.virtnet: 1 18973 1.93 MB/sec execute 28 sec latency 17.902 ms oleg348-client.virtnet: 1 19328 1.97 MB/sec execute 29 sec latency 18.164 ms oleg348-client.virtnet: 1 19606 1.91 MB/sec execute 30 sec latency 19.897 ms oleg348-client.virtnet: 1 19959 1.87 MB/sec execute 31 sec latency 12.485 ms oleg348-client.virtnet: 1 20287 1.87 MB/sec execute 32 sec latency 17.639 ms oleg348-client.virtnet: 1 20960 1.98 MB/sec execute 33 sec latency 18.060 ms oleg348-client.virtnet: 1 21409 2.05 MB/sec execute 34 sec latency 77.386 ms oleg348-client.virtnet: 1 21409 1.99 MB/sec execute 35 sec latency 1077.588 ms oleg348-client.virtnet: 1 21409 1.94 MB/sec execute 36 sec latency 2077.781 ms oleg348-client.virtnet: 1 21409 1.89 MB/sec execute 37 sec latency 3077.980 ms oleg348-client.virtnet: 1 21409 1.84 MB/sec execute 38 sec latency 4078.221 ms oleg348-client.virtnet: 1 21409 1.79 MB/sec execute 39 sec latency 5078.436 ms oleg348-client.virtnet: 1 21409 1.74 MB/sec execute 40 sec latency 6078.667 ms oleg348-client.virtnet: 1 21409 1.70 MB/sec execute 41 sec latency 7078.876 ms oleg348-client.virtnet: 1 21409 1.66 MB/sec execute 42 sec latency 8079.135 ms oleg348-client.virtnet: 1 21409 1.62 MB/sec execute 43 sec latency 9079.351 ms oleg348-client.virtnet: 1 21409 1.59 MB/sec execute 44 sec latency 10079.613 ms oleg348-client.virtnet: 1 21409 1.55 MB/sec execute 45 sec latency 11079.788 ms oleg348-client.virtnet: 1 21409 1.52 MB/sec execute 46 sec latency 12079.951 ms oleg348-client.virtnet: 1 21409 1.48 MB/sec execute 47 sec latency 13080.096 ms oleg348-client.virtnet: 1 21409 1.45 MB/sec execute 48 sec latency 14080.309 ms oleg348-client.virtnet: 1 21409 1.42 MB/sec execute 49 sec latency 15080.498 ms oleg348-client.virtnet: 1 21409 1.40 MB/sec execute 50 sec latency 16080.682 ms oleg348-client.virtnet: 1 21409 1.37 MB/sec execute 51 sec latency 17080.879 ms oleg348-client.virtnet: 1 21409 1.34 MB/sec execute 52 sec latency 18081.040 ms oleg348-client.virtnet: 1 21409 1.32 MB/sec execute 53 sec latency 19081.265 ms oleg348-client.virtnet: 1 21473 1.29 MB/sec execute 54 sec latency 19902.957 ms oleg348-client.virtnet: 1 21732 1.29 MB/sec execute 55 sec latency 16.785 ms oleg348-client.virtnet: 1 22000 1.27 MB/sec execute 56 sec latency 15.156 ms oleg348-client.virtnet: 1 22256 1.26 MB/sec execute 57 sec latency 21.623 ms oleg348-client.virtnet: 1 22526 1.26 MB/sec execute 58 sec latency 137.766 ms oleg348-client.virtnet: 1 22890 1.29 MB/sec execute 59 sec latency 18.401 ms oleg348-client.virtnet: 1 23135 1.28 MB/sec execute 60 sec latency 20.002 ms oleg348-client.virtnet: 1 23477 1.26 MB/sec execute 61 sec latency 13.416 ms oleg348-client.virtnet: 1 23785 1.27 MB/sec execute 62 sec latency 17.660 ms oleg348-client.virtnet: 1 24313 1.33 MB/sec execute 63 sec latency 18.756 ms oleg348-clienfacet: mds1 facet_host: oleg348-server facet_failover_host: oleg348-server Failover mds1 to oleg348-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 08:45:34 (1746794734) targets are mounted 08:45:34 (1746794734) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13060 3592892 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36564 3569724 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7162616 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg348-server Stopping /mnt/lustre-mds2 (opts:) on oleg348-server 08:45:47 (1746794747) shut down facet: mds2 facet_host: oleg348-server facet_failover_host: oleg348-server Failover mds2 to oleg348-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 t.virtnet: 1 24904 1.39 MB/sec execute 64 sec latency 17.766 ms oleg348-client.virtnet: 1 25179 1.38 MB/sec execute 65 sec latency 17.616 ms oleg348-client.virtnet: 1 25431 1.37 MB/sec execute 66 sec latency 15.552 ms oleg348-client.virtnet: 1 25755 1.36 MB/sec execute 67 sec latency 18.338 ms oleg348-client.virtnet: 1 26045 1.36 MB/sec execute 68 sec latency 18.646 ms oleg348-client.virtnet: 1 26414 1.38 MB/sec execute 69 sec latency 18.056 ms oleg348-client.virtnet: 1 26680 1.37 MB/sec execute 70 sec latency 20.420 ms oleg348-client.virtnet: 1 27026 1.35 MB/sec execute 71 sec latency 12.008 ms oleg348-client.virtnet: 1 27349 1.36 MB/sec execute 72 sec latency 17.902 ms oleg348-client.virtnet: 1 27871 1.41 MB/sec execute 73 sec latency 18.780 ms oleg348-client.virtnet: 1 28454 1.46 MB/sec execute 74 sec latency 19.236 ms oleg348-client.virtnet: 1 28739 1.46 MB/sec execute 75 sec latency 16.861 ms oleg348-client.virtnet: 1 28996 1.44 MB/sec execute 76 sec latency 18.276 ms oleg348-client.virtnet: 1 29303 1.43 MB/sec execute 77 sec latency 19.124 ms oleg348-client.virtnet: 1 29623 1.43 MB/sec execute 78 sec latency 19.178 ms oleg348-client.virtnet: 1 30005 1.45 MB/sec execute 79 sec latency 17.670 ms oleg348-client.virtnet: 1 30279 1.44 MB/sec execute 80 sec latency 21.336 ms oleg348-client.virtnet: 1 30650 1.42 MB/sec execute 81 sec latency 11.421 ms oleg348-client.virtnet: 1 31006 1.43 MB/sec execute 82 sec latency 18.316 ms oleg348-client.virtnet: 1 31692 1.49 MB/sec execute 83 sec latency 18.358 ms oleg348-client.virtnet: 1 32166 1.53 MB/sec execute 84 sec latency 13.080 ms oleg348-client.virtnet: 1 32407 1.51 MB/sec execute 85 sec latency 18.077 ms oleg348-client.virtnet: 1 32714 1.50 MB/sec execute 86 sec latency 18.026 ms oleg348-client.virtnet: 1 33004 1.50 MB/sec execute 87 sec latency 19.848 ms oleg348-client.virtnet: 1 33392 1.52 MB/sec execute 88 sec latency 18.299 ms oleg348-client.virtnet: 1 33668 1.50 MB/sec execute 89 sec latency 19.094 ms oleg348-client.virtnet: 1 34026 1.49 MB/sec execute 90 sec latency 12.548 ms oleg348-client.virtnet: 1 34394 1.49 MB/sec execute 91 sec latency 16.576 ms oleg348-client.virtnet: 1 34940 1.53 MB/sec execute 92 sec latency 30.028 ms oleg348-client.virtnet: 1 34940 1.52 MB/sec execute 93 sec latency 1030.228 ms oleg348-client.virtnet: 1 34940 1.50 MB/sec execute 94 sec latency 2030.440 ms oleg348-client.virtnet: 1 34940 1.49 MB/sec execute 95 sec latency 3030.666 ms oleg348-client.virtnet: 1 34940 1.47 MB/sec execute 96 sec latency 4030.879 ms oleg348-client.virtnet: 1 34940 1.45 MB/sec execute 97 sec latency 5031.080 ms oleg348-client.virtnet: 1 34940 1.44 MB/sec execute 98 sec latency 6031.278 ms oleg348-client.virtnet: 1 34940 1.43 MB/sec execute 99 sec latency 7031.527 ms oleg348-client.virtnet: 1 34940 1.41 MB/sec execute 100 sec latency 8031.790 ms oleg348-client.virtnet: 1 34940 1.40 MB/sec execute 101 sec latency 9031.960 ms oleg348-client.virtnet: 1 34940 1.38 MB/sec execute 102 sec latency 10032.132 ms oleg348-client.virtnet: 1 34940 1.37 MB/sec execute 103 sec latency 11032.244 ms oleg348-client.virtnet: 1 34940 1.36 MB/sec execute 104 sec latency 12032.408 ms oleg348-client.virtnet: 1 34940 1.34 MB/sec execute 105 sec latency 13032.670 ms oleg348-client.virtnet: 1 34940 1.33 MB/sec execute 106 sec latency 14032.869 ms oleg348-client.virtnet: 1 34940 1.32 MB/sec execute 107 sec latency 15033.095 ms oleg348-client.virtnet: 1 34940 1.31 MB/sec execute 108 sStarted lustre-MDT0001 08:46:02 (1746794762) targets are mounted 08:46:02 (1746794762) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13052 3592940 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36596 3568036 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7160976 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg348-server Stopping /mnt/lustre-mds1 (opts:) on oleg348-server 08:46:16 (1746794776) shut down facet: mds1 facet_host: oleg348-server facet_failover_host: oleg348-server Failover mds1 to oleg348-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 08:46:31 (1746794791) targets are mounted 08:46:31 (1746794791) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13264 3592096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36516 3569916 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49780 7162012 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg348-server Stopping /mnt/lustre-mds2 (opts:) on oleg348-server 08:46:45 (1746794805) shut down facet: mds2 facet_host: oleg348-server facet_failover_host: oleg348-server ec latency 16033.284 ms oleg348-client.virtnet: 1 34940 1.29 MB/sec execute 109 sec latency 17033.506 ms oleg348-client.virtnet: 1 34940 1.28 MB/sec execute 110 sec latency 18033.763 ms oleg348-client.virtnet: 1 34940 1.27 MB/sec execute 111 sec latency 19033.961 ms oleg348-client.virtnet: 1 35232 1.28 MB/sec execute 112 sec latency 19645.879 ms oleg348-client.virtnet: 1 35673 1.30 MB/sec execute 113 sec latency 16.529 ms oleg348-client.virtnet: 1 35935 1.30 MB/sec execute 114 sec latency 15.654 ms oleg348-client.virtnet: 1 36278 1.29 MB/sec execute 115 sec latency 16.406 ms oleg348-client.virtnet: 1 36555 1.29 MB/sec execute 116 sec latency 17.475 ms oleg348-client.virtnet: 1 36943 1.31 MB/sec execute 117 sec latency 17.668 ms oleg348-client.virtnet: 1 37229 1.30 MB/sec execute 118 sec latency 19.702 ms oleg348-client.virtnet: 1 37530 1.29 MB/sec execute 119 sec latency 14.832 ms oleg348-client.virtnet: 1 37916 1.29 MB/sec execute 120 sec latency 16.903 ms oleg348-client.virtnet: 1 38423 1.32 MB/sec execute 121 sec latency 17.935 ms oleg348-client.virtnet: 1 39031 1.35 MB/sec execute 122 sec latency 21.123 ms oleg348-client.virtnet: 1 39315 1.35 MB/sec execute 123 sec latency 16.169 ms oleg348-client.virtnet: 1 39544 1.34 MB/sec execute 124 sec latency 16.525 ms oleg348-client.virtnet: 1 39858 1.34 MB/sec execute 125 sec latency 21.995 ms oleg348-client.virtnet: 1 40149 1.34 MB/sec execute 126 sec latency 21.309 ms oleg348-client.virtnet: 1 40516 1.35 MB/sec execute 127 sec latency 20.494 ms oleg348-client.virtnet: 1 40787 1.34 MB/sec execute 128 sec latency 20.516 ms oleg348-client.virtnet: 1 41102 1.34 MB/sec execute 129 sec latency 12.361 ms oleg348-client.virtnet: 1 41458 1.34 MB/sec execute 130 sec latency 19.182 ms oleg348-client.virtnet: 1 41953 1.36 MB/sec execute 131 sec latency 19.754 ms oleg348-client.virtnet: 1 42567 1.40 MB/sec execute 132 sec latency 18.611 ms oleg348-client.virtnet: 1 42871 1.40 MB/sec execute 133 sec latency 16.385 ms oleg348-client.virtnet: 1 43128 1.39 MB/sec execute 134 sec latency 14.981 ms oleg348-client.virtnet: 1 43465 1.38 MB/sec execute 135 sec latency 19.283 ms oleg348-client.virtnet: 1 43835 1.38 MB/sec execute 136 sec latency 17.871 ms oleg348-client.virtnet: 1 44312 1.40 MB/sec execute 137 sec latency 18.436 ms oleg348-client.virtnet: 1 44621 1.39 MB/sec execute 138 sec latency 13.216 ms oleg348-client.virtnet: 1 44995 1.39 MB/sec execute 139 sec latency 17.440 ms oleg348-client.virtnet: 1 45513 1.42 MB/sec execute 140 sec latency 18.196 ms oleg348-client.virtnet: 1 46171 1.44 MB/sec execute 141 sec latency 18.020 ms oleg348-client.virtnet: 1 46471 1.44 MB/sec execute 142 sec latency 15.571 ms oleg348-client.virtnet: 1 46729 1.43 MB/sec execute 143 sec latency 15.181 ms oleg348-client.virtnet: 1 47063 1.43 MB/sec execute 144 sec latency 17.233 ms oleg348-client.virtnet: 1 47379 1.43 MB/sec execute 145 sec latency 17.247 ms oleg348-client.virtnet: 1 47738 1.44 MB/sec execute 146 sec latency 18.153 ms oleg348-client.virtnet: 1 48014 1.43 MB/sec execute 147 sec latency 17.656 ms oleg348-client.virtnet: 1 48359 1.42 MB/sec execute 148 sec latency 12.625 ms oleg348-client.virtnet: 1 48730 1.43 MB/sec execute 149 sec latency 18.338 ms oleg348-client.virtnet: 1 49118 1.45 MB/sec execute 150 sec latency 420.562 ms oleg348-client.virtnet: 1 49118 1.44 MB/sec execute 151 sec latency 1420.826 ms oleg348-client.virtnet: 1 49118 1.43 MB/sec execute 152 sec latency 2421.131 ms oleg348-client.virtnet: 1 49118 1.42 MBFailover mds2 to oleg348-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0001 08:46:58 (1746794818) targets are mounted 08:46:58 (1746794818) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13140 3591112 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36508 3569864 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7160976 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg348-server Stopping /mnt/lustre-mds1 (opts:) on oleg348-server 08:47:12 (1746794832) shut down facet: mds1 facet_host: oleg348-server facet_failover_host: oleg348-server Failover mds1 to oleg348-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 08:47:28 (1746794848) targets are mounted 08:47:28 (1746794848) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /sec execute 153 sec latency 3421.442 ms oleg348-client.virtnet: 1 49118 1.41 MB/sec execute 154 sec latency 4421.689 ms oleg348-client.virtnet: 1 49118 1.40 MB/sec execute 155 sec latency 5421.975 ms oleg348-client.virtnet: 1 49118 1.39 MB/sec execute 156 sec latency 6422.230 ms oleg348-client.virtnet: 1 49118 1.39 MB/sec execute 157 sec latency 7422.497 ms oleg348-client.virtnet: 1 49118 1.38 MB/sec execute 158 sec latency 8422.674 ms oleg348-client.virtnet: 1 49118 1.37 MB/sec execute 159 sec latency 9422.878 ms oleg348-client.virtnet: 1 49118 1.36 MB/sec execute 160 sec latency 10423.032 ms oleg348-client.virtnet: 1 49118 1.35 MB/sec execute 161 sec latency 11423.158 ms oleg348-client.virtnet: 1 49118 1.34 MB/sec execute 162 sec latency 12423.285 ms oleg348-client.virtnet: 1 49118 1.33 MB/sec execute 163 sec latency 13423.440 ms oleg348-client.virtnet: 1 49118 1.33 MB/sec execute 164 sec latency 14423.566 ms oleg348-client.virtnet: 1 49118 1.32 MB/sec execute 165 sec latency 15423.734 ms oleg348-client.virtnet: 1 49118 1.31 MB/sec execute 166 sec latency 16423.877 ms oleg348-client.virtnet: 1 49118 1.30 MB/sec execute 167 sec latency 17424.061 ms oleg348-client.virtnet: 1 49118 1.30 MB/sec execute 168 sec latency 18424.155 ms oleg348-client.virtnet: 1 49828 1.32 MB/sec execute 169 sec latency 18446.968 ms oleg348-client.virtnet: 1 50102 1.32 MB/sec execute 170 sec latency 16.732 ms oleg348-client.virtnet: 1 50372 1.31 MB/sec execute 171 sec latency 21.150 ms oleg348-client.virtnet: 1 50628 1.31 MB/sec execute 172 sec latency 18.958 ms oleg348-client.virtnet: 1 50941 1.31 MB/sec execute 173 sec latency 18.860 ms oleg348-client.virtnet: 1 51299 1.32 MB/sec execute 174 sec latency 17.351 ms oleg348-client.virtnet: 1 51574 1.31 MB/sec execute 175 sec latency 20.388 ms oleg348-client.virtnet: 1 51933 1.31 MB/sec execute 176 sec latency 12.813 ms oleg348-client.virtnet: 1 52281 1.31 MB/sec execute 177 sec latency 18.983 ms oleg348-client.virtnet: 1 52956 1.34 MB/sec execute 178 sec latency 18.170 ms oleg348-client.virtnet: 1 53375 1.35 MB/sec execute 179 sec latency 13.979 ms oleg348-client.virtnet: 1 53700 1.35 MB/sec execute 180 sec latency 17.077 ms oleg348-client.virtnet: 1 54012 1.35 MB/sec execute 181 sec latency 17.191 ms oleg348-client.virtnet: 1 54287 1.35 MB/sec execute 182 sec latency 18.445 ms oleg348-client.virtnet: 1 54669 1.36 MB/sec execute 183 sec latency 17.845 ms oleg348-client.virtnet: 1 54967 1.35 MB/sec execute 184 sec latency 13.799 ms oleg348-client.virtnet: 1 55298 1.35 MB/sec execute 185 sec latency 12.658 ms oleg348-client.virtnet: 1 55656 1.35 MB/sec execute 186 sec latency 17.874 ms oleg348-client.virtnet: 1 56210 1.37 MB/sec execute 187 sec latency 18.330 ms oleg348-client.virtnet: 1 56852 1.39 MB/sec execute 188 sec latency 18.472 ms oleg348-client.virtnet: 1 57131 1.39 MB/sec execute 189 sec latency 15.394 ms oleg348-client.virtnet: 1 57530 1.38 MB/sec execute 190 sec latency 15.247 ms oleg348-client.virtnet: 1 57848 1.38 MB/sec execute 191 sec latency 17.549 ms oleg348-client.virtnet: 1 58242 1.39 MB/sec execute 192 sec latency 17.558 ms oleg348-client.virtnet: 1 58561 1.39 MB/sec execute 193 sec latency 18.641 ms oleg348-client.virtnet: 1 58910 1.38 MB/sec execute 194 sec latency 11.938 ms oleg348-client.virtnet: 1 59387 1.39 MB/sec execute 195 sec latency 12.992 ms oleg348-client.virtnet: 1 60155 1.41 MB/sec execute 196 sec latency 17.686 ms oleg348-client.virtnet: 1 60579 1.43 MB/sec execute 197 sec latency 15.655 ms oleg348UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12936 3593092 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36636 3569744 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49572 7162836 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg348-server Stopping /mnt/lustre-mds2 (opts:) on oleg348-server 08:47:41 (1746794861) shut down facet: mds2 facet_host: oleg348-server facet_failover_host: oleg348-server Failover mds2 to oleg348-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0001 08:47:56 (1746794876) targets are mounted 08:47:56 (1746794876) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13128 3593044 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38528 3565820 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51656 7158864 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg348-server Stopping /mnt/lustre-mds1 (opts:) on oleg348-server 08:48:10 (1746794890) shut down facet: mds1 facet_host: oleg348-server facet_failover_host: oleg348-server -client.virtnet: 1 60812 1.42 MB/sec execute 198 sec latency 17.089 ms oleg348-client.virtnet: 1 61129 1.42 MB/sec execute 199 sec latency 24.451 ms oleg348-client.virtnet: 1 61423 1.42 MB/sec execute 200 sec latency 17.336 ms oleg348-client.virtnet: 1 61797 1.43 MB/sec execute 201 sec latency 18.504 ms oleg348-client.virtnet: 1 62081 1.42 MB/sec execute 202 sec latency 21.065 ms oleg348-client.virtnet: 1 62219 1.41 MB/sec execute 203 sec latency 512.284 ms oleg348-client.virtnet: 1 62576 1.41 MB/sec execute 204 sec latency 566.398 ms oleg348-client.virtnet: 1 62945 1.41 MB/sec execute 205 sec latency 17.711 ms oleg348-client.virtnet: 1 63495 1.43 MB/sec execute 206 sec latency 317.850 ms oleg348-client.virtnet: 1 63495 1.42 MB/sec execute 207 sec latency 1318.079 ms oleg348-client.virtnet: 1 63495 1.42 MB/sec execute 208 sec latency 2318.276 ms oleg348-client.virtnet: 1 63495 1.41 MB/sec execute 209 sec latency 3318.498 ms oleg348-client.virtnet: 1 63495 1.40 MB/sec execute 210 sec latency 4318.654 ms oleg348-client.virtnet: 1 63495 1.40 MB/sec execute 211 sec latency 5318.807 ms oleg348-client.virtnet: 1 63495 1.39 MB/sec execute 212 sec latency 6318.956 ms oleg348-client.virtnet: 1 63495 1.38 MB/sec execute 213 sec latency 7319.119 ms oleg348-client.virtnet: 1 63495 1.38 MB/sec execute 214 sec latency 8319.261 ms oleg348-client.virtnet: 1 63495 1.37 MB/sec execute 215 sec latency 9319.469 ms oleg348-client.virtnet: 1 63495 1.36 MB/sec execute 216 sec latency 10319.647 ms oleg348-client.virtnet: 1 63495 1.36 MB/sec execute 217 sec latency 11319.828 ms oleg348-client.virtnet: 1 63495 1.35 MB/sec execute 218 sec latency 12319.999 ms oleg348-client.virtnet: 1 63495 1.35 MB/sec execute 219 sec latency 13320.167 ms oleg348-client.virtnet: 1 63495 1.34 MB/sec execute 220 sec latency 14320.409 ms oleg348-client.virtnet: 1 63495 1.33 MB/sec execute 221 sec latency 15320.615 ms oleg348-client.virtnet: 1 63495 1.33 MB/sec execute 222 sec latency 16320.803 ms oleg348-client.virtnet: 1 63495 1.32 MB/sec execute 223 sec latency 17321.003 ms oleg348-client.virtnet: 1 63495 1.32 MB/sec execute 224 sec latency 18321.227 ms oleg348-client.virtnet: 1 63495 1.31 MB/sec execute 225 sec latency 19321.406 ms oleg348-client.virtnet: 1 63864 1.32 MB/sec execute 226 sec latency 19662.775 ms oleg348-client.virtnet: 1 64158 1.32 MB/sec execute 227 sec latency 16.839 ms oleg348-client.virtnet: 1 64390 1.32 MB/sec execute 228 sec latency 16.019 ms oleg348-client.virtnet: 1 64679 1.31 MB/sec execute 229 sec latency 23.413 ms oleg348-client.virtnet: 1 64964 1.32 MB/sec execute 230 sec latency 18.417 ms oleg348-client.virtnet: 1 65338 1.32 MB/sec execute 231 sec latency 20.707 ms oleg348-client.virtnet: 1 65605 1.32 MB/sec execute 232 sec latency 20.284 ms oleg348-client.virtnet: 1 65925 1.31 MB/sec execute 233 sec latency 13.323 ms oleg348-client.virtnet: 1 66289 1.32 MB/sec execute 234 sec latency 18.813 ms oleg348-client.virtnet: 1 66843 1.33 MB/sec execute 235 sec latency 19.230 ms oleg348-client.virtnet: 1 67441 1.35 MB/sec execute 236 sec latency 17.468 ms oleg348-client.virtnet: 1 67729 1.35 MB/sec execute 237 sec latency 16.096 ms oleg348-client.virtnet: 1 67995 1.34 MB/sec execute 238 sec latency 15.700 ms oleg348-client.virtnet: 1 68305 1.34 MB/sec execute 239 sec latency 19.269 ms oleg348-client.virtnet: 1 68600 1.34 MB/sec execute 240 sec latency 17.885 ms oleg348-client.virtnet: 1 68967 1.35 MB/sec execute 241 sec latency 18.024 ms oleg348-client.virtnet: 1 69223 1.Failover mds1 to oleg348-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 08:48:25 (1746794905) targets are mounted 08:48:25 (1746794905) facet_failover done oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 34 MB/sec execute 242 sec latency 19.455 ms oleg348-client.virtnet: 1 69575 1.34 MB/sec execute 243 sec latency 11.917 ms oleg348-client.virtnet: 1 69985 1.34 MB/sec execute 244 sec latency 17.977 ms oleg348-client.virtnet: 1 70617 1.36 MB/sec execute 245 sec latency 18.276 ms oleg348-client.virtnet: 1 71096 1.37 MB/sec execute 246 sec latency 18.056 ms oleg348-client.virtnet: 1 71365 1.37 MB/sec execute 247 sec latency 16.841 ms oleg348-client.virtnet: 1 71703 1.37 MB/sec execute 248 sec latency 18.263 ms oleg348-client.virtnet: 1 71972 1.37 MB/sec execute 249 sec latency 18.591 ms oleg348-client.virtnet: 1 72419 1.38 MB/sec execute 250 sec latency 18.469 ms oleg348-client.virtnet: 1 72696 1.37 MB/sec execute 251 sec latency 19.192 ms oleg348-client.virtnet: 1 73025 1.37 MB/sec execute 252 sec latency 11.746 ms oleg348-client.virtnet: 1 73386 1.37 MB/sec execute 253 sec latency 13.291 ms oleg348-client.virtnet: 1 73892 1.38 MB/sec execute 254 sec latency 17.642 ms oleg348-client.virtnet: 1 74523 1.40 MB/sec execute 255 sec latency 18.848 ms oleg348-client.virtnet: 1 74835 1.40 MB/sec execute 256 sec latency 16.416 ms oleg348-client.virtnet: 1 75080 1.39 MB/sec execute 257 sec latency 16.755 ms oleg348-client.virtnet: 1 75404 1.39 MB/sec execute 258 sec latency 23.751 ms oleg348-client.virtnet: 1 75740 1.39 MB/sec execute 259 sec latency 17.406 ms oleg348-client.virtnet: 1 76107 1.40 MB/sec execute 260 sec latency 17.428 ms oleg348-client.virtnet: 1 76370 1.39 MB/sec execute 261 sec latency 18.926 ms oleg348-client.virtnet: 1 76712 1.39 MB/sec execute 262 sec latency 11.956 ms oleg348-client.virtnet: 1 77047 1.39 MB/sec execute 263 sec latency 17.539 ms oleg348-client.virtnet: 1 77725 1.41 MB/sec execute 264 sec latency 19.993 ms oleg348-client.virtnet: 1 78219 1.42 MB/sec execute 265 sec latency 17.838 ms oleg348-client.virtnet: 1 78478 1.42 MB/sec execute 266 sec latency 16.872 ms oleg348-client.virtnet: 1 78784 1.41 MB/sec execute 267 sec latency 21.581 ms oleg348-client.virtnet: 1 79053 1.41 MB/sec execute 268 sec latency 17.317 ms oleg348-client.virtnet: 1 79365 1.41 MB/sec execute 269 sec latency 18.319 ms oleg348-client.virtnet: 1 79735 1.42 MB/sec execute 270 sec latency 15.309 ms oleg348-client.virtnet: 1 80067 1.41 MB/sec execute 271 sec latency 18.720 ms oleg348-client.virtnet: 1 80447 1.41 MB/sec execute 272 sec latency 12.576 ms oleg348-client.virtnet: 1 80958 1.42 MB/sec execute 273 sec latency 18.222 ms oleg348-client.virtnet: 1 81573 1.44 MB/sec execute 274 sec latency 19.452 ms oleg348-client.virtnet: 1 81881 1.44 MB/sec execute 275 sec latency 16.189 ms oleg348-client.virtnet: 1 82114 1.44 MB/sec execute 276 sec latency 15.439 ms oleg348-client.virtnet: 1 82452 1.43 MB/sec execute 277 sec latency 20.720 ms oleg348-client.virtnet: 1 82769 1.43 MB/sec execute 278 sec latency 18.610 ms oleg348-client.virtnet: 1 83144 1.44 MB/sec execute 279 sec latency 17.535 ms oleg348-client.virtnet: 1 83424 1.43 MB/sec execute 280 sec latency 20.585 ms oleg348-client.virtnet: 1 83776 1.43 MB/sec execute 281 sec latency 12.270 ms oleg348-client.virtnet: 1 84099 1.43 MB/sec execute 282 sec latency 17.350 ms oleg348-client.virtnet: 1 84632 1.44 MB/sec execute 283 sec latency 19.265 ms oleg348-client.virtnet: 1 85250 1.46 MB/sec execute 284 sec latency 17.185 ms oleg348-client.virtnet: 1 85518 1.46 MB/sec execute 285 sec latency 16.348 ms oleg348-client.virtnet: 1 85855 1.45 MB/sec execute 286 sec latency 20.268 ms oleg348-client.virtnet: 1 86122 1.45 MB/sec execute 287 sec latency 18.779 ms oleg348-client.virtnet: 1 86629 1.46 MB/sec execute 288 sec latency 16.848 ms oleg348-client.virtnet: 1 86901 1.46 MB/sec execute 289 sec latency 19.344 ms oleg348-client.virtnet: 1 87204 1.45 MB/sec execute 290 sec latency 12.928 ms oleg348-client.virtnet: 1 87578 1.45 MB/sec execute 291 sec latency 17.815 ms oleg348-client.virtnet: 1 88097 1.47 MB/sec execute 292 sec latency 18.848 ms oleg348-client.virtnet: 1 88720 1.48 MB/sec execute 293 sec latency 18.607 ms oleg348-client.virtnet: 1 89019 1.48 MB/sec execute 294 sec latency 16.218 ms oleg348-client.virtnet: 1 89260 1.47 MB/sec execute 295 sec latency 16.651 ms oleg348-client.virtnet: 1 89592 1.47 MB/sec execute 296 sec latency 18.334 ms oleg348-client.virtnet: 1 89885 1.47 MB/sec execute 297 sec latency 17.877 ms oleg348-client.virtnet: 1 90245 1.48 MB/sec execute 298 sec latency 18.675 ms oleg348-client.virtnet: 1 90560 1.47 MB/sec execute 299 sec latency 20.272 ms oleg348-client.virtnet: 1 cleanup 300 sec oleg348-client.virtnet: 0 cleanup 300 sec oleg348-client.virtnet: oleg348-client.virtnet: Operation Count AvgLat MaxLat oleg348-client.virtnet: ---------------------------------------- oleg348-client.virtnet: NTCreateX 14217 10.028 19645.866 oleg348-client.virtnet: Close 10455 1.757 11.521 oleg348-client.virtnet: Rename 598 46.838 19902.943 oleg348-client.virtnet: Unlink 2858 2.955 5.741 oleg348-client.virtnet: Qpathinfo 12897 5.361 19662.750 oleg348-client.virtnet: Qfileinfo 2244 0.219 2.683 oleg348-client.virtnet: Qfsinfo 2346 0.370 17.607 oleg348-client.virtnet: Sfileinfo 1150 8.091 18.601 oleg348-client.virtnet: Find 4970 0.513 12.318 oleg348-client.virtnet: WriteX 7001 1.362 8.793 oleg348-client.virtnet: ReadX 22223 0.032 6.931 oleg348-client.virtnet: LockX 46 1.715 2.099 oleg348-client.virtnet: UnlockX 46 1.886 2.297 oleg348-client.virtnet: Flush 992 8.993 566.352 oleg348-client.virtnet: oleg348-client.virtnet: Throughput 1.47116 MB/sec 1 clients 1 procs max_latency=19902.957 ms oleg348-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg348-client.virtnet at Fri May 9 08:49:15 EDT 2025 with return code 0 oleg348-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg348-client.virtnet oleg348-client.virtnet: /mnt/lustre/d70b.replay-single/oleg348-client.virtnet /mnt/lustre/d70b.replay-single/oleg348-client.virtnet oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg348-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg348-client.virtnet: removed directory: 'clients/client0' oleg348-client.virtnet: removed directory: 'clients' oleg348-client.virtnet: removed 'client.txt' oleg348-client.virtnet: /mnt/lustre/d70b.replay-single/oleg348-client.virtnet oleg348-client.virtnet: dbench successfully finished