== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 00:32:17 (1721881937) Starting client oleg410-client.virtnet: -o user_xattr,flock oleg410-server@tcp:/lustre /mnt/lustre Started clients oleg410-client.virtnet: 192.168.204.110@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg410-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=11526 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26148 3556152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1896 3592912 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28044 7149064 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 00:32:24 (1721881944) shut down Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 00:32:38 (1721881958) targets are mounted 00:32:38 (1721881958) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2172 1285516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26404 3569924 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1736 3593200 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28140 7163124 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 00:32:50 (1721881970) shut down oleg410-client.virtnet: looking for dbench program oleg410-client.virtnet: /usr/bin/dbench oleg410-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg410-client.virtnet oleg410-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg410-client.virtnet' oleg410-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg410-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg410-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg410-client.virtnet at Thu Jul 25 00:32:18 EDT 2024 oleg410-client.virtnet: waiting for dbench pid 11551 oleg410-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg410-client.virtnet: oleg410-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg410-client.virtnet: failed to create barrier semaphore oleg410-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg410-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg410-client.virtnet: releasing clients oleg410-client.virtnet: 1 603 20.97 MB/sec warmup 1 sec latency 12.283 ms oleg410-client.virtnet: 1 1225 11.54 MB/sec warmup 2 sec latency 8.471 ms oleg410-client.virtnet: 1 1913 8.82 MB/sec warmup 3 sec latency 10.016 ms oleg410-client.virtnet: 1 2761 7.99 MB/sec warmup 4 sec latency 8.704 ms oleg410-client.virtnet: 1 3812 7.95 MB/sec warmup 5 sec latency 8.744 ms oleg410-client.virtnet: 1 4349 6.75 MB/sec warmup 6 sec latency 10.516 ms oleg410-client.virtnet: 1 5040 6.45 MB/sec warmup 7 sec latency 13.837 ms oleg410-client.virtnet: 1 5710 5.73 MB/sec warmup 8 sec latency 9.741 ms oleg410-client.virtnet: 1 6732 5.92 MB/sec warmup 9 sec latency 8.906 ms oleg410-client.virtnet: 1 7527 5.92 MB/sec warmup 10 sec latency 8.730 ms oleg410-client.virtnet: 1 8148 5.56 MB/sec warmup 11 sec latency 10.277 ms oleg410-client.virtnet: 1 8833 5.37 MB/sec warmup 12 sec latency 8.971 ms oleg410-client.virtnet: 1 9557 5.10 MB/sec warmup 13 sec latency 8.269 ms oleg410-client.virtnet: 1 10742 5.49 MB/sec warmup 14 sec latency 8.488 ms oleg410-client.virtnet: 1 11330 5.23 MB/sec warmup 15 sec latency 11.327 ms oleg410-client.virtnet: 1 11941 5.03 MB/sec warmup 16 sec latency 9.953 ms oleg410-client.virtnet: 1 12644 4.93 MB/sec warmup 17 sec latency 8.987 ms oleg410-client.virtnet: 1 13530 4.98 MB/sec warmup 18 sec latency 9.111 ms oleg410-client.virtnet: 1 14462 5.11 MB/sec warmup 19 sec latency 9.266 ms oleg410-client.virtnet: 1 15023 4.90 MB/sec warmup 20 sec latency 9.512 ms oleg410-client.virtnet: 1 15714 4.88 MB/sec warmup 21 sec latency 9.219 ms oleg410-client.virtnet: 1 16358 4.69 MB/sec warmup 22 sec latency 9.288 ms oleg410-client.virtnet: 1 17380 4.81 MB/sec warmup 23 sec latency 8.570 ms oleg410-client.virtnet: 1 18122 4.86 MB/sec warmup 24 sec latency 15.476 ms oleg410-client.virtnet: 1 18685 4.69 MB/sec warmup 25 sec latency 13.155 ms oleg410-client.virtnet: 1 19364 4.68 MB/sec warmup 26 sec latency 9.921 ms oleg410-client.virtnet: 1 20015 4.53 MB/sec warmup 27 sec latency 9.410 ms oleg410-client.virtnet: 1 21050 4.68 MB/sec warmup 28 sec latency 10.796 ms oleg410-client.virtnet: 1 21629 4.67 MB/sec warmup 29 sec latency 13.903 ms oleg410-client.virtnet: 1 21924 4.52 MB/sec warmup 30 sec latency 15.973 ms oleg410-client.virtnet: 1 22435 4.44 MB/sec warmup 31 sec latency 17.700 ms oleg410-client.virtnet: 1 22456 4.30 MB/sec warmup 32 sec latency 982.640 ms oleg410-client.virtnet: 1 22456 4.17 MB/sec warmup 33 sec latency 1982.855 ms oleg410-client.virtnet: 1 22456 4.05 MB/sec Failover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 00:33:04 (1721881984) targets are mounted 00:33:04 (1721881984) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36452 3569952 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13180 3592884 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49632 7162836 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 00:33:16 (1721881996) shut down Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 00:33:30 (1721882010) targets are mounted 00:33:30 (1721882010) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36508 3569752 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13108 3590860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160612 1% /mnt/lustre warmup 34 sec latency 2983.068 ms oleg410-client.virtnet: 1 22456 3.94 MB/sec warmup 35 sec latency 3983.327 ms oleg410-client.virtnet: 1 22456 3.83 MB/sec warmup 36 sec latency 4983.506 ms oleg410-client.virtnet: 1 22456 3.72 MB/sec warmup 37 sec latency 5983.665 ms oleg410-client.virtnet: 1 22456 3.62 MB/sec warmup 38 sec latency 6983.883 ms oleg410-client.virtnet: 1 22456 3.53 MB/sec warmup 39 sec latency 7984.170 ms oleg410-client.virtnet: 1 22456 3.44 MB/sec warmup 40 sec latency 8984.512 ms oleg410-client.virtnet: 1 22456 3.36 MB/sec warmup 41 sec latency 9984.732 ms oleg410-client.virtnet: 1 22456 3.28 MB/sec warmup 42 sec latency 10984.963 ms oleg410-client.virtnet: 1 22456 3.20 MB/sec warmup 43 sec latency 11985.214 ms oleg410-client.virtnet: 1 22456 3.13 MB/sec warmup 44 sec latency 12985.434 ms oleg410-client.virtnet: 1 22456 3.06 MB/sec warmup 45 sec latency 13985.574 ms oleg410-client.virtnet: 1 22456 2.99 MB/sec warmup 46 sec latency 14985.696 ms oleg410-client.virtnet: 1 22456 2.93 MB/sec warmup 47 sec latency 15985.848 ms oleg410-client.virtnet: 1 22456 2.87 MB/sec warmup 48 sec latency 16986.123 ms oleg410-client.virtnet: 1 22456 2.81 MB/sec warmup 49 sec latency 17986.372 ms oleg410-client.virtnet: 1 22487 2.76 MB/sec warmup 50 sec latency 18915.259 ms oleg410-client.virtnet: 1 23127 2.77 MB/sec warmup 51 sec latency 9.737 ms oleg410-client.virtnet: 1 23810 2.76 MB/sec warmup 52 sec latency 10.296 ms oleg410-client.virtnet: 1 24961 2.89 MB/sec warmup 53 sec latency 9.781 ms oleg410-client.virtnet: 1 25541 2.87 MB/sec warmup 54 sec latency 9.077 ms oleg410-client.virtnet: 1 26127 2.85 MB/sec warmup 55 sec latency 9.496 ms oleg410-client.virtnet: 1 26758 2.86 MB/sec warmup 56 sec latency 9.597 ms oleg410-client.virtnet: 1 27478 2.86 MB/sec warmup 57 sec latency 9.164 ms oleg410-client.virtnet: 1 28619 2.99 MB/sec warmup 58 sec latency 14.175 ms oleg410-client.virtnet: 1 29195 2.96 MB/sec warmup 59 sec latency 12.825 ms oleg410-client.virtnet: 1 30445 0.52 MB/sec execute 1 sec latency 10.836 ms oleg410-client.virtnet: 1 31312 3.40 MB/sec execute 2 sec latency 9.321 ms oleg410-client.virtnet: 1 32231 4.58 MB/sec execute 3 sec latency 9.311 ms oleg410-client.virtnet: 1 32782 3.64 MB/sec execute 4 sec latency 13.187 ms oleg410-client.virtnet: 1 33464 3.83 MB/sec execute 5 sec latency 8.928 ms oleg410-client.virtnet: 1 34106 3.29 MB/sec execute 6 sec latency 10.187 ms oleg410-client.virtnet: 1 35119 3.88 MB/sec execute 7 sec latency 9.000 ms oleg410-client.virtnet: 1 35890 4.14 MB/sec execute 8 sec latency 8.520 ms oleg410-client.virtnet: 1 36440 3.78 MB/sec execute 9 sec latency 9.999 ms oleg410-client.virtnet: 1 37094 3.84 MB/sec execute 10 sec latency 9.594 ms oleg410-client.virtnet: 1 37714 3.54 MB/sec execute 11 sec latency 9.803 ms oleg410-client.virtnet: 1 38705 3.87 MB/sec execute 12 sec latency 10.043 ms oleg410-client.virtnet: 1 39423 4.02 MB/sec execute 13 sec latency 9.007 ms oleg410-client.virtnet: 1 40005 3.79 MB/sec execute 14 sec latency 11.828 ms oleg410-client.virtnet: 1 40687 3.83 MB/sec execute 15 sec latency 9.708 ms oleg410-client.virtnet: 1 41324 3.64 MB/sec execute 16 sec latency 16.370 ms oleg410-client.virtnet: 1 42300 3.89 MB/sec execute 17 sec latency 9.244 ms oleg410-client.virtnet: 1 43005 3.97 MB/sec execute 18 sec latency 8.011 ms oleg410-client.virtnet: 1 43584 3.81 MB/sec execute 19 sec latency 10.713 ms oleg410-client.virtnet: 1 44085 3.82 MB/sec extest_70b fail mds2 4 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 00:33:41 (1721882021) shut down Failover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 00:33:55 (1721882035) targets are mounted 00:33:55 (1721882035) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36976 3569524 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12936 3590920 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49912 7160444 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 00:34:06 (1721882046) shut down Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 00:34:20 (1721882060) targets are mounted 00:34:20 (1721882060) facet_failover done ecute 20 sec latency 315.592 ms oleg410-client.virtnet: 1 44642 3.67 MB/sec execute 21 sec latency 388.343 ms oleg410-client.virtnet: 1 45527 3.81 MB/sec execute 22 sec latency 10.255 ms oleg410-client.virtnet: 1 45804 3.68 MB/sec execute 23 sec latency 792.615 ms oleg410-client.virtnet: 1 45804 3.53 MB/sec execute 24 sec latency 1792.869 ms oleg410-client.virtnet: 1 45804 3.39 MB/sec execute 25 sec latency 2793.088 ms oleg410-client.virtnet: 1 45804 3.26 MB/sec execute 26 sec latency 3793.295 ms oleg410-client.virtnet: 1 45804 3.14 MB/sec execute 27 sec latency 4793.481 ms oleg410-client.virtnet: 1 45804 3.03 MB/sec execute 28 sec latency 5793.672 ms oleg410-client.virtnet: 1 45804 2.92 MB/sec execute 29 sec latency 6793.911 ms oleg410-client.virtnet: 1 45804 2.82 MB/sec execute 30 sec latency 7794.107 ms oleg410-client.virtnet: 1 45804 2.73 MB/sec execute 31 sec latency 8794.297 ms oleg410-client.virtnet: 1 45804 2.65 MB/sec execute 32 sec latency 9794.503 ms oleg410-client.virtnet: 1 45804 2.57 MB/sec execute 33 sec latency 10794.678 ms oleg410-client.virtnet: 1 45804 2.49 MB/sec execute 34 sec latency 11794.833 ms oleg410-client.virtnet: 1 45804 2.42 MB/sec execute 35 sec latency 12794.964 ms oleg410-client.virtnet: 1 45804 2.35 MB/sec execute 36 sec latency 13795.168 ms oleg410-client.virtnet: 1 45804 2.29 MB/sec execute 37 sec latency 14795.367 ms oleg410-client.virtnet: 1 45804 2.23 MB/sec execute 38 sec latency 15795.560 ms oleg410-client.virtnet: 1 45804 2.17 MB/sec execute 39 sec latency 16795.757 ms oleg410-client.virtnet: 1 45804 2.12 MB/sec execute 40 sec latency 17795.958 ms oleg410-client.virtnet: 1 46239 2.17 MB/sec execute 41 sec latency 18337.087 ms oleg410-client.virtnet: 1 46759 2.16 MB/sec execute 42 sec latency 8.528 ms oleg410-client.virtnet: 1 47387 2.15 MB/sec execute 43 sec latency 11.287 ms oleg410-client.virtnet: 1 48037 2.18 MB/sec execute 44 sec latency 9.780 ms oleg410-client.virtnet: 1 48732 2.19 MB/sec execute 45 sec latency 8.665 ms oleg410-client.virtnet: 1 49827 2.35 MB/sec execute 46 sec latency 9.025 ms oleg410-client.virtnet: 1 50399 2.34 MB/sec execute 47 sec latency 10.555 ms oleg410-client.virtnet: 1 50982 2.33 MB/sec execute 48 sec latency 14.800 ms oleg410-client.virtnet: 1 51671 2.35 MB/sec execute 49 sec latency 10.097 ms oleg410-client.virtnet: 1 52591 2.43 MB/sec execute 50 sec latency 8.511 ms oleg410-client.virtnet: 1 53510 2.52 MB/sec execute 51 sec latency 8.995 ms oleg410-client.virtnet: 1 54065 2.49 MB/sec execute 52 sec latency 9.772 ms oleg410-client.virtnet: 1 54781 2.53 MB/sec execute 53 sec latency 9.733 ms oleg410-client.virtnet: 1 55401 2.49 MB/sec execute 54 sec latency 8.888 ms oleg410-client.virtnet: 1 56248 2.57 MB/sec execute 55 sec latency 9.447 ms oleg410-client.virtnet: 1 57134 2.64 MB/sec execute 56 sec latency 8.265 ms oleg410-client.virtnet: 1 57712 2.61 MB/sec execute 57 sec latency 10.521 ms oleg410-client.virtnet: 1 58418 2.64 MB/sec execute 58 sec latency 9.110 ms oleg410-client.virtnet: 1 59126 2.62 MB/sec execute 59 sec latency 9.230 ms oleg410-client.virtnet: 1 60135 2.71 MB/sec execute 60 sec latency 9.922 ms oleg410-client.virtnet: 1 60767 2.74 MB/sec execute 61 sec latency 8.440 ms oleg410-client.virtnet: 1 61378 2.72 MB/sec execute 62 sec latency 10.916 ms oleg410-client.virtnet: 1 62017 2.73 MB/sec execute 63 sec latency 9.243 ms oleg410-client.virtnet: 1 62665 2.72 MB/sec execute 64 sec latency 10.346 ms oleg410-client.virtnet: oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36552 3569956 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12936 3593196 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163152 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 00:34:31 (1721882071) shut down Failover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 00:34:45 (1721882085) targets are mounted 00:34:45 (1721882085) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36604 3569660 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13124 3593120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49728 7162780 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 00:34:57 (1721882097) shut down Failover mds1 to oleg410-server 1 63561 2.78 MB/sec execute 65 sec latency 13.454 ms oleg410-client.virtnet: 1 64265 2.82 MB/sec execute 66 sec latency 10.569 ms oleg410-client.virtnet: 1 64848 2.79 MB/sec execute 67 sec latency 10.023 ms oleg410-client.virtnet: 1 65560 2.81 MB/sec execute 68 sec latency 9.136 ms oleg410-client.virtnet: 1 66280 2.80 MB/sec execute 69 sec latency 8.876 ms oleg410-client.virtnet: 1 67417 2.91 MB/sec execute 70 sec latency 8.999 ms oleg410-client.virtnet: 1 67934 2.89 MB/sec execute 71 sec latency 8.876 ms oleg410-client.virtnet: 1 68599 2.88 MB/sec execute 72 sec latency 9.875 ms oleg410-client.virtnet: 1 69116 2.88 MB/sec execute 73 sec latency 229.163 ms oleg410-client.virtnet: 1 69116 2.84 MB/sec execute 74 sec latency 1229.344 ms oleg410-client.virtnet: 1 69116 2.81 MB/sec execute 75 sec latency 2229.505 ms oleg410-client.virtnet: 1 69116 2.77 MB/sec execute 76 sec latency 3229.656 ms oleg410-client.virtnet: 1 69116 2.73 MB/sec execute 77 sec latency 4229.807 ms oleg410-client.virtnet: 1 69116 2.70 MB/sec execute 78 sec latency 5230.003 ms oleg410-client.virtnet: 1 69116 2.66 MB/sec execute 79 sec latency 6230.167 ms oleg410-client.virtnet: 1 69116 2.63 MB/sec execute 80 sec latency 7230.361 ms oleg410-client.virtnet: 1 69116 2.60 MB/sec execute 81 sec latency 8230.553 ms oleg410-client.virtnet: 1 69116 2.57 MB/sec execute 82 sec latency 9230.767 ms oleg410-client.virtnet: 1 69116 2.54 MB/sec execute 83 sec latency 10230.956 ms oleg410-client.virtnet: 1 69116 2.51 MB/sec execute 84 sec latency 11231.130 ms oleg410-client.virtnet: 1 69116 2.48 MB/sec execute 85 sec latency 12231.294 ms oleg410-client.virtnet: 1 69116 2.45 MB/sec execute 86 sec latency 13231.468 ms oleg410-client.virtnet: 1 69116 2.42 MB/sec execute 87 sec latency 14231.586 ms oleg410-client.virtnet: 1 69116 2.39 MB/sec execute 88 sec latency 15231.780 ms oleg410-client.virtnet: 1 69116 2.37 MB/sec execute 89 sec latency 16231.935 ms oleg410-client.virtnet: 1 69116 2.34 MB/sec execute 90 sec latency 17232.081 ms oleg410-client.virtnet: 1 69116 2.31 MB/sec execute 91 sec latency 18232.218 ms oleg410-client.virtnet: 1 69591 2.29 MB/sec execute 92 sec latency 18473.483 ms oleg410-client.virtnet: 1 70407 2.34 MB/sec execute 93 sec latency 10.177 ms oleg410-client.virtnet: 1 71253 2.38 MB/sec execute 94 sec latency 9.501 ms oleg410-client.virtnet: 1 71789 2.37 MB/sec execute 95 sec latency 9.552 ms oleg410-client.virtnet: 1 72427 2.39 MB/sec execute 96 sec latency 9.776 ms oleg410-client.virtnet: 1 73033 2.37 MB/sec execute 97 sec latency 10.550 ms oleg410-client.virtnet: 1 73897 2.41 MB/sec execute 98 sec latency 9.109 ms oleg410-client.virtnet: 1 74812 2.46 MB/sec execute 99 sec latency 14.116 ms oleg410-client.virtnet: 1 75399 2.44 MB/sec execute 100 sec latency 12.511 ms oleg410-client.virtnet: 1 76073 2.46 MB/sec execute 101 sec latency 8.814 ms oleg410-client.virtnet: 1 76727 2.44 MB/sec execute 102 sec latency 9.607 ms oleg410-client.virtnet: 1 77752 2.49 MB/sec execute 103 sec latency 9.154 ms oleg410-client.virtnet: 1 78521 2.53 MB/sec execute 104 sec latency 7.694 ms oleg410-client.virtnet: 1 79153 2.52 MB/sec execute 105 sec latency 10.453 ms oleg410-client.virtnet: 1 79861 2.53 MB/sec execute 106 sec latency 8.904 ms oleg410-client.virtnet: 1 80556 2.53 MB/sec execute 107 sec latency 7.813 ms oleg410-client.virtnet: 1 81753 2.60 MB/sec execute 108 sec latency 8.401 ms oleg410-client.virtnet: 1 82355 2.59 MB/sec execute 109 sec latency 10.28mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 00:35:11 (1721882111) targets are mounted 00:35:11 (1721882111) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36720 3569540 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13060 3592968 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49780 7162508 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 00:35:22 (1721882122) shut down Failover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 00:35:36 (1721882136) targets are mounted 00:35:36 (1721882136) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36392 3567288 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13172 3592908 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7160196 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 00:35:48 (1721882148) shut down 7 ms oleg410-client.virtnet: 1 83045 2.61 MB/sec execute 110 sec latency 8.843 ms oleg410-client.virtnet: 1 83650 2.59 MB/sec execute 111 sec latency 9.998 ms oleg410-client.virtnet: 1 84536 2.62 MB/sec execute 112 sec latency 9.616 ms oleg410-client.virtnet: 1 85464 2.66 MB/sec execute 113 sec latency 9.218 ms oleg410-client.virtnet: 1 86044 2.64 MB/sec execute 114 sec latency 10.153 ms oleg410-client.virtnet: 1 86744 2.66 MB/sec execute 115 sec latency 8.327 ms oleg410-client.virtnet: 1 87432 2.64 MB/sec execute 116 sec latency 8.866 ms oleg410-client.virtnet: 1 88447 2.69 MB/sec execute 117 sec latency 9.847 ms oleg410-client.virtnet: 1 89134 2.71 MB/sec execute 118 sec latency 8.679 ms oleg410-client.virtnet: 1 89722 2.70 MB/sec execute 119 sec latency 11.766 ms oleg410-client.virtnet: 1 90423 2.71 MB/sec execute 120 sec latency 9.292 ms oleg410-client.virtnet: 1 91072 2.70 MB/sec execute 121 sec latency 86.285 ms oleg410-client.virtnet: 1 91655 2.73 MB/sec execute 122 sec latency 450.003 ms oleg410-client.virtnet: 1 92562 2.75 MB/sec execute 123 sec latency 10.446 ms oleg410-client.virtnet: 1 92777 2.73 MB/sec execute 124 sec latency 558.002 ms oleg410-client.virtnet: 1 92777 2.71 MB/sec execute 125 sec latency 1558.206 ms oleg410-client.virtnet: 1 92777 2.69 MB/sec execute 126 sec latency 2558.437 ms oleg410-client.virtnet: 1 92777 2.67 MB/sec execute 127 sec latency 3558.680 ms oleg410-client.virtnet: 1 92777 2.65 MB/sec execute 128 sec latency 4558.877 ms oleg410-client.virtnet: 1 92777 2.63 MB/sec execute 129 sec latency 5559.083 ms oleg410-client.virtnet: 1 92777 2.61 MB/sec execute 130 sec latency 6559.300 ms oleg410-client.virtnet: 1 92777 2.59 MB/sec execute 131 sec latency 7559.507 ms oleg410-client.virtnet: 1 92777 2.57 MB/sec execute 132 sec latency 8559.735 ms oleg410-client.virtnet: 1 92777 2.55 MB/sec execute 133 sec latency 9559.955 ms oleg410-client.virtnet: 1 92777 2.53 MB/sec execute 134 sec latency 10560.194 ms oleg410-client.virtnet: 1 92777 2.51 MB/sec execute 135 sec latency 11560.355 ms oleg410-client.virtnet: 1 92777 2.49 MB/sec execute 136 sec latency 12560.482 ms oleg410-client.virtnet: 1 92777 2.48 MB/sec execute 137 sec latency 13560.674 ms oleg410-client.virtnet: 1 92777 2.46 MB/sec execute 138 sec latency 14560.862 ms oleg410-client.virtnet: 1 92777 2.44 MB/sec execute 139 sec latency 15561.080 ms oleg410-client.virtnet: 1 92777 2.42 MB/sec execute 140 sec latency 16561.304 ms oleg410-client.virtnet: 1 92777 2.41 MB/sec execute 141 sec latency 17561.488 ms oleg410-client.virtnet: 1 92845 2.39 MB/sec execute 142 sec latency 18397.892 ms oleg410-client.virtnet: 1 93408 2.39 MB/sec execute 143 sec latency 12.111 ms oleg410-client.virtnet: 1 93900 2.39 MB/sec execute 144 sec latency 14.469 ms oleg410-client.virtnet: 1 94617 2.39 MB/sec execute 145 sec latency 9.386 ms oleg410-client.virtnet: 1 95633 2.42 MB/sec execute 146 sec latency 8.930 ms oleg410-client.virtnet: 1 96249 2.44 MB/sec execute 147 sec latency 9.135 ms oleg410-client.virtnet: 1 96636 2.42 MB/sec execute 148 sec latency 18.857 ms oleg410-client.virtnet: 1 97241 2.44 MB/sec execute 149 sec latency 14.275 ms oleg410-client.virtnet: 1 97656 2.42 MB/sec execute 150 sec latency 12.023 ms oleg410-client.virtnet: 1 98168 2.42 MB/sec execute 151 sec latency 11.117 ms oleg410-client.virtnet: 1 98840 2.44 MB/sec execute 152 sec latency 16.505 ms oleg410-client.virtnet: 1 99513 2.46 MB/sec execute 153 sec latency 13.635 ms oleg410-client.virtnet: 1 99887 2.45 MB/sFailover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 00:36:02 (1721882162) targets are mounted 00:36:02 (1721882162) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36580 3569852 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13108 3590860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49688 7160712 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 00:36:13 (1721882173) shut down Failover mds2 to oleg410-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 00:36:27 (1721882187) targets are mounted 00:36:27 (1721882187) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36632 3569576 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12944 3592928 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49576 7162504 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg410-server ec execute 154 sec latency 14.549 ms oleg410-client.virtnet: 1 100516 2.45 MB/sec execute 155 sec latency 10.907 ms oleg410-client.virtnet: 1 100975 2.45 MB/sec execute 156 sec latency 12.454 ms oleg410-client.virtnet: 1 101503 2.44 MB/sec execute 157 sec latency 10.627 ms oleg410-client.virtnet: 1 102339 2.47 MB/sec execute 158 sec latency 9.762 ms oleg410-client.virtnet: 1 103151 2.49 MB/sec execute 159 sec latency 13.026 ms oleg410-client.virtnet: 1 103632 2.48 MB/sec execute 160 sec latency 10.749 ms oleg410-client.virtnet: 1 104306 2.49 MB/sec execute 161 sec latency 9.168 ms oleg410-client.virtnet: 1 104917 2.48 MB/sec execute 162 sec latency 11.158 ms oleg410-client.virtnet: 1 105809 2.51 MB/sec execute 163 sec latency 9.255 ms oleg410-client.virtnet: 1 106709 2.53 MB/sec execute 164 sec latency 9.793 ms oleg410-client.virtnet: 1 107240 2.52 MB/sec execute 165 sec latency 11.750 ms oleg410-client.virtnet: 1 107925 2.53 MB/sec execute 166 sec latency 10.127 ms oleg410-client.virtnet: 1 108591 2.52 MB/sec execute 167 sec latency 9.449 ms oleg410-client.virtnet: 1 109462 2.55 MB/sec execute 168 sec latency 14.816 ms oleg410-client.virtnet: 1 110383 2.57 MB/sec execute 169 sec latency 8.655 ms oleg410-client.virtnet: 1 110944 2.56 MB/sec execute 170 sec latency 12.634 ms oleg410-client.virtnet: 1 111619 2.57 MB/sec execute 171 sec latency 9.305 ms oleg410-client.virtnet: 1 112338 2.57 MB/sec execute 172 sec latency 9.132 ms oleg410-client.virtnet: 1 113367 2.60 MB/sec execute 173 sec latency 9.373 ms oleg410-client.virtnet: 1 114002 2.61 MB/sec execute 174 sec latency 8.212 ms oleg410-client.virtnet: 1 114025 2.59 MB/sec execute 175 sec latency 957.177 ms oleg410-client.virtnet: 1 114025 2.58 MB/sec execute 176 sec latency 1957.358 ms oleg410-client.virtnet: 1 114025 2.56 MB/sec execute 177 sec latency 2957.563 ms oleg410-client.virtnet: 1 114025 2.55 MB/sec execute 178 sec latency 3957.749 ms oleg410-client.virtnet: 1 114025 2.54 MB/sec execute 179 sec latency 4957.965 ms oleg410-client.virtnet: 1 114025 2.52 MB/sec execute 180 sec latency 5958.115 ms oleg410-client.virtnet: 1 114025 2.51 MB/sec execute 181 sec latency 6958.403 ms oleg410-client.virtnet: 1 114025 2.49 MB/sec execute 182 sec latency 7958.616 ms oleg410-client.virtnet: 1 114025 2.48 MB/sec execute 183 sec latency 8958.806 ms oleg410-client.virtnet: 1 114025 2.47 MB/sec execute 184 sec latency 9958.963 ms oleg410-client.virtnet: 1 114025 2.45 MB/sec execute 185 sec latency 10959.136 ms oleg410-client.virtnet: 1 114025 2.44 MB/sec execute 186 sec latency 11959.252 ms oleg410-client.virtnet: 1 114025 2.43 MB/sec execute 187 sec latency 12959.415 ms oleg410-client.virtnet: 1 114025 2.41 MB/sec execute 188 sec latency 13959.535 ms oleg410-client.virtnet: 1 114025 2.40 MB/sec execute 189 sec latency 14959.692 ms oleg410-client.virtnet: 1 114025 2.39 MB/sec execute 190 sec latency 15959.876 ms oleg410-client.virtnet: 1 114025 2.38 MB/sec execute 191 sec latency 16960.070 ms oleg410-client.virtnet: 1 114025 2.36 MB/sec execute 192 sec latency 17960.267 ms oleg410-client.virtnet: 1 114320 2.35 MB/sec execute 193 sec latency 18421.994 ms oleg410-client.virtnet: 1 114956 2.37 MB/sec execute 194 sec latency 10.604 ms oleg410-client.virtnet: 1 115546 2.36 MB/sec execute 195 sec latency 10.724 ms oleg410-client.virtnet: 1 116419 2.37 MB/sec execute 196 sec latency 8.808 ms oleg410-client.virtnet: 1 117343 2.40 MB/sec execute 197 sec latency 9.690 ms oleg410-client.virtnet: 1 117884 2.39 MB/sec execute 198 sec latency 15.808 ms oleg410-client.virtStopping /mnt/lustre-mds1 (opts:) on oleg410-server 00:36:38 (1721882198) shut down Failover mds1 to oleg410-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 00:36:52 (1721882212) targets are mounted 00:36:52 (1721882212) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36560 3569712 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12936 3593196 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7162908 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg410-server Stopping /mnt/lustre-mds2 (opts:) on oleg410-server 00:37:09 (1721882229) shut down Failover mds2 to oleg410-server mount facets: mds2 net: 1 118565 2.40 MB/sec execute 199 sec latency 9.059 ms oleg410-client.virtnet: 1 119228 2.39 MB/sec execute 200 sec latency 9.316 ms oleg410-client.virtnet: 1 120225 2.42 MB/sec execute 201 sec latency 8.431 ms oleg410-client.virtnet: 1 121016 2.44 MB/sec execute 202 sec latency 8.319 ms oleg410-client.virtnet: 1 121595 2.43 MB/sec execute 203 sec latency 9.351 ms oleg410-client.virtnet: 1 122290 2.44 MB/sec execute 204 sec latency 9.290 ms oleg410-client.virtnet: 1 123010 2.43 MB/sec execute 205 sec latency 8.985 ms oleg410-client.virtnet: 1 124168 2.47 MB/sec execute 206 sec latency 8.719 ms oleg410-client.virtnet: 1 124667 2.47 MB/sec execute 207 sec latency 9.183 ms oleg410-client.virtnet: 1 125320 2.47 MB/sec execute 208 sec latency 11.013 ms oleg410-client.virtnet: 1 125994 2.47 MB/sec execute 209 sec latency 8.876 ms oleg410-client.virtnet: 1 126666 2.47 MB/sec execute 210 sec latency 13.254 ms oleg410-client.virtnet: 1 127812 2.51 MB/sec execute 211 sec latency 12.689 ms oleg410-client.virtnet: 1 128412 2.50 MB/sec execute 212 sec latency 8.967 ms oleg410-client.virtnet: 1 129007 2.50 MB/sec execute 213 sec latency 8.439 ms oleg410-client.virtnet: 1 129700 2.50 MB/sec execute 214 sec latency 10.220 ms oleg410-client.virtnet: 1 130323 2.50 MB/sec execute 215 sec latency 12.135 ms oleg410-client.virtnet: 1 131477 2.54 MB/sec execute 216 sec latency 9.467 ms oleg410-client.virtnet: 1 131929 2.53 MB/sec execute 217 sec latency 10.855 ms oleg410-client.virtnet: 1 132484 2.53 MB/sec execute 218 sec latency 15.391 ms oleg410-client.virtnet: 1 133143 2.53 MB/sec execute 219 sec latency 8.911 ms oleg410-client.virtnet: 1 133839 2.53 MB/sec execute 220 sec latency 10.256 ms oleg410-client.virtnet: 1 134659 2.55 MB/sec execute 221 sec latency 16.911 ms oleg410-client.virtnet: 1 135299 2.56 MB/sec execute 222 sec latency 10.536 ms oleg410-client.virtnet: 1 135770 2.55 MB/sec execute 223 sec latency 241.183 ms oleg410-client.virtnet: 1 136326 2.56 MB/sec execute 224 sec latency 13.345 ms oleg410-client.virtnet: 1 136687 2.55 MB/sec execute 225 sec latency 369.398 ms oleg410-client.virtnet: 1 136687 2.54 MB/sec execute 226 sec latency 1369.630 ms oleg410-client.virtnet: 1 136687 2.53 MB/sec execute 227 sec latency 2369.837 ms oleg410-client.virtnet: 1 136687 2.52 MB/sec execute 228 sec latency 3370.011 ms oleg410-client.virtnet: 1 136687 2.51 MB/sec execute 229 sec latency 4370.194 ms oleg410-client.virtnet: 1 136687 2.50 MB/sec execute 230 sec latency 5370.460 ms oleg410-client.virtnet: 1 136687 2.48 MB/sec execute 231 sec latency 6370.607 ms oleg410-client.virtnet: 1 136687 2.47 MB/sec execute 232 sec latency 7370.786 ms oleg410-client.virtnet: 1 136687 2.46 MB/sec execute 233 sec latency 8370.999 ms oleg410-client.virtnet: 1 136687 2.45 MB/sec execute 234 sec latency 9371.218 ms oleg410-client.virtnet: 1 136687 2.44 MB/sec execute 235 sec latency 10371.442 ms oleg410-client.virtnet: 1 136687 2.43 MB/sec execute 236 sec latency 11371.670 ms oleg410-client.virtnet: 1 136687 2.42 MB/sec execute 237 sec latency 12371.930 ms oleg410-client.virtnet: 1 136687 2.41 MB/sec execute 238 sec latency 13372.160 ms oleg410-client.virtnet: 1 136687 2.40 MB/sec execute 239 sec latency 14372.388 ms oleg410-client.virtnet: 1 136687 2.39 MB/sec execute 240 sec latency 15372.581 ms oleg410-client.virtnet: 1 136687 2.38 MB/sec execute 241 sec latency 16372.781 ms oleg410-client.virtnet: 1 136687 2.37 MB/sec execute 242 sec latency 17372.948 ms oleg410-client.virtnet: 1 136687 2.36 MB/sec execute 243 sec latency Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 00:37:23 (1721882243) targets are mounted 00:37:23 (1721882243) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 18373.068 ms oleg410-client.virtnet: 1 136687 2.35 MB/sec execute 244 sec latency 19373.226 ms oleg410-client.virtnet: 1 136687 2.34 MB/sec execute 245 sec latency 20373.382 ms oleg410-client.virtnet: 1 136687 2.33 MB/sec execute 246 sec latency 21373.510 ms oleg410-client.virtnet: 1 136687 2.32 MB/sec execute 247 sec latency 22373.767 ms oleg410-client.virtnet: 1 136687 2.31 MB/sec execute 248 sec latency 23373.986 ms oleg410-client.virtnet: 1 136687 2.31 MB/sec execute 249 sec latency 24374.124 ms oleg410-client.virtnet: 1 137311 2.31 MB/sec execute 250 sec latency 24421.335 ms oleg410-client.virtnet: 1 138461 2.34 MB/sec execute 251 sec latency 9.202 ms oleg410-client.virtnet: 1 139029 2.33 MB/sec execute 252 sec latency 9.089 ms oleg410-client.virtnet: 1 139633 2.33 MB/sec execute 253 sec latency 10.968 ms oleg410-client.virtnet: 1 140351 2.34 MB/sec execute 254 sec latency 9.529 ms oleg410-client.virtnet: 1 141257 2.35 MB/sec execute 255 sec latency 8.657 ms oleg410-client.virtnet: 1 142174 2.37 MB/sec execute 256 sec latency 9.061 ms oleg410-client.virtnet: 1 142770 2.36 MB/sec execute 257 sec latency 14.345 ms oleg410-client.virtnet: 1 143475 2.37 MB/sec execute 258 sec latency 9.389 ms oleg410-client.virtnet: 1 144143 2.37 MB/sec execute 259 sec latency 9.264 ms oleg410-client.virtnet: 1 145249 2.39 MB/sec execute 260 sec latency 9.217 ms oleg410-client.virtnet: 1 145814 2.40 MB/sec execute 261 sec latency 12.895 ms oleg410-client.virtnet: 1 146250 2.39 MB/sec execute 262 sec latency 14.122 ms oleg410-client.virtnet: 1 146735 2.39 MB/sec execute 263 sec latency 12.999 ms oleg410-client.virtnet: 1 147449 2.39 MB/sec execute 264 sec latency 9.416 ms oleg410-client.virtnet: 1 148093 2.39 MB/sec execute 265 sec latency 11.920 ms oleg410-client.virtnet: 1 149212 2.42 MB/sec execute 266 sec latency 10.417 ms oleg410-client.virtnet: 1 149512 2.42 MB/sec execute 267 sec latency 13.669 ms oleg410-client.virtnet: 1 149991 2.41 MB/sec execute 268 sec latency 12.999 ms oleg410-client.virtnet: 1 150680 2.42 MB/sec execute 269 sec latency 9.571 ms oleg410-client.virtnet: 1 151375 2.41 MB/sec execute 270 sec latency 9.512 ms oleg410-client.virtnet: 1 152498 2.44 MB/sec execute 271 sec latency 9.732 ms oleg410-client.virtnet: 1 153013 2.44 MB/sec execute 272 sec latency 12.942 ms oleg410-client.virtnet: 1 153622 2.44 MB/sec execute 273 sec latency 15.307 ms oleg410-client.virtnet: 1 154302 2.44 MB/sec execute 274 sec latency 9.416 ms oleg410-client.virtnet: 1 154899 2.44 MB/sec execute 275 sec latency 11.994 ms oleg410-client.virtnet: 1 155905 2.46 MB/sec execute 276 sec latency 10.577 ms oleg410-client.virtnet: 1 156533 2.47 MB/sec execute 277 sec latency 10.305 ms oleg410-client.virtnet: 1 157139 2.46 MB/sec execute 278 sec latency 11.744 ms oleg410-client.virtnet: 1 157750 2.47 MB/sec execute 279 sec latency 10.603 ms oleg410-client.virtnet: 1 158462 2.47 MB/sec execute 280 sec latency 10.223 ms oleg410-client.virtnet: 1 159602 2.49 MB/sec execute 281 sec latency 9.243 ms oleg410-client.virtnet: 1 160148 2.49 MB/sec execute 282 sec latency 8.041 ms oleg410-client.virtnet: 1 160806 2.49 MB/sec execute 283 sec latency 10.458 ms oleg410-client.virtnet: 1 161461 2.49 MB/sec execute 284 sec latency 9.419 ms oleg410-client.virtnet: 1 162203 2.49 MB/sec execute 285 sec latency 8.769 ms oleg410-client.virtnet: 1 163321 2.52 MB/sec execute 286 sec latency 8.918 ms oleg410-client.virtnet: 1 163881 2.51 MB/sec execute 287 sec latency 11.477 ms oleg410-client.virtnet: 1 164464 2.51 MB/sec execute 288 sec latency 12.252 ms oleg410-client.virtnet: 1 164887 2.51 MB/sec execute 289 sec latency 13.625 ms oleg410-client.virtnet: 1 165522 2.51 MB/sec execute 290 sec latency 8.941 ms oleg410-client.virtnet: 1 166539 2.53 MB/sec execute 291 sec latency 9.491 ms oleg410-client.virtnet: 1 167125 2.54 MB/sec execute 292 sec latency 11.421 ms oleg410-client.virtnet: 1 167730 2.53 MB/sec execute 293 sec latency 10.802 ms oleg410-client.virtnet: 1 168389 2.54 MB/sec execute 294 sec latency 10.056 ms oleg410-client.virtnet: 1 169068 2.53 MB/sec execute 295 sec latency 9.756 ms oleg410-client.virtnet: 1 170081 2.55 MB/sec execute 296 sec latency 9.876 ms oleg410-client.virtnet: 1 170632 2.56 MB/sec execute 297 sec latency 11.065 ms oleg410-client.virtnet: 1 171064 2.55 MB/sec execute 298 sec latency 16.244 ms oleg410-client.virtnet: 1 171487 2.55 MB/sec execute 299 sec latency 14.633 ms oleg410-client.virtnet: 1 cleanup 300 sec oleg410-client.virtnet: 0 cleanup 300 sec oleg410-client.virtnet: oleg410-client.virtnet: Operation Count AvgLat MaxLat oleg410-client.virtnet: ---------------------------------------- oleg410-client.virtnet: NTCreateX 24583 7.012 24421.318 oleg410-client.virtnet: Close 18034 1.871 18473.454 oleg410-client.virtnet: Rename 1040 24.664 18337.069 oleg410-client.virtnet: Unlink 4962 1.640 4.736 oleg410-client.virtnet: Qpathinfo 22286 1.270 14.089 oleg410-client.virtnet: Qfileinfo 3887 0.128 2.075 oleg410-client.virtnet: Qfsinfo 4080 0.198 11.049 oleg410-client.virtnet: Sfileinfo 2000 4.084 12.587 oleg410-client.virtnet: Find 8610 0.365 11.578 oleg410-client.virtnet: WriteX 12138 0.747 6.296 oleg410-client.virtnet: ReadX 38519 0.018 1.073 oleg410-client.virtnet: LockX 80 0.805 1.469 oleg410-client.virtnet: UnlockX 80 0.811 1.231 oleg410-client.virtnet: Flush 1721 4.923 449.991 oleg410-client.virtnet: oleg410-client.virtnet: Throughput 2.54796 MB/sec 1 clients 1 procs max_latency=24421.335 ms oleg410-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg410-client.virtnet at Thu Jul 25 00:38:19 EDT 2024 with return code 0 oleg410-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg410-client.virtnet oleg410-client.virtnet: /mnt/lustre/d70b.replay-single/oleg410-client.virtnet /mnt/lustre/d70b.replay-single/oleg410-client.virtnet oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg410-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg410-client.virtnet: removed directory: 'clients/client0' oleg410-client.virtnet: removed directory: 'clients' oleg410-client.virtnet: removed 'client.txt' oleg410-client.virtnet: /mnt/lustre/d70b.replay-single/oleg410-client.virtnet oleg410-client.virtnet: dbench successfully finished