== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 06:19:23 (1715336363) Starting client oleg350-client.virtnet: -o user_xattr,flock oleg350-server@tcp:/lustre /mnt/lustre Started clients oleg350-client.virtnet: 192.168.203.150@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg350-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=21769 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3556144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1952 3592984 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28108 7149128 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 06:19:31 (1715336371) shut down Failover mds1 to oleg350-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:19:45 (1715336385) targets are mounted 06:19:45 (1715336385) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26548 3568428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1728 3590640 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28276 7159068 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg350-server Stopping /mnt/lustre-mds2 (opts:) on oleg350-server 06:19:57 (1715336397) shut down oleg350-client.virtnet: looking for dbench program oleg350-client.virtnet: /usr/bin/dbench oleg350-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg350-client.virtnet oleg350-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg350-client.virtnet' oleg350-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg350-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg350-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg350-client.virtnet at Fri May 10 06:19:24 EDT 2024 oleg350-client.virtnet: waiting for dbench pid 21793 oleg350-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg350-client.virtnet: oleg350-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg350-client.virtnet: failed to create barrier semaphore oleg350-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg350-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg350-client.virtnet: releasing clients oleg350-client.virtnet: 1 660 21.15 MB/sec warmup 1 sec latency 10.123 ms oleg350-client.virtnet: 1 979 10.95 MB/sec warmup 2 sec latency 15.831 ms oleg350-client.virtnet: 1 1662 8.74 MB/sec warmup 3 sec latency 11.122 ms oleg350-client.virtnet: 1 2059 6.66 MB/sec warmup 4 sec latency 320.120 ms oleg350-client.virtnet: 1 2776 6.39 MB/sec warmup 5 sec latency 10.786 ms oleg350-client.virtnet: 1 3766 6.63 MB/sec warmup 6 sec latency 78.460 ms oleg350-client.virtnet: 1 3766 5.68 MB/sec warmup 7 sec latency 1078.703 ms oleg350-client.virtnet: 1 3766 4.97 MB/sec warmup 8 sec latency 2078.919 ms oleg350-client.virtnet: 1 3766 4.42 MB/sec warmup 9 sec latency 3079.176 ms oleg350-client.virtnet: 1 3766 3.98 MB/sec warmup 10 sec latency 4079.432 ms oleg350-client.virtnet: 1 3766 3.61 MB/sec warmup 11 sec latency 5079.656 ms oleg350-client.virtnet: 1 3766 3.31 MB/sec warmup 12 sec latency 6079.815 ms oleg350-client.virtnet: 1 3766 3.06 MB/sec warmup 13 sec latency 7080.045 ms oleg350-client.virtnet: 1 3766 2.84 MB/sec warmup 14 sec latency 8080.210 ms oleg350-client.virtnet: 1 3766 2.65 MB/sec warmup 15 sec latency 9080.387 ms oleg350-client.virtnet: 1 3766 2.48 MB/sec warmup 16 sec latency 10080.631 ms oleg350-client.virtnet: 1 3766 2.34 MB/sec warmup 17 sec latency 11080.793 ms oleg350-client.virtnet: 1 3766 2.21 MB/sec warmup 18 sec latency 12080.937 ms oleg350-client.virtnet: 1 3766 2.09 MB/sec warmup 19 sec latency 13081.037 ms oleg350-client.virtnet: 1 3766 1.99 MB/sec warmup 20 sec latency 14081.167 ms oleg350-client.virtnet: 1 3766 1.89 MB/sec warmup 21 sec latency 15081.307 ms oleg350-client.virtnet: 1 3766 1.81 MB/sec warmup 22 sec latency 16081.539 ms oleg350-client.virtnet: 1 3766 1.73 MB/sec warmup 23 sec latency 17081.759 ms oleg350-client.virtnet: 1 3766 1.66 MB/sec warmup 24 sec latency 18081.940 ms oleg350-client.virtnet: 1 4116 1.60 MB/sec warmup 25 sec latency 18265.092 ms oleg350-client.virtnet: 1 4650 1.62 MB/sec warmup 26 sec latency 16.338 ms oleg350-client.virtnet: 1 5284 1.68 MB/sec warmup 27 sec latency 11.911 ms oleg350-client.virtnet: 1 5947 1.68 MB/sec warmup 28 sec latency 7.639 ms oleg350-client.virtnet: 1 6825 1.88 MB/sec warmup 29 sec latency 12.056 ms oleg350-client.virtnet: 1 7471 1.97 MB/sec warmup 30 sec latency 9.333 ms oleg350-client.virtnet: 1 8070 1.94 MB/sec warmup 31 sec latency 13.873 ms oleg350-client.virtnet: 1 8666 2.01 MB/sec warmup 32 sec latency 12.861 ms oleg350-client.virtnet: 1 9105 1.96 MB/sec warmup 33 sec latency 11.Failover mds2 to oleg350-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 06:20:11 (1715336411) targets are mounted 06:20:11 (1715336411) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36572 3567516 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13052 3592976 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7160492 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 06:20:22 (1715336422) shut down Failover mds1 to oleg350-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:20:36 (1715336436) targets are mounted 06:20:36 (1715336436) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 719 ms oleg350-client.virtnet: 1 9572 1.97 MB/sec warmup 34 sec latency 12.170 ms oleg350-client.virtnet: 1 10345 2.09 MB/sec warmup 35 sec latency 16.976 ms oleg350-client.virtnet: 1 10934 2.17 MB/sec warmup 36 sec latency 10.006 ms oleg350-client.virtnet: 1 11504 2.13 MB/sec warmup 37 sec latency 10.740 ms oleg350-client.virtnet: 1 11870 2.12 MB/sec warmup 38 sec latency 14.478 ms oleg350-client.virtnet: 1 12287 2.14 MB/sec warmup 39 sec latency 15.656 ms oleg350-client.virtnet: 1 12689 2.10 MB/sec warmup 40 sec latency 15.761 ms oleg350-client.virtnet: 1 13226 2.10 MB/sec warmup 41 sec latency 11.572 ms oleg350-client.virtnet: 1 14149 2.28 MB/sec warmup 42 sec latency 12.678 ms oleg350-client.virtnet: 1 14518 2.26 MB/sec warmup 43 sec latency 13.474 ms oleg350-client.virtnet: 1 14938 2.22 MB/sec warmup 44 sec latency 13.777 ms oleg350-client.virtnet: 1 15370 2.21 MB/sec warmup 45 sec latency 83.521 ms oleg350-client.virtnet: 1 15370 2.16 MB/sec warmup 46 sec latency 1083.719 ms oleg350-client.virtnet: 1 15370 2.12 MB/sec warmup 47 sec latency 2083.898 ms oleg350-client.virtnet: 1 15370 2.07 MB/sec warmup 48 sec latency 3084.145 ms oleg350-client.virtnet: 1 15370 2.03 MB/sec warmup 49 sec latency 4084.346 ms oleg350-client.virtnet: 1 15417 1.99 MB/sec warmup 50 sec latency 4976.238 ms oleg350-client.virtnet: 1 15932 2.01 MB/sec warmup 51 sec latency 15.782 ms oleg350-client.virtnet: 1 16591 2.01 MB/sec warmup 52 sec latency 9.488 ms oleg350-client.virtnet: 1 17546 2.11 MB/sec warmup 53 sec latency 10.929 ms oleg350-client.virtnet: 1 18176 2.16 MB/sec warmup 54 sec latency 9.271 ms oleg350-client.virtnet: 1 18524 2.13 MB/sec warmup 55 sec latency 405.574 ms oleg350-client.virtnet: 1 18974 2.12 MB/sec warmup 56 sec latency 16.021 ms oleg350-client.virtnet: 1 19596 2.14 MB/sec warmup 57 sec latency 12.893 ms oleg350-client.virtnet: 1 19674 2.10 MB/sec warmup 58 sec latency 921.169 ms oleg350-client.virtnet: 1 19674 2.07 MB/sec warmup 59 sec latency 1921.420 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 1 sec latency 3921.867 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 2 sec latency 4922.157 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 3 sec latency 5922.468 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 4 sec latency 6922.775 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 5 sec latency 7923.006 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 6 sec latency 8923.252 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 7 sec latency 9923.506 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 8 sec latency 10923.808 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 9 sec latency 11923.984 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 10 sec latency 12924.117 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 11 sec latency 13924.297 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 12 sec latency 14924.526 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 13 sec latency 15924.699 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 14 sec latency 16924.898 ms oleg350-client.virtnet: 1 19674 0.00 MB/sec execute 15 sec latency 17925.107 ms oleg350-client.virtnet: 1 19831 0.01 MB/sec execute 16 sec latency 18502.993 ms oleg350-client.virtnet: 1 20237 0.10 MB/sec execute 17 sec latency 15.155 ms oleg350-client.virtnet: 1 20950 0.43 MB/sec execute 18 sec latency 15.107 ms oleg350-client.virtnet: 1 21441 0.65 MB/sec exUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36584 3567604 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13040 3593044 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7160648 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg350-server Stopping /mnt/lustre-mds2 (opts:) on oleg350-server 06:20:49 (1715336449) shut down Failover mds2 to oleg350-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 06:21:04 (1715336464) targets are mounted 06:21:04 (1715336464) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36880 3569488 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12936 3593168 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49816 7162656 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 06:21:16 (1715336476) shut down Failover mds1 to oleg350-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 ecute 19 sec latency 15.723 ms oleg350-client.virtnet: 1 21708 0.68 MB/sec execute 20 sec latency 19.295 ms oleg350-client.virtnet: 1 22031 0.66 MB/sec execute 21 sec latency 18.856 ms oleg350-client.virtnet: 1 22362 0.71 MB/sec execute 22 sec latency 16.212 ms oleg350-client.virtnet: 1 22778 0.81 MB/sec execute 23 sec latency 16.136 ms oleg350-client.virtnet: 1 23158 0.79 MB/sec execute 24 sec latency 12.908 ms oleg350-client.virtnet: 1 23712 0.83 MB/sec execute 25 sec latency 11.316 ms oleg350-client.virtnet: 1 24234 1.00 MB/sec execute 26 sec latency 16.383 ms oleg350-client.virtnet: 1 24960 1.16 MB/sec execute 27 sec latency 16.468 ms oleg350-client.virtnet: 1 25499 1.17 MB/sec execute 28 sec latency 10.325 ms oleg350-client.virtnet: 1 26144 1.21 MB/sec execute 29 sec latency 10.665 ms oleg350-client.virtnet: 1 26715 1.27 MB/sec execute 30 sec latency 14.869 ms oleg350-client.virtnet: 1 27265 1.28 MB/sec execute 31 sec latency 12.733 ms oleg350-client.virtnet: 1 28080 1.44 MB/sec execute 32 sec latency 14.288 ms oleg350-client.virtnet: 1 28803 1.57 MB/sec execute 33 sec latency 9.701 ms oleg350-client.virtnet: 1 29362 1.55 MB/sec execute 34 sec latency 14.317 ms oleg350-client.virtnet: 1 29928 1.63 MB/sec execute 35 sec latency 15.539 ms oleg350-client.virtnet: 1 30583 1.60 MB/sec execute 36 sec latency 8.453 ms oleg350-client.virtnet: 1 31011 1.62 MB/sec execute 37 sec latency 15.938 ms oleg350-client.virtnet: 1 31964 1.83 MB/sec execute 38 sec latency 14.802 ms oleg350-client.virtnet: 1 32374 1.82 MB/sec execute 39 sec latency 15.390 ms oleg350-client.virtnet: 1 32536 1.78 MB/sec execute 40 sec latency 404.210 ms oleg350-client.virtnet: 1 32536 1.73 MB/sec execute 41 sec latency 1404.492 ms oleg350-client.virtnet: 1 32536 1.69 MB/sec execute 42 sec latency 2404.707 ms oleg350-client.virtnet: 1 32593 1.65 MB/sec execute 43 sec latency 3326.370 ms oleg350-client.virtnet: 1 33023 1.66 MB/sec execute 44 sec latency 15.856 ms oleg350-client.virtnet: 1 33638 1.70 MB/sec execute 45 sec latency 14.459 ms oleg350-client.virtnet: 1 33980 1.67 MB/sec execute 46 sec latency 18.349 ms oleg350-client.virtnet: 1 34392 1.66 MB/sec execute 47 sec latency 12.956 ms oleg350-client.virtnet: 1 35307 1.78 MB/sec execute 48 sec latency 15.587 ms oleg350-client.virtnet: 1 35938 1.84 MB/sec execute 49 sec latency 10.056 ms oleg350-client.virtnet: 1 36488 1.82 MB/sec execute 50 sec latency 14.039 ms oleg350-client.virtnet: 1 36989 1.87 MB/sec execute 51 sec latency 356.135 ms oleg350-client.virtnet: 1 36989 1.83 MB/sec execute 52 sec latency 1356.371 ms oleg350-client.virtnet: 1 36989 1.80 MB/sec execute 53 sec latency 2356.660 ms oleg350-client.virtnet: 1 36989 1.76 MB/sec execute 54 sec latency 3356.918 ms oleg350-client.virtnet: 1 36989 1.73 MB/sec execute 55 sec latency 4357.179 ms oleg350-client.virtnet: 1 36989 1.70 MB/sec execute 56 sec latency 5357.395 ms oleg350-client.virtnet: 1 36989 1.67 MB/sec execute 57 sec latency 6357.723 ms oleg350-client.virtnet: 1 36989 1.64 MB/sec execute 58 sec latency 7358.012 ms oleg350-client.virtnet: 1 36989 1.61 MB/sec execute 59 sec latency 8358.252 ms oleg350-client.virtnet: 1 36989 1.59 MB/sec execute 60 sec latency 9358.620 ms oleg350-client.virtnet: 1 36989 1.56 MB/sec execute 61 sec latency 10358.884 ms oleg350-client.virtnet: 1 36989 1.54 MB/sec execute 62 sec latency 11359.037 ms oleg350-client.virtnet: 1 36989 1.51 MB/sec execute 63 sec latency 12359.243 ms oleg350-client.virtnet: oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:21:31 (1715336491) targets are mounted 06:21:31 (1715336491) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36644 3569108 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12924 3593032 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49568 7162140 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg350-server Stopping /mnt/lustre-mds2 (opts:) on oleg350-server 06:21:44 (1715336504) shut down Failover mds2 to oleg350-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 06:21:59 (1715336519) targets are mounted 06:21:59 (1715336519) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36572 3567692 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13040 3593848 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7161540 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 06:22:11 (1715336531) shut down 1 36989 1.49 MB/sec execute 64 sec latency 13359.358 ms oleg350-client.virtnet: 1 36989 1.47 MB/sec execute 65 sec latency 14359.529 ms oleg350-client.virtnet: 1 36989 1.44 MB/sec execute 66 sec latency 15359.763 ms oleg350-client.virtnet: 1 36989 1.42 MB/sec execute 67 sec latency 16360.048 ms oleg350-client.virtnet: 1 36989 1.40 MB/sec execute 68 sec latency 17360.382 ms oleg350-client.virtnet: 1 36989 1.38 MB/sec execute 69 sec latency 18360.731 ms oleg350-client.virtnet: 1 36989 1.36 MB/sec execute 70 sec latency 19360.929 ms oleg350-client.virtnet: 1 37363 1.35 MB/sec execute 71 sec latency 19536.156 ms oleg350-client.virtnet: 1 37998 1.36 MB/sec execute 72 sec latency 13.180 ms oleg350-client.virtnet: 1 38675 1.42 MB/sec execute 73 sec latency 21.864 ms oleg350-client.virtnet: 1 39178 1.46 MB/sec execute 74 sec latency 20.792 ms oleg350-client.virtnet: 1 39490 1.46 MB/sec execute 75 sec latency 32.158 ms oleg350-client.virtnet: 1 39832 1.44 MB/sec execute 76 sec latency 17.155 ms oleg350-client.virtnet: 1 40260 1.45 MB/sec execute 77 sec latency 16.431 ms oleg350-client.virtnet: 1 40895 1.47 MB/sec execute 78 sec latency 11.584 ms oleg350-client.virtnet: 1 41401 1.47 MB/sec execute 79 sec latency 17.775 ms oleg350-client.virtnet: 1 41830 1.50 MB/sec execute 80 sec latency 16.399 ms oleg350-client.virtnet: 1 42718 1.57 MB/sec execute 81 sec latency 10.658 ms oleg350-client.virtnet: 1 42990 1.57 MB/sec execute 82 sec latency 18.760 ms oleg350-client.virtnet: 1 43327 1.55 MB/sec execute 83 sec latency 21.707 ms oleg350-client.virtnet: 1 43801 1.55 MB/sec execute 84 sec latency 14.937 ms oleg350-client.virtnet: 1 44384 1.57 MB/sec execute 85 sec latency 11.995 ms oleg350-client.virtnet: 1 44825 1.56 MB/sec execute 86 sec latency 12.017 ms oleg350-client.virtnet: 1 45475 1.60 MB/sec execute 87 sec latency 14.601 ms oleg350-client.virtnet: 1 46161 1.66 MB/sec execute 88 sec latency 15.740 ms oleg350-client.virtnet: 1 46508 1.66 MB/sec execute 89 sec latency 14.949 ms oleg350-client.virtnet: 1 46928 1.64 MB/sec execute 90 sec latency 15.005 ms oleg350-client.virtnet: 1 47281 1.64 MB/sec execute 91 sec latency 16.654 ms oleg350-client.virtnet: 1 47692 1.66 MB/sec execute 92 sec latency 15.678 ms oleg350-client.virtnet: 1 48127 1.65 MB/sec execute 93 sec latency 17.818 ms oleg350-client.virtnet: 1 48670 1.65 MB/sec execute 94 sec latency 13.765 ms oleg350-client.virtnet: 1 49133 1.68 MB/sec execute 95 sec latency 328.211 ms oleg350-client.virtnet: 1 49133 1.67 MB/sec execute 96 sec latency 1328.520 ms oleg350-client.virtnet: 1 49133 1.65 MB/sec execute 97 sec latency 2329.016 ms oleg350-client.virtnet: 1 49427 1.65 MB/sec execute 98 sec latency 3025.392 ms oleg350-client.virtnet: 1 49932 1.68 MB/sec execute 99 sec latency 14.726 ms oleg350-client.virtnet: 1 50177 1.67 MB/sec execute 100 sec latency 15.883 ms oleg350-client.virtnet: 1 50575 1.66 MB/sec execute 101 sec latency 22.105 ms oleg350-client.virtnet: 1 51103 1.68 MB/sec execute 102 sec latency 16.380 ms oleg350-client.virtnet: 1 51499 1.67 MB/sec execute 103 sec latency 15.626 ms oleg350-client.virtnet: 1 51818 1.66 MB/sec execute 104 sec latency 341.025 ms oleg350-client.virtnet: 1 52188 1.66 MB/sec execute 105 sec latency 17.357 ms oleg350-client.virtnet: 1 52899 1.70 MB/sec execute 106 sec latency 15.788 ms oleg350-client.virtnet: 1 53052 1.69 MB/sec execute 107 sec latency 785.122 ms oleg350-client.virtnet: 1 53052 1.68 MB/sec execute 108 sec latency 1785Failover mds1 to oleg350-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:22:27 (1715336547) targets are mounted 06:22:27 (1715336547) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38884 3567240 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12916 3593256 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 51800 7160496 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg350-server Stopping /mnt/lustre-mds2 (opts:) on oleg350-server 06:22:38 (1715336558) shut down Failover mds2 to oleg350-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 06:22:53 (1715336573) targets are mounted 06:22:53 (1715336573) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec .403 ms oleg350-client.virtnet: 1 53052 1.66 MB/sec execute 109 sec latency 2785.667 ms oleg350-client.virtnet: 1 53052 1.65 MB/sec execute 110 sec latency 3786.063 ms oleg350-client.virtnet: 1 53052 1.63 MB/sec execute 111 sec latency 4786.443 ms oleg350-client.virtnet: 1 53052 1.62 MB/sec execute 112 sec latency 5786.717 ms oleg350-client.virtnet: 1 53052 1.60 MB/sec execute 113 sec latency 6787.046 ms oleg350-client.virtnet: 1 53052 1.59 MB/sec execute 114 sec latency 7787.337 ms oleg350-client.virtnet: 1 53052 1.57 MB/sec execute 115 sec latency 8787.648 ms oleg350-client.virtnet: 1 53052 1.56 MB/sec execute 116 sec latency 9787.994 ms oleg350-client.virtnet: 1 53052 1.55 MB/sec execute 117 sec latency 10788.216 ms oleg350-client.virtnet: 1 53052 1.53 MB/sec execute 118 sec latency 11788.426 ms oleg350-client.virtnet: 1 53052 1.52 MB/sec execute 119 sec latency 12788.609 ms oleg350-client.virtnet: 1 53052 1.51 MB/sec execute 120 sec latency 13788.792 ms oleg350-client.virtnet: 1 53052 1.50 MB/sec execute 121 sec latency 14789.058 ms oleg350-client.virtnet: 1 53052 1.48 MB/sec execute 122 sec latency 15789.230 ms oleg350-client.virtnet: 1 53052 1.47 MB/sec execute 123 sec latency 16789.491 ms oleg350-client.virtnet: 1 53052 1.46 MB/sec execute 124 sec latency 17789.774 ms oleg350-client.virtnet: 1 53052 1.45 MB/sec execute 125 sec latency 18790.038 ms oleg350-client.virtnet: 1 53373 1.46 MB/sec execute 126 sec latency 19373.159 ms oleg350-client.virtnet: 1 53829 1.46 MB/sec execute 127 sec latency 10.748 ms oleg350-client.virtnet: 1 54344 1.47 MB/sec execute 128 sec latency 18.627 ms oleg350-client.virtnet: 1 54902 1.48 MB/sec execute 129 sec latency 10.528 ms oleg350-client.virtnet: 1 55497 1.47 MB/sec execute 130 sec latency 9.811 ms oleg350-client.virtnet: 1 56248 1.51 MB/sec execute 131 sec latency 11.313 ms oleg350-client.virtnet: 1 57047 1.55 MB/sec execute 132 sec latency 14.828 ms oleg350-client.virtnet: 1 57571 1.54 MB/sec execute 133 sec latency 19.171 ms oleg350-client.virtnet: 1 58242 1.57 MB/sec execute 134 sec latency 10.491 ms oleg350-client.virtnet: 1 58931 1.56 MB/sec execute 135 sec latency 8.545 ms oleg350-client.virtnet: 1 59775 1.60 MB/sec execute 136 sec latency 12.066 ms oleg350-client.virtnet: 1 60688 1.63 MB/sec execute 137 sec latency 9.312 ms oleg350-client.virtnet: 1 61308 1.63 MB/sec execute 138 sec latency 10.853 ms oleg350-client.virtnet: 1 61956 1.65 MB/sec execute 139 sec latency 12.218 ms oleg350-client.virtnet: 1 62623 1.65 MB/sec execute 140 sec latency 8.858 ms oleg350-client.virtnet: 1 63237 1.67 MB/sec execute 141 sec latency 17.502 ms oleg350-client.virtnet: 1 63892 1.70 MB/sec execute 142 sec latency 33.858 ms oleg350-client.virtnet: 1 64234 1.70 MB/sec execute 143 sec latency 14.771 ms oleg350-client.virtnet: 1 64580 1.69 MB/sec execute 144 sec latency 14.918 ms oleg350-client.virtnet: 1 64907 1.69 MB/sec execute 145 sec latency 17.094 ms oleg350-client.virtnet: 1 65339 1.70 MB/sec execute 146 sec latency 15.147 ms oleg350-client.virtnet: 1 65677 1.69 MB/sec execute 147 sec latency 17.619 ms oleg350-client.virtnet: 1 66248 1.69 MB/sec execute 148 sec latency 11.511 ms oleg350-client.virtnet: 1 66424 1.68 MB/sec execute 149 sec latency 619.829 ms oleg350-client.virtnet: 1 66424 1.67 MB/sec execute 150 sec latency 1620.091 ms oleg350-client.virtnet: 1 66424 1.66 MB/sec execute 151 sec latency 2620.293 ms oleg350-client.virtnet: 1 66424 1.65 MB/sec execute 152 sec latency 3620.510 ms oleg350-client.virtnet: 1 67UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36584 3567504 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13052 3592976 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160480 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 06:23:05 (1715336585) shut down Failover mds1 to oleg350-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:23:21 (1715336601) targets are mounted 06:23:21 (1715336601) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36560 3569948 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12916 3593084 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49476 7163032 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg350-server Stopping /mnt/lustre-mds2 (opts:) on oleg350-server 06:23:32 (1715336612) shut down 385 1.70 MB/sec execute 153 sec latency 3807.000 ms oleg350-client.virtnet: 1 67820 1.70 MB/sec execute 154 sec latency 15.129 ms oleg350-client.virtnet: 1 68140 1.69 MB/sec execute 155 sec latency 22.393 ms oleg350-client.virtnet: 1 68556 1.69 MB/sec execute 156 sec latency 14.820 ms oleg350-client.virtnet: 1 69201 1.70 MB/sec execute 157 sec latency 10.492 ms oleg350-client.virtnet: 1 69547 1.70 MB/sec execute 158 sec latency 83.074 ms oleg350-client.virtnet: 1 69894 1.70 MB/sec execute 159 sec latency 15.706 ms oleg350-client.virtnet: 1 70587 1.72 MB/sec execute 160 sec latency 16.805 ms oleg350-client.virtnet: 1 70687 1.72 MB/sec execute 161 sec latency 823.722 ms oleg350-client.virtnet: 1 70687 1.71 MB/sec execute 162 sec latency 1823.856 ms oleg350-client.virtnet: 1 70687 1.70 MB/sec execute 163 sec latency 2824.133 ms oleg350-client.virtnet: 1 70687 1.69 MB/sec execute 164 sec latency 3824.390 ms oleg350-client.virtnet: 1 70687 1.68 MB/sec execute 165 sec latency 4824.646 ms oleg350-client.virtnet: 1 70687 1.67 MB/sec execute 166 sec latency 5824.924 ms oleg350-client.virtnet: 1 70687 1.66 MB/sec execute 167 sec latency 6825.203 ms oleg350-client.virtnet: 1 70687 1.65 MB/sec execute 168 sec latency 7825.497 ms oleg350-client.virtnet: 1 70687 1.64 MB/sec execute 169 sec latency 8825.714 ms oleg350-client.virtnet: 1 70687 1.63 MB/sec execute 170 sec latency 9825.954 ms oleg350-client.virtnet: 1 70687 1.62 MB/sec execute 171 sec latency 10826.181 ms oleg350-client.virtnet: 1 70687 1.61 MB/sec execute 172 sec latency 11826.374 ms oleg350-client.virtnet: 1 70687 1.60 MB/sec execute 173 sec latency 12826.560 ms oleg350-client.virtnet: 1 70687 1.59 MB/sec execute 174 sec latency 13826.798 ms oleg350-client.virtnet: 1 70687 1.58 MB/sec execute 175 sec latency 14826.982 ms oleg350-client.virtnet: 1 70687 1.57 MB/sec execute 176 sec latency 15827.143 ms oleg350-client.virtnet: 1 70687 1.56 MB/sec execute 177 sec latency 16827.436 ms oleg350-client.virtnet: 1 70687 1.55 MB/sec execute 178 sec latency 17827.816 ms oleg350-client.virtnet: 1 70687 1.55 MB/sec execute 179 sec latency 18828.076 ms oleg350-client.virtnet: 1 70945 1.55 MB/sec execute 180 sec latency 19573.026 ms oleg350-client.virtnet: 1 71396 1.55 MB/sec execute 181 sec latency 10.215 ms oleg350-client.virtnet: 1 71823 1.55 MB/sec execute 182 sec latency 15.423 ms oleg350-client.virtnet: 1 72350 1.56 MB/sec execute 183 sec latency 12.757 ms oleg350-client.virtnet: 1 72825 1.56 MB/sec execute 184 sec latency 10.153 ms oleg350-client.virtnet: 1 73475 1.56 MB/sec execute 185 sec latency 10.001 ms oleg350-client.virtnet: 1 74358 1.59 MB/sec execute 186 sec latency 13.977 ms oleg350-client.virtnet: 1 74930 1.61 MB/sec execute 187 sec latency 10.167 ms oleg350-client.virtnet: 1 75539 1.60 MB/sec execute 188 sec latency 10.460 ms oleg350-client.virtnet: 1 76054 1.62 MB/sec execute 189 sec latency 16.677 ms oleg350-client.virtnet: 1 76387 1.61 MB/sec execute 190 sec latency 17.288 ms oleg350-client.virtnet: 1 76872 1.61 MB/sec execute 191 sec latency 11.004 ms oleg350-client.virtnet: 1 77311 1.62 MB/sec execute 192 sec latency 16.187 ms oleg350-client.virtnet: 1 77922 1.63 MB/sec execute 193 sec latency 14.985 ms oleg350-client.virtnet: 1 78350 1.65 MB/sec execute 194 sec latency 16.482 ms oleg350-client.virtnet: 1 78614 1.64 MB/sec execute 195 sec latency 15.239 ms oleg350-client.virtnet: 1 79228 1.64 MB/sec execute 196 sec latency 18.990 ms oleg350-client.virtnet: 1 79781 1.65 MB/sec execute 197 sec latFailover mds2 to oleg350-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 06:23:48 (1715336628) targets are mounted 06:23:48 (1715336628) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36560 3569736 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12916 3593104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49476 7162840 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 06:23:59 (1715336639) shut down Failover mds1 to oleg350-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:24:14 (1715336654) targets are mounted 06:24:14 (1715336654) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36576 3569868 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12924 3593104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162972 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg350-server Stopping /mnt/lustre-mds2 (opts:) on oleg350-server ency 17.764 ms oleg350-client.virtnet: 1 80173 1.64 MB/sec execute 198 sec latency 15.388 ms oleg350-client.virtnet: 1 80741 1.65 MB/sec execute 199 sec latency 11.209 ms oleg350-client.virtnet: 1 81481 1.67 MB/sec execute 200 sec latency 15.333 ms oleg350-client.virtnet: 1 81886 1.68 MB/sec execute 201 sec latency 16.090 ms oleg350-client.virtnet: 1 82122 1.68 MB/sec execute 202 sec latency 16.807 ms oleg350-client.virtnet: 1 82198 1.67 MB/sec execute 203 sec latency 789.798 ms oleg350-client.virtnet: 1 82198 1.66 MB/sec execute 204 sec latency 1790.075 ms oleg350-client.virtnet: 1 82198 1.65 MB/sec execute 205 sec latency 2790.350 ms oleg350-client.virtnet: 1 82198 1.64 MB/sec execute 206 sec latency 3790.594 ms oleg350-client.virtnet: 1 82516 1.64 MB/sec execute 207 sec latency 3947.364 ms oleg350-client.virtnet: 1 82844 1.64 MB/sec execute 208 sec latency 15.636 ms oleg350-client.virtnet: 1 83244 1.65 MB/sec execute 209 sec latency 15.442 ms oleg350-client.virtnet: 1 83754 1.64 MB/sec execute 210 sec latency 16.159 ms oleg350-client.virtnet: 1 84512 1.66 MB/sec execute 211 sec latency 11.225 ms oleg350-client.virtnet: 1 85413 1.69 MB/sec execute 212 sec latency 12.978 ms oleg350-client.virtnet: 1 85747 1.68 MB/sec execute 213 sec latency 14.867 ms oleg350-client.virtnet: 1 86350 1.68 MB/sec execute 214 sec latency 77.436 ms oleg350-client.virtnet: 1 86350 1.67 MB/sec execute 215 sec latency 1077.711 ms oleg350-client.virtnet: 1 86350 1.67 MB/sec execute 216 sec latency 2077.970 ms oleg350-client.virtnet: 1 86350 1.66 MB/sec execute 217 sec latency 3078.237 ms oleg350-client.virtnet: 1 86350 1.65 MB/sec execute 218 sec latency 4078.558 ms oleg350-client.virtnet: 1 86350 1.64 MB/sec execute 219 sec latency 5078.843 ms oleg350-client.virtnet: 1 86350 1.64 MB/sec execute 220 sec latency 6079.005 ms oleg350-client.virtnet: 1 86350 1.63 MB/sec execute 221 sec latency 7079.219 ms oleg350-client.virtnet: 1 86350 1.62 MB/sec execute 222 sec latency 8079.425 ms oleg350-client.virtnet: 1 86350 1.61 MB/sec execute 223 sec latency 9079.585 ms oleg350-client.virtnet: 1 86350 1.61 MB/sec execute 224 sec latency 10079.799 ms oleg350-client.virtnet: 1 86350 1.60 MB/sec execute 225 sec latency 11079.981 ms oleg350-client.virtnet: 1 86350 1.59 MB/sec execute 226 sec latency 12080.171 ms oleg350-client.virtnet: 1 86350 1.59 MB/sec execute 227 sec latency 13080.332 ms oleg350-client.virtnet: 1 86350 1.58 MB/sec execute 228 sec latency 14080.705 ms oleg350-client.virtnet: 1 86350 1.57 MB/sec execute 229 sec latency 15080.949 ms oleg350-client.virtnet: 1 86350 1.56 MB/sec execute 230 sec latency 16081.186 ms oleg350-client.virtnet: 1 86350 1.56 MB/sec execute 231 sec latency 17081.471 ms oleg350-client.virtnet: 1 86350 1.55 MB/sec execute 232 sec latency 18081.745 ms oleg350-client.virtnet: 1 86350 1.54 MB/sec execute 233 sec latency 19082.045 ms oleg350-client.virtnet: 1 86640 1.55 MB/sec execute 234 sec latency 19594.140 ms oleg350-client.virtnet: 1 87125 1.55 MB/sec execute 235 sec latency 12.845 ms oleg350-client.virtnet: 1 87618 1.55 MB/sec execute 236 sec latency 17.004 ms oleg350-client.virtnet: 1 88364 1.57 MB/sec execute 237 sec latency 14.944 ms oleg350-client.virtnet: 1 88986 1.58 MB/sec execute 238 sec latency 13.940 ms oleg350-client.virtnet: 1 89447 1.58 MB/sec execute 239 sec latency 15.850 ms oleg350-client.virtnet: 1 89858 1.58 MB/sec execute 240 sec latency 14.813 ms oleg350-client.virtnet: 1 90323 1.59 MB/sec execute 241 sec latency 12.346 ms oleg350-client.vi06:24:27 (1715336667) shut down Failover mds2 to oleg350-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 06:24:41 (1715336681) targets are mounted 06:24:41 (1715336681) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec rtnet: 1 90868 1.58 MB/sec execute 242 sec latency 16.200 ms oleg350-client.virtnet: 1 91378 1.59 MB/sec execute 243 sec latency 15.391 ms oleg350-client.virtnet: 1 92384 1.62 MB/sec execute 244 sec latency 13.338 ms oleg350-client.virtnet: 1 92671 1.62 MB/sec execute 245 sec latency 16.141 ms oleg350-client.virtnet: 1 93172 1.61 MB/sec execute 246 sec latency 14.965 ms oleg350-client.virtnet: 1 93665 1.62 MB/sec execute 247 sec latency 14.163 ms oleg350-client.virtnet: 1 93964 1.62 MB/sec execute 248 sec latency 18.638 ms oleg350-client.virtnet: 1 94329 1.61 MB/sec execute 249 sec latency 14.507 ms oleg350-client.virtnet: 1 94720 1.61 MB/sec execute 250 sec latency 15.508 ms oleg350-client.virtnet: 1 95436 1.63 MB/sec execute 251 sec latency 15.456 ms oleg350-client.virtnet: 1 95941 1.64 MB/sec execute 252 sec latency 15.425 ms oleg350-client.virtnet: 1 96228 1.64 MB/sec execute 253 sec latency 16.264 ms oleg350-client.virtnet: 1 96548 1.64 MB/sec execute 254 sec latency 21.154 ms oleg350-client.virtnet: 1 96996 1.64 MB/sec execute 255 sec latency 16.021 ms oleg350-client.virtnet: 1 97533 1.64 MB/sec execute 256 sec latency 15.659 ms oleg350-client.virtnet: 1 98235 1.64 MB/sec execute 257 sec latency 10.307 ms oleg350-client.virtnet: 1 98264 1.64 MB/sec execute 258 sec latency 936.531 ms oleg350-client.virtnet: 1 98264 1.63 MB/sec execute 259 sec latency 1936.765 ms oleg350-client.virtnet: 1 98264 1.62 MB/sec execute 260 sec latency 2936.893 ms oleg350-client.virtnet: 1 98745 1.64 MB/sec execute 261 sec latency 3376.924 ms oleg350-client.virtnet: 1 99597 1.66 MB/sec execute 262 sec latency 12.618 ms oleg350-client.virtnet: 1 100189 1.65 MB/sec execute 263 sec latency 11.440 ms oleg350-client.virtnet: 1 100943 1.66 MB/sec execute 264 sec latency 9.688 ms oleg350-client.virtnet: 1 101729 1.67 MB/sec execute 265 sec latency 8.717 ms oleg350-client.virtnet: 1 102895 1.70 MB/sec execute 266 sec latency 7.910 ms oleg350-client.virtnet: 1 103481 1.70 MB/sec execute 267 sec latency 8.008 ms oleg350-client.virtnet: 1 104171 1.70 MB/sec execute 268 sec latency 13.543 ms oleg350-client.virtnet: 1 104913 1.71 MB/sec execute 269 sec latency 9.460 ms oleg350-client.virtnet: 1 105838 1.73 MB/sec execute 270 sec latency 8.750 ms oleg350-client.virtnet: 1 106798 1.74 MB/sec execute 271 sec latency 7.876 ms oleg350-client.virtnet: 1 107430 1.74 MB/sec execute 272 sec latency 8.669 ms oleg350-client.virtnet: 1 108018 1.75 MB/sec execute 273 sec latency 14.000 ms oleg350-client.virtnet: 1 108340 1.74 MB/sec execute 274 sec latency 16.267 ms oleg350-client.virtnet: 1 108825 1.74 MB/sec execute 275 sec latency 11.221 ms oleg350-client.virtnet: 1 109384 1.76 MB/sec execute 276 sec latency 16.282 ms oleg350-client.virtnet: 1 110022 1.77 MB/sec execute 277 sec latency 16.197 ms oleg350-client.virtnet: 1 110389 1.77 MB/sec execute 278 sec latency 16.108 ms oleg350-client.virtnet: 1 110907 1.76 MB/sec execute 279 sec latency 13.068 ms oleg350-client.virtnet: 1 111516 1.77 MB/sec execute 280 sec latency 15.283 ms oleg350-client.virtnet: 1 112120 1.77 MB/sec execute 281 sec latency 10.857 ms oleg350-client.virtnet: 1 112479 1.77 MB/sec execute 282 sec latency 16.176 ms oleg350-client.virtnet: 1 113020 1.78 MB/sec execute 283 sec latency 17.829 ms oleg350-client.virtnet: 1 113753 1.80 MB/sec execute 284 sec latency 13.860 ms oleg350-client.virtnet: 1 114025 1.79 MB/sec execute 285 sec latency 15.602 ms oleg350-client.virtnet: 1 114529 1.79 MB/sec execute 286 sec latency 14.822 ms oleg350-client.virtnet: 1 114976 1.80 MB/sec execute 287 sec latency 16.231 ms oleg350-client.virtnet: 1 115306 1.79 MB/sec execute 288 sec latency 16.918 ms oleg350-client.virtnet: 1 115961 1.79 MB/sec execute 289 sec latency 10.368 ms oleg350-client.virtnet: 1 116892 1.81 MB/sec execute 290 sec latency 15.217 ms oleg350-client.virtnet: 1 117568 1.82 MB/sec execute 291 sec latency 7.689 ms oleg350-client.virtnet: 1 118256 1.82 MB/sec execute 292 sec latency 11.223 ms oleg350-client.virtnet: 1 118980 1.83 MB/sec execute 293 sec latency 9.509 ms oleg350-client.virtnet: 1 119845 1.84 MB/sec execute 294 sec latency 8.803 ms oleg350-client.virtnet: 1 120850 1.86 MB/sec execute 295 sec latency 15.356 ms oleg350-client.virtnet: 1 121126 1.85 MB/sec execute 296 sec latency 15.823 ms oleg350-client.virtnet: 1 121598 1.85 MB/sec execute 297 sec latency 24.691 ms oleg350-client.virtnet: 1 122050 1.86 MB/sec execute 298 sec latency 17.730 ms oleg350-client.virtnet: 1 122416 1.85 MB/sec execute 299 sec latency 16.394 ms oleg350-client.virtnet: 1 cleanup 300 sec oleg350-client.virtnet: 0 cleanup 300 sec oleg350-client.virtnet: oleg350-client.virtnet: Operation Count AvgLat MaxLat oleg350-client.virtnet: ---------------------------------------- oleg350-client.virtnet: NTCreateX 17845 8.374 19536.141 oleg350-client.virtnet: Close 13130 4.151 19594.115 oleg350-client.virtnet: Rename 754 8.447 18.616 oleg350-client.virtnet: Unlink 3600 3.310 3947.349 oleg350-client.virtnet: Qpathinfo 16182 2.796 19573.008 oleg350-client.virtnet: Qfileinfo 2819 0.167 2.467 oleg350-client.virtnet: Qfsinfo 2958 1.402 3376.900 oleg350-client.virtnet: Sfileinfo 1450 4.273 11.594 oleg350-client.virtnet: Find 6248 0.996 3326.352 oleg350-client.virtnet: WriteX 8808 1.026 8.787 oleg350-client.virtnet: ReadX 27962 0.022 2.389 oleg350-client.virtnet: LockX 58 1.446 2.647 oleg350-client.virtnet: UnlockX 58 1.470 2.163 oleg350-client.virtnet: Flush 1247 6.304 341.001 oleg350-client.virtnet: oleg350-client.virtnet: Throughput 1.85409 MB/sec 1 clients 1 procs max_latency=19594.140 ms oleg350-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg350-client.virtnet at Fri May 10 06:25:25 EDT 2024 with return code 0 oleg350-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg350-client.virtnet oleg350-client.virtnet: /mnt/lustre/d70b.replay-single/oleg350-client.virtnet /mnt/lustre/d70b.replay-single/oleg350-client.virtnet oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg350-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg350-client.virtnet: removed directory: 'clients/client0' oleg350-client.virtnet: removed directory: 'clients' oleg350-client.virtnet: removed 'client.txt' oleg350-client.virtnet: /mnt/lustre/d70b.replay-single/oleg350-client.virtnet oleg350-client.virtnet: dbench successfully finished