== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 07:08:31 (1733400511) Starting client oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre Started clients oleg122-client.virtnet: 192.168.201.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg122-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13079 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2484 1285204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26152 3580868 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1656 3594052 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27808 7174920 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg122-server Stopping /mnt/lustre-mds1 (opts:) on oleg122-server 07:08:40 (1733400520) shut down Failover mds1 to oleg122-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 07:08:56 (1733400536) targets are mounted 07:08:56 (1733400536) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26532 3568672 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1744 3592076 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28276 7160748 1% /mnt/lustre oleg122-client.virtnet: looking for dbench program oleg122-client.virtnet: /usr/bin/dbench oleg122-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg122-client.virtnet oleg122-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg122-client.virtnet' oleg122-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg122-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg122-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg122-client.virtnet at Thu Dec 5 07:08:33 EST 2024 oleg122-client.virtnet: waiting for dbench pid 13104 oleg122-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg122-client.virtnet: oleg122-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg122-client.virtnet: failed to create barrier semaphore oleg122-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg122-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg122-client.virtnet: releasing clients oleg122-client.virtnet: 1 297 10.96 MB/sec warmup 1 sec latency 23.849 ms oleg122-client.virtnet: 1 655 10.53 MB/sec warmup 2 sec latency 18.887 ms oleg122-client.virtnet: 1 957 7.27 MB/sec warmup 3 sec latency 17.865 ms oleg122-client.virtnet: 1 1278 5.78 MB/sec warmup 4 sec latency 17.595 ms oleg122-client.virtnet: 1 1686 5.24 MB/sec warmup 5 sec latency 15.847 ms oleg122-client.virtnet: 1 2051 4.44 MB/sec warmup 6 sec latency 17.503 ms oleg122-client.virtnet: 1 2412 4.00 MB/sec warmup 7 sec latency 74.034 ms oleg122-client.virtnet: 1 2412 3.50 MB/sec warmup 8 sec latency 1074.234 ms oleg122-client.virtnet: 1 2412 3.11 MB/sec warmup 9 sec latency 2074.444 ms oleg122-client.virtnet: 1 2412 2.80 MB/sec warmup 10 sec latency 3074.649 ms oleg122-client.virtnet: 1 2412 2.55 MB/sec warmup 11 sec latency 4074.935 ms oleg122-client.virtnet: 1 2412 2.34 MB/sec warmup 12 sec latency 5075.162 ms oleg122-client.virtnet: 1 2412 2.16 MB/sec warmup 13 sec latency 6075.383 ms oleg122-client.virtnet: 1 2412 2.00 MB/sec warmup 14 sec latency 7075.600 ms oleg122-client.virtnet: 1 2412 1.87 MB/sec warmup 15 sec latency 8075.874 ms oleg122-client.virtnet: 1 2412 1.75 MB/sec warmup 16 sec latency 9076.057 ms oleg122-client.virtnet: 1 2412 1.65 MB/sec warmup 17 sec latency 10076.279 ms oleg122-client.virtnet: 1 2412 1.56 MB/sec warmup 18 sec latency 11076.480 ms oleg122-client.virtnet: 1 2412 1.47 MB/sec warmup 19 sec latency 12076.753 ms oleg122-client.virtnet: 1 2412 1.40 MB/sec warmup 20 sec latency 13076.968 ms oleg122-client.virtnet: 1 2412 1.33 MB/sec warmup 21 sec latency 14077.250 ms oleg122-client.virtnet: 1 2412 1.27 MB/sec warmup 22 sec latency 15077.522 ms oleg122-client.virtnet: 1 2412 1.22 MB/sec warmup 23 sec latency 16077.746 ms oleg122-client.virtnet: 1 2412 1.17 MB/sec warmup 24 sec latency 17077.848 ms oleg122-client.virtnet: 1 2412 1.12 MB/sec warmup 25 sec latency 18078.098 ms oleg122-client.virtnet: 1 2412 1.08 MB/sec warmup 26 sec latency 19078.280 ms oleg122-client.virtnet: 1 2603 1.07 MB/sec warmup 27 sec latency 19659.303 ms oleg122-client.virtnet: 1 3377 1.27 MB/sec warmup 28 sec latency 17.556 ms oleg122-client.virtnet: 1 3821 1.37 MB/sec warmup 29 sec latency 15.189 ms oleg122-client.virtnet: 1 4112 1.34 MB/sec warmup 30 sec latency 14.217 ms oleg122-client.virtnet: 1 4440 1.31 MB/sec warmup 31 sec latency 25.880 ms oleg122-client.virtnet: 1 4828 1.32 MB/sec warmup 32 sec latency 17.447 ms oleg122-client.virtnet: 1 5449 1.38 MB/sec warmup 33 sec latencytest_70b fail mds2 2 times Failing mds2 on oleg122-server Stopping /mnt/lustre-mds2 (opts:) on oleg122-server 07:09:09 (1733400549) shut down Failover mds2 to oleg122-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0001 07:09:24 (1733400564) targets are mounted 07:09:24 (1733400564) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3568716 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12124 3591680 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49548 7160396 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg122-server Stopping /mnt/lustre-mds1 (opts:) on oleg122-server 07:09:37 (1733400577) shut down Failover mds1 to oleg122-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 8.632 ms oleg122-client.virtnet: 1 5994 1.39 MB/sec warmup 34 sec latency 17.956 ms oleg122-client.virtnet: 1 6548 1.50 MB/sec warmup 35 sec latency 20.496 ms oleg122-client.virtnet: 1 7180 1.60 MB/sec warmup 36 sec latency 18.910 ms oleg122-client.virtnet: 1 7451 1.60 MB/sec warmup 37 sec latency 16.859 ms oleg122-client.virtnet: 1 7752 1.56 MB/sec warmup 38 sec latency 16.074 ms oleg122-client.virtnet: 1 8025 1.54 MB/sec warmup 39 sec latency 25.486 ms oleg122-client.virtnet: 1 8339 1.53 MB/sec warmup 40 sec latency 18.624 ms oleg122-client.virtnet: 1 8707 1.57 MB/sec warmup 41 sec latency 16.614 ms oleg122-client.virtnet: 1 9127 1.54 MB/sec warmup 42 sec latency 11.131 ms oleg122-client.virtnet: 1 9504 1.54 MB/sec warmup 43 sec latency 17.003 ms oleg122-client.virtnet: 1 10016 1.63 MB/sec warmup 44 sec latency 18.103 ms oleg122-client.virtnet: 1 10636 1.70 MB/sec warmup 45 sec latency 19.465 ms oleg122-client.virtnet: 1 10925 1.70 MB/sec warmup 46 sec latency 15.497 ms oleg122-client.virtnet: 1 11165 1.67 MB/sec warmup 47 sec latency 15.973 ms oleg122-client.virtnet: 1 11480 1.64 MB/sec warmup 48 sec latency 17.766 ms oleg122-client.virtnet: 1 11810 1.64 MB/sec warmup 49 sec latency 19.257 ms oleg122-client.virtnet: 1 12240 1.67 MB/sec warmup 50 sec latency 13.586 ms oleg122-client.virtnet: 1 12444 1.64 MB/sec warmup 51 sec latency 298.713 ms oleg122-client.virtnet: 1 12444 1.61 MB/sec warmup 52 sec latency 1298.965 ms oleg122-client.virtnet: 1 12444 1.58 MB/sec warmup 53 sec latency 2299.295 ms oleg122-client.virtnet: 1 12444 1.55 MB/sec warmup 54 sec latency 3299.559 ms oleg122-client.virtnet: 1 12622 1.52 MB/sec warmup 55 sec latency 3825.892 ms oleg122-client.virtnet: 1 12996 1.52 MB/sec warmup 56 sec latency 13.557 ms oleg122-client.virtnet: 1 13409 1.57 MB/sec warmup 57 sec latency 18.341 ms oleg122-client.virtnet: 1 14027 1.60 MB/sec warmup 58 sec latency 18.839 ms oleg122-client.virtnet: 1 14424 1.65 MB/sec warmup 59 sec latency 16.683 ms oleg122-client.virtnet: 1 14872 0.21 MB/sec execute 1 sec latency 399.227 ms oleg122-client.virtnet: 1 15110 0.37 MB/sec execute 2 sec latency 517.912 ms oleg122-client.virtnet: 1 15431 0.71 MB/sec execute 3 sec latency 18.163 ms oleg122-client.virtnet: 1 15740 1.29 MB/sec execute 4 sec latency 211.940 ms oleg122-client.virtnet: 1 15740 1.03 MB/sec execute 5 sec latency 1212.154 ms oleg122-client.virtnet: 1 15740 0.86 MB/sec execute 6 sec latency 2212.356 ms oleg122-client.virtnet: 1 15740 0.74 MB/sec execute 7 sec latency 3212.545 ms oleg122-client.virtnet: 1 15740 0.64 MB/sec execute 8 sec latency 4212.771 ms oleg122-client.virtnet: 1 15740 0.57 MB/sec execute 9 sec latency 5213.045 ms oleg122-client.virtnet: 1 15740 0.52 MB/sec execute 10 sec latency 6213.218 ms oleg122-client.virtnet: 1 15740 0.47 MB/sec execute 11 sec latency 7213.424 ms oleg122-client.virtnet: 1 15740 0.43 MB/sec execute 12 sec latency 8213.675 ms oleg122-client.virtnet: 1 15740 0.40 MB/sec execute 13 sec latency 9213.905 ms oleg122-client.virtnet: 1 15740 0.37 MB/sec execute 14 sec latency 10214.154 ms oleg122-client.virtnet: 1 15740 0.34 MB/sec execute 15 sec latency 11214.357 ms oleg122-client.virtnet: 1 15740 0.32 MB/sec execute 16 sec latency 12214.585 ms oleg122-client.virtnet: 1 15740 0.30 MB/sec execute 17 sec latency 13214.893 ms oleg122-client.virtnet: 1 15740 0.29 MB/sec execute 18 sec latency 14215.129 ms oleg122-client.virtnet: 1 15740 0.27 MB/sec executeoleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 07:09:52 (1733400592) targets are mounted 07:09:52 (1733400592) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37432 3566692 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12160 3593692 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49592 7160384 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg122-server Stopping /mnt/lustre-mds2 (opts:) on oleg122-server 07:10:07 (1733400607) shut down Failover mds2 to oleg122-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0001 07:10:22 (1733400622) targets are mounted 07:10:22 (1733400622) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37580 3568928 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11912 3594248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7163176 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg122-server Stopping /mnt/lustre-mds1 (opts:) on oleg122-server 07:10:34 (1733400634) shut down 19 sec latency 15215.315 ms oleg122-client.virtnet: 1 15740 0.26 MB/sec execute 20 sec latency 16215.502 ms oleg122-client.virtnet: 1 15740 0.25 MB/sec execute 21 sec latency 17215.683 ms oleg122-client.virtnet: 1 15740 0.23 MB/sec execute 22 sec latency 18215.870 ms oleg122-client.virtnet: 1 15740 0.22 MB/sec execute 23 sec latency 19216.071 ms oleg122-client.virtnet: 1 15804 0.22 MB/sec execute 24 sec latency 19965.151 ms oleg122-client.virtnet: 1 16069 0.22 MB/sec execute 25 sec latency 19.463 ms oleg122-client.virtnet: 1 16406 0.22 MB/sec execute 26 sec latency 18.419 ms oleg122-client.virtnet: 1 16761 0.29 MB/sec execute 27 sec latency 18.406 ms oleg122-client.virtnet: 1 17369 0.47 MB/sec execute 28 sec latency 18.899 ms oleg122-client.virtnet: 1 17861 0.61 MB/sec execute 29 sec latency 17.805 ms oleg122-client.virtnet: 1 18158 0.64 MB/sec execute 30 sec latency 15.789 ms oleg122-client.virtnet: 1 18455 0.62 MB/sec execute 31 sec latency 19.546 ms oleg122-client.virtnet: 1 18723 0.62 MB/sec execute 32 sec latency 18.564 ms oleg122-client.virtnet: 1 19069 0.68 MB/sec execute 33 sec latency 17.573 ms oleg122-client.virtnet: 1 19412 0.72 MB/sec execute 34 sec latency 16.637 ms oleg122-client.virtnet: 1 19718 0.70 MB/sec execute 35 sec latency 20.302 ms oleg122-client.virtnet: 1 20086 0.72 MB/sec execute 36 sec latency 11.983 ms oleg122-client.virtnet: 1 20551 0.82 MB/sec execute 37 sec latency 18.737 ms oleg122-client.virtnet: 1 21166 0.89 MB/sec execute 38 sec latency 18.645 ms oleg122-client.virtnet: 1 21561 0.97 MB/sec execute 39 sec latency 16.633 ms oleg122-client.virtnet: 1 21795 0.96 MB/sec execute 40 sec latency 15.321 ms oleg122-client.virtnet: 1 22121 0.95 MB/sec execute 41 sec latency 18.333 ms oleg122-client.virtnet: 1 22417 0.96 MB/sec execute 42 sec latency 18.367 ms oleg122-client.virtnet: 1 22792 1.01 MB/sec execute 43 sec latency 17.923 ms oleg122-client.virtnet: 1 23059 0.99 MB/sec execute 44 sec latency 18.995 ms oleg122-client.virtnet: 1 23387 0.98 MB/sec execute 45 sec latency 12.260 ms oleg122-client.virtnet: 1 23749 0.99 MB/sec execute 46 sec latency 17.290 ms oleg122-client.virtnet: 1 24292 1.08 MB/sec execute 47 sec latency 17.549 ms oleg122-client.virtnet: 1 24989 1.17 MB/sec execute 48 sec latency 18.176 ms oleg122-client.virtnet: 1 25304 1.17 MB/sec execute 49 sec latency 32.630 ms oleg122-client.virtnet: 1 25304 1.15 MB/sec execute 50 sec latency 1032.893 ms oleg122-client.virtnet: 1 25304 1.13 MB/sec execute 51 sec latency 2033.200 ms oleg122-client.virtnet: 1 25304 1.10 MB/sec execute 52 sec latency 3033.405 ms oleg122-client.virtnet: 1 25480 1.08 MB/sec execute 53 sec latency 3719.651 ms oleg122-client.virtnet: 1 26020 1.10 MB/sec execute 54 sec latency 9.481 ms oleg122-client.virtnet: 1 26391 1.14 MB/sec execute 55 sec latency 17.965 ms oleg122-client.virtnet: 1 26676 1.12 MB/sec execute 56 sec latency 20.178 ms oleg122-client.virtnet: 1 27040 1.11 MB/sec execute 57 sec latency 12.112 ms oleg122-client.virtnet: 1 27448 1.13 MB/sec execute 58 sec latency 17.510 ms oleg122-client.virtnet: 1 28199 1.22 MB/sec execute 59 sec latency 309.075 ms oleg122-client.virtnet: 1 28738 1.27 MB/sec execute 60 sec latency 12.714 ms oleg122-client.virtnet: 1 29045 1.26 MB/sec execute 61 sec latency 140.579 ms oleg122-client.virtnet: 1 29045 1.23 MB/sec execute 62 sec latency 1140.827 ms oleg122-client.virtnet: 1 29045 1.22 MB/sec execute 63 sec latency 2141.102 ms oleg122-client.virtnet: 1 Failover mds1 to oleg122-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 07:10:50 (1733400650) targets are mounted 07:10:50 (1733400650) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37732 3568752 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11908 3594312 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49640 7163064 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg122-server Stopping /mnt/lustre-mds2 (opts:) on oleg122-server 07:11:02 (1733400662) shut down Failover mds2 to oleg122-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0001 07:11:17 (1733400677) targets are mounted 07:11:17 (1733400677) facet_failover done 29045 1.20 MB/sec execute 64 sec latency 3141.358 ms oleg122-client.virtnet: 1 29045 1.18 MB/sec execute 65 sec latency 4141.624 ms oleg122-client.virtnet: 1 29045 1.16 MB/sec execute 66 sec latency 5141.876 ms oleg122-client.virtnet: 1 29045 1.14 MB/sec execute 67 sec latency 6142.156 ms oleg122-client.virtnet: 1 29045 1.13 MB/sec execute 68 sec latency 7142.416 ms oleg122-client.virtnet: 1 29045 1.11 MB/sec execute 69 sec latency 8142.605 ms oleg122-client.virtnet: 1 29045 1.09 MB/sec execute 70 sec latency 9142.770 ms oleg122-client.virtnet: 1 29045 1.08 MB/sec execute 71 sec latency 10143.022 ms oleg122-client.virtnet: 1 29045 1.06 MB/sec execute 72 sec latency 11143.266 ms oleg122-client.virtnet: 1 29045 1.05 MB/sec execute 73 sec latency 12143.459 ms oleg122-client.virtnet: 1 29045 1.03 MB/sec execute 74 sec latency 13143.611 ms oleg122-client.virtnet: 1 29045 1.02 MB/sec execute 75 sec latency 14143.814 ms oleg122-client.virtnet: 1 29045 1.01 MB/sec execute 76 sec latency 15144.008 ms oleg122-client.virtnet: 1 29045 0.99 MB/sec execute 77 sec latency 16144.237 ms oleg122-client.virtnet: 1 29045 0.98 MB/sec execute 78 sec latency 17144.416 ms oleg122-client.virtnet: 1 29045 0.97 MB/sec execute 79 sec latency 18144.715 ms oleg122-client.virtnet: 1 29045 0.96 MB/sec execute 80 sec latency 19145.021 ms oleg122-client.virtnet: 1 29190 0.95 MB/sec execute 81 sec latency 19840.017 ms oleg122-client.virtnet: 1 29524 0.96 MB/sec execute 82 sec latency 18.609 ms oleg122-client.virtnet: 1 29897 0.98 MB/sec execute 83 sec latency 16.899 ms oleg122-client.virtnet: 1 30210 0.97 MB/sec execute 84 sec latency 15.458 ms oleg122-client.virtnet: 1 30605 0.97 MB/sec execute 85 sec latency 13.096 ms oleg122-client.virtnet: 1 31178 1.02 MB/sec execute 86 sec latency 16.157 ms oleg122-client.virtnet: 1 31974 1.08 MB/sec execute 87 sec latency 13.842 ms oleg122-client.virtnet: 1 32358 1.09 MB/sec execute 88 sec latency 13.678 ms oleg122-client.virtnet: 1 32714 1.08 MB/sec execute 89 sec latency 18.841 ms oleg122-client.virtnet: 1 33102 1.09 MB/sec execute 90 sec latency 13.864 ms oleg122-client.virtnet: 1 33483 1.11 MB/sec execute 91 sec latency 15.050 ms oleg122-client.virtnet: 1 33757 1.10 MB/sec execute 92 sec latency 19.672 ms oleg122-client.virtnet: 1 34082 1.09 MB/sec execute 93 sec latency 13.149 ms oleg122-client.virtnet: 1 34407 1.09 MB/sec execute 94 sec latency 18.915 ms oleg122-client.virtnet: 1 34959 1.14 MB/sec execute 95 sec latency 17.472 ms oleg122-client.virtnet: 1 35580 1.18 MB/sec execute 96 sec latency 17.192 ms oleg122-client.virtnet: 1 35842 1.18 MB/sec execute 97 sec latency 15.940 ms oleg122-client.virtnet: 1 36110 1.17 MB/sec execute 98 sec latency 16.003 ms oleg122-client.virtnet: 1 36404 1.17 MB/sec execute 99 sec latency 21.223 ms oleg122-client.virtnet: 1 36707 1.17 MB/sec execute 100 sec latency 17.957 ms oleg122-client.virtnet: 1 37074 1.19 MB/sec execute 101 sec latency 18.566 ms oleg122-client.virtnet: 1 37369 1.18 MB/sec execute 102 sec latency 15.629 ms oleg122-client.virtnet: 1 37769 1.17 MB/sec execute 103 sec latency 11.744 ms oleg122-client.virtnet: 1 37909 1.17 MB/sec execute 104 sec latency 677.899 ms oleg122-client.virtnet: 1 37909 1.16 MB/sec execute 105 sec latency 1678.074 ms oleg122-client.virtnet: 1 37909 1.15 MB/sec execute 106 sec latency 2678.263 ms oleg122-client.virtnet: 1 37909 1.14 MB/sec execute 107 sec latency 3678.479 ms oleg122-client.virtnet: 1 37909 1.13 MB/sec execute 108 secoleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37664 3568456 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12044 3593984 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49708 7162440 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg122-server Stopping /mnt/lustre-mds1 (opts:) on oleg122-server 07:11:31 (1733400691) shut down Failover mds1 to oleg122-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 07:11:46 (1733400706) targets are mounted 07:11:46 (1733400706) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37716 3568736 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11920 3593032 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7161768 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg122-server Stopping /mnt/lustre-mds2 (opts:) on oleg122-server 07:11:59 (1733400719) shut down latency 4678.674 ms oleg122-client.virtnet: 1 38514 1.17 MB/sec execute 109 sec latency 4764.068 ms oleg122-client.virtnet: 1 39167 1.20 MB/sec execute 110 sec latency 15.494 ms oleg122-client.virtnet: 1 39439 1.21 MB/sec execute 111 sec latency 16.331 ms oleg122-client.virtnet: 1 39727 1.20 MB/sec execute 112 sec latency 15.493 ms oleg122-client.virtnet: 1 39987 1.19 MB/sec execute 113 sec latency 23.192 ms oleg122-client.virtnet: 1 40304 1.19 MB/sec execute 114 sec latency 18.812 ms oleg122-client.virtnet: 1 40638 1.21 MB/sec execute 115 sec latency 111.823 ms oleg122-client.virtnet: 1 40918 1.20 MB/sec execute 116 sec latency 20.093 ms oleg122-client.virtnet: 1 41270 1.19 MB/sec execute 117 sec latency 11.640 ms oleg122-client.virtnet: 1 41436 1.19 MB/sec execute 118 sec latency 638.906 ms oleg122-client.virtnet: 1 41436 1.18 MB/sec execute 119 sec latency 1639.089 ms oleg122-client.virtnet: 1 41436 1.17 MB/sec execute 120 sec latency 2639.293 ms oleg122-client.virtnet: 1 41436 1.16 MB/sec execute 121 sec latency 3639.482 ms oleg122-client.virtnet: 1 41436 1.15 MB/sec execute 122 sec latency 4639.739 ms oleg122-client.virtnet: 1 41436 1.15 MB/sec execute 123 sec latency 5639.940 ms oleg122-client.virtnet: 1 41436 1.14 MB/sec execute 124 sec latency 6640.118 ms oleg122-client.virtnet: 1 41436 1.13 MB/sec execute 125 sec latency 7640.310 ms oleg122-client.virtnet: 1 41436 1.12 MB/sec execute 126 sec latency 8640.560 ms oleg122-client.virtnet: 1 41436 1.11 MB/sec execute 127 sec latency 9640.759 ms oleg122-client.virtnet: 1 41436 1.10 MB/sec execute 128 sec latency 10641.045 ms oleg122-client.virtnet: 1 41436 1.09 MB/sec execute 129 sec latency 11641.240 ms oleg122-client.virtnet: 1 41436 1.08 MB/sec execute 130 sec latency 12641.439 ms oleg122-client.virtnet: 1 41436 1.08 MB/sec execute 131 sec latency 13641.657 ms oleg122-client.virtnet: 1 41436 1.07 MB/sec execute 132 sec latency 14641.863 ms oleg122-client.virtnet: 1 41436 1.06 MB/sec execute 133 sec latency 15642.046 ms oleg122-client.virtnet: 1 41436 1.05 MB/sec execute 134 sec latency 16642.320 ms oleg122-client.virtnet: 1 41436 1.04 MB/sec execute 135 sec latency 17642.715 ms oleg122-client.virtnet: 1 41436 1.04 MB/sec execute 136 sec latency 18642.943 ms oleg122-client.virtnet: 1 41506 1.03 MB/sec execute 137 sec latency 19386.615 ms oleg122-client.virtnet: 1 42066 1.06 MB/sec execute 138 sec latency 17.736 ms oleg122-client.virtnet: 1 42669 1.09 MB/sec execute 139 sec latency 18.564 ms oleg122-client.virtnet: 1 42996 1.09 MB/sec execute 140 sec latency 17.078 ms oleg122-client.virtnet: 1 43286 1.09 MB/sec execute 141 sec latency 16.207 ms oleg122-client.virtnet: 1 43635 1.09 MB/sec execute 142 sec latency 13.996 ms oleg122-client.virtnet: 1 44090 1.11 MB/sec execute 143 sec latency 15.539 ms oleg122-client.virtnet: 1 44386 1.10 MB/sec execute 144 sec latency 20.082 ms oleg122-client.virtnet: 1 44758 1.09 MB/sec execute 145 sec latency 12.148 ms oleg122-client.virtnet: 1 45174 1.10 MB/sec execute 146 sec latency 16.833 ms oleg122-client.virtnet: 1 45947 1.14 MB/sec execute 147 sec latency 17.777 ms oleg122-client.virtnet: 1 46389 1.16 MB/sec execute 148 sec latency 14.494 ms oleg122-client.virtnet: 1 46722 1.16 MB/sec execute 149 sec latency 15.544 ms oleg122-client.virtnet: 1 47145 1.15 MB/sec execute 150 sec latency 18.127 ms oleg122-client.virtnet: 1 47557 1.17 MB/sec execute 151 sec latency 19.867 ms oleg122-client.virtnet: 1 47815 1.17 MB/sec execute 152 sec latency 17.230 ms oleg122-client.virFailover mds2 to oleg122-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0001 07:12:14 (1733400734) targets are mounted 07:12:14 (1733400734) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37536 3568724 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12068 3591900 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49604 7160624 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg122-server Stopping /mnt/lustre-mds1 (opts:) on oleg122-server 07:12:28 (1733400748) shut down Failover mds1 to oleg122-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 07:12:42 (1733400762) targets are mounted 07:12:42 (1733400762) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec tnet: 1 48169 1.16 MB/sec execute 153 sec latency 21.537 ms oleg122-client.virtnet: 1 48548 1.16 MB/sec execute 154 sec latency 15.533 ms oleg122-client.virtnet: 1 49112 1.19 MB/sec execute 155 sec latency 17.080 ms oleg122-client.virtnet: 1 49724 1.22 MB/sec execute 156 sec latency 18.004 ms oleg122-client.virtnet: 1 50015 1.22 MB/sec execute 157 sec latency 16.115 ms oleg122-client.virtnet: 1 50367 1.21 MB/sec execute 158 sec latency 17.826 ms oleg122-client.virtnet: 1 50638 1.21 MB/sec execute 159 sec latency 18.807 ms oleg122-client.virtnet: 1 50960 1.21 MB/sec execute 160 sec latency 17.630 ms oleg122-client.virtnet: 1 51301 1.22 MB/sec execute 161 sec latency 122.484 ms oleg122-client.virtnet: 1 51301 1.21 MB/sec execute 162 sec latency 1122.775 ms oleg122-client.virtnet: 1 51301 1.21 MB/sec execute 163 sec latency 2123.042 ms oleg122-client.virtnet: 1 51301 1.20 MB/sec execute 164 sec latency 3123.283 ms oleg122-client.virtnet: 1 51443 1.19 MB/sec execute 165 sec latency 3639.596 ms oleg122-client.virtnet: 1 51831 1.19 MB/sec execute 166 sec latency 11.298 ms oleg122-client.virtnet: 1 52158 1.19 MB/sec execute 167 sec latency 17.865 ms oleg122-client.virtnet: 1 52687 1.21 MB/sec execute 168 sec latency 17.581 ms oleg122-client.virtnet: 1 53272 1.23 MB/sec execute 169 sec latency 18.712 ms oleg122-client.virtnet: 1 53549 1.24 MB/sec execute 170 sec latency 16.330 ms oleg122-client.virtnet: 1 53790 1.23 MB/sec execute 171 sec latency 16.159 ms oleg122-client.virtnet: 1 54100 1.23 MB/sec execute 172 sec latency 29.486 ms oleg122-client.virtnet: 1 54393 1.23 MB/sec execute 173 sec latency 18.958 ms oleg122-client.virtnet: 1 54751 1.24 MB/sec execute 174 sec latency 19.912 ms oleg122-client.virtnet: 1 54775 1.23 MB/sec execute 175 sec latency 944.599 ms oleg122-client.virtnet: 1 54775 1.22 MB/sec execute 176 sec latency 1944.867 ms oleg122-client.virtnet: 1 54775 1.22 MB/sec execute 177 sec latency 2945.138 ms oleg122-client.virtnet: 1 54775 1.21 MB/sec execute 178 sec latency 3945.547 ms oleg122-client.virtnet: 1 54775 1.20 MB/sec execute 179 sec latency 4945.892 ms oleg122-client.virtnet: 1 54775 1.20 MB/sec execute 180 sec latency 5946.137 ms oleg122-client.virtnet: 1 54775 1.19 MB/sec execute 181 sec latency 6946.494 ms oleg122-client.virtnet: 1 54775 1.18 MB/sec execute 182 sec latency 7946.713 ms oleg122-client.virtnet: 1 54775 1.18 MB/sec execute 183 sec latency 8946.995 ms oleg122-client.virtnet: 1 54775 1.17 MB/sec execute 184 sec latency 9947.285 ms oleg122-client.virtnet: 1 54775 1.16 MB/sec execute 185 sec latency 10947.501 ms oleg122-client.virtnet: 1 54775 1.16 MB/sec execute 186 sec latency 11947.759 ms oleg122-client.virtnet: 1 54775 1.15 MB/sec execute 187 sec latency 12948.088 ms oleg122-client.virtnet: 1 54775 1.15 MB/sec execute 188 sec latency 13948.342 ms oleg122-client.virtnet: 1 54775 1.14 MB/sec execute 189 sec latency 14948.511 ms oleg122-client.virtnet: 1 54775 1.13 MB/sec execute 190 sec latency 15948.792 ms oleg122-client.virtnet: 1 54775 1.13 MB/sec execute 191 sec latency 16949.036 ms oleg122-client.virtnet: 1 54775 1.12 MB/sec execute 192 sec latency 17949.282 ms oleg122-client.virtnet: 1 54775 1.12 MB/sec execute 193 sec latency 18949.486 ms oleg122-client.virtnet: 1 54869 1.11 MB/sec execute 194 sec latency 19697.237 ms oleg122-client.virtnet: 1 55304 1.11 MB/sec execute 195 sec latency 15.647 ms oleg122-client.virtnet: 1 56131 1.13 MB/sec execute 196 sec latency 10.046 ms oleg122-client.virtnet: 1 57025 1.16 MB/sUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37508 3568736 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12152 3591864 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49660 7160600 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg122-server Stopping /mnt/lustre-mds2 (opts:) on oleg122-server 07:12:55 (1733400775) shut down Failover mds2 to oleg122-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0001 07:13:10 (1733400790) targets are mounted 07:13:10 (1733400790) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37420 3569156 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12228 3593860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7163016 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg122-server Stopping /mnt/lustre-mds1 (opts:) on oleg122-server 07:13:24 (1733400804) shut down Failover mds1 to oleg122-server mount facets: mds1 ec execute 197 sec latency 9.159 ms oleg122-client.virtnet: 1 57554 1.16 MB/sec execute 198 sec latency 10.107 ms oleg122-client.virtnet: 1 58205 1.18 MB/sec execute 199 sec latency 9.472 ms oleg122-client.virtnet: 1 58794 1.18 MB/sec execute 200 sec latency 10.421 ms oleg122-client.virtnet: 1 59355 1.18 MB/sec execute 201 sec latency 13.000 ms oleg122-client.virtnet: 1 60361 1.22 MB/sec execute 202 sec latency 13.338 ms oleg122-client.virtnet: 1 60807 1.22 MB/sec execute 203 sec latency 11.331 ms oleg122-client.virtnet: 1 61397 1.23 MB/sec execute 204 sec latency 12.677 ms oleg122-client.virtnet: 1 62071 1.24 MB/sec execute 205 sec latency 9.451 ms oleg122-client.virtnet: 1 62536 1.23 MB/sec execute 206 sec latency 11.420 ms oleg122-client.virtnet: 1 62912 1.24 MB/sec execute 207 sec latency 18.251 ms oleg122-client.virtnet: 1 63571 1.26 MB/sec execute 208 sec latency 18.709 ms oleg122-client.virtnet: 1 64023 1.27 MB/sec execute 209 sec latency 12.922 ms oleg122-client.virtnet: 1 64297 1.27 MB/sec execute 210 sec latency 15.959 ms oleg122-client.virtnet: 1 64598 1.27 MB/sec execute 211 sec latency 20.361 ms oleg122-client.virtnet: 1 64865 1.27 MB/sec execute 212 sec latency 17.846 ms oleg122-client.virtnet: 1 65279 1.28 MB/sec execute 213 sec latency 17.475 ms oleg122-client.virtnet: 1 65563 1.28 MB/sec execute 214 sec latency 20.018 ms oleg122-client.virtnet: 1 65880 1.27 MB/sec execute 215 sec latency 14.449 ms oleg122-client.virtnet: 1 66339 1.27 MB/sec execute 216 sec latency 18.217 ms oleg122-client.virtnet: 1 66861 1.29 MB/sec execute 217 sec latency 130.467 ms oleg122-client.virtnet: 1 66861 1.29 MB/sec execute 218 sec latency 1130.670 ms oleg122-client.virtnet: 1 66861 1.28 MB/sec execute 219 sec latency 2130.982 ms oleg122-client.virtnet: 1 66861 1.27 MB/sec execute 220 sec latency 3131.330 ms oleg122-client.virtnet: 1 67207 1.28 MB/sec execute 221 sec latency 3746.097 ms oleg122-client.virtnet: 1 67585 1.29 MB/sec execute 222 sec latency 12.957 ms oleg122-client.virtnet: 1 67836 1.29 MB/sec execute 223 sec latency 16.338 ms oleg122-client.virtnet: 1 68142 1.28 MB/sec execute 224 sec latency 21.779 ms oleg122-client.virtnet: 1 68400 1.28 MB/sec execute 225 sec latency 18.769 ms oleg122-client.virtnet: 1 68830 1.29 MB/sec execute 226 sec latency 18.353 ms oleg122-client.virtnet: 1 69058 1.29 MB/sec execute 227 sec latency 162.948 ms oleg122-client.virtnet: 1 69223 1.28 MB/sec execute 228 sec latency 521.879 ms oleg122-client.virtnet: 1 69573 1.28 MB/sec execute 229 sec latency 12.377 ms oleg122-client.virtnet: 1 69899 1.28 MB/sec execute 230 sec latency 19.761 ms oleg122-client.virtnet: 1 69927 1.28 MB/sec execute 231 sec latency 894.007 ms oleg122-client.virtnet: 1 69927 1.27 MB/sec execute 232 sec latency 1894.271 ms oleg122-client.virtnet: 1 69927 1.26 MB/sec execute 233 sec latency 2894.503 ms oleg122-client.virtnet: 1 69927 1.26 MB/sec execute 234 sec latency 3894.721 ms oleg122-client.virtnet: 1 69927 1.25 MB/sec execute 235 sec latency 4894.930 ms oleg122-client.virtnet: 1 69927 1.25 MB/sec execute 236 sec latency 5895.153 ms oleg122-client.virtnet: 1 69927 1.24 MB/sec execute 237 sec latency 6895.436 ms oleg122-client.virtnet: 1 69927 1.24 MB/sec execute 238 sec latency 7895.739 ms oleg122-client.virtnet: 1 69927 1.23 MB/sec execute 239 sec latency 8896.021 ms oleg122-client.virtnet: 1 69927 1.23 MB/sec execute 240 sec latency 9896.297 ms oleg122-client.virtnet: 1 69927 1.22 MB/sec execute 241 sec latency 10896.501 ms oleg122-client.virtnet:Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 07:13:39 (1733400819) targets are mounted 07:13:39 (1733400819) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 69927 1.22 MB/sec execute 242 sec latency 11896.710 ms oleg122-client.virtnet: 1 69927 1.21 MB/sec execute 243 sec latency 12896.938 ms oleg122-client.virtnet: 1 69927 1.21 MB/sec execute 244 sec latency 13897.118 ms oleg122-client.virtnet: 1 69927 1.20 MB/sec execute 245 sec latency 14897.347 ms oleg122-client.virtnet: 1 69927 1.20 MB/sec execute 246 sec latency 15897.527 ms oleg122-client.virtnet: 1 69927 1.19 MB/sec execute 247 sec latency 16897.716 ms oleg122-client.virtnet: 1 69927 1.19 MB/sec execute 248 sec latency 17897.910 ms oleg122-client.virtnet: 1 69927 1.18 MB/sec execute 249 sec latency 18898.101 ms oleg122-client.virtnet: 1 69973 1.18 MB/sec execute 250 sec latency 19861.151 ms oleg122-client.virtnet: 1 70437 1.19 MB/sec execute 251 sec latency 115.984 ms oleg122-client.virtnet: 1 71095 1.21 MB/sec execute 252 sec latency 13.833 ms oleg122-client.virtnet: 1 71350 1.21 MB/sec execute 253 sec latency 16.351 ms oleg122-client.virtnet: 1 71663 1.21 MB/sec execute 254 sec latency 23.712 ms oleg122-client.virtnet: 1 71924 1.20 MB/sec execute 255 sec latency 18.780 ms oleg122-client.virtnet: 1 72231 1.20 MB/sec execute 256 sec latency 18.584 ms oleg122-client.virtnet: 1 72588 1.21 MB/sec execute 257 sec latency 15.533 ms oleg122-client.virtnet: 1 72879 1.21 MB/sec execute 258 sec latency 19.904 ms oleg122-client.virtnet: 1 73263 1.20 MB/sec execute 259 sec latency 12.277 ms oleg122-client.virtnet: 1 73598 1.21 MB/sec execute 260 sec latency 18.041 ms oleg122-client.virtnet: 1 74513 1.24 MB/sec execute 261 sec latency 18.112 ms oleg122-client.virtnet: 1 74813 1.24 MB/sec execute 262 sec latency 15.962 ms oleg122-client.virtnet: 1 75136 1.24 MB/sec execute 263 sec latency 16.348 ms oleg122-client.virtnet: 1 75468 1.23 MB/sec execute 264 sec latency 17.607 ms oleg122-client.virtnet: 1 75783 1.23 MB/sec execute 265 sec latency 18.576 ms oleg122-client.virtnet: 1 76200 1.24 MB/sec execute 266 sec latency 17.759 ms oleg122-client.virtnet: 1 76644 1.24 MB/sec execute 267 sec latency 10.609 ms oleg122-client.virtnet: 1 77102 1.24 MB/sec execute 268 sec latency 14.666 ms oleg122-client.virtnet: 1 78076 1.27 MB/sec execute 269 sec latency 17.336 ms oleg122-client.virtnet: 1 78457 1.27 MB/sec execute 270 sec latency 15.078 ms oleg122-client.virtnet: 1 78748 1.27 MB/sec execute 271 sec latency 15.465 ms oleg122-client.virtnet: 1 79010 1.27 MB/sec execute 272 sec latency 20.349 ms oleg122-client.virtnet: 1 79322 1.27 MB/sec execute 273 sec latency 18.447 ms oleg122-client.virtnet: 1 79683 1.28 MB/sec execute 274 sec latency 17.137 ms oleg122-client.virtnet: 1 79961 1.27 MB/sec execute 275 sec latency 18.620 ms oleg122-client.virtnet: 1 80334 1.27 MB/sec execute 276 sec latency 12.040 ms oleg122-client.virtnet: 1 80669 1.27 MB/sec execute 277 sec latency 18.482 ms oleg122-client.virtnet: 1 81435 1.29 MB/sec execute 278 sec latency 18.440 ms oleg122-client.virtnet: 1 81906 1.30 MB/sec execute 279 sec latency 15.801 ms oleg122-client.virtnet: 1 82144 1.30 MB/sec execute 280 sec latency 15.738 ms oleg122-client.virtnet: 1 82470 1.29 MB/sec execute 281 sec latency 20.378 ms oleg122-client.virtnet: 1 82798 1.30 MB/sec execute 282 sec latency 17.868 ms oleg122-client.virtnet: 1 83154 1.30 MB/sec execute 283 sec latency 17.768 ms oleg122-client.virtnet: 1 83438 1.30 MB/sec execute 284 sec latency 19.979 ms oleg122-client.virtnet: 1 83779 1.29 MB/sec execute 285 sec latency 13.259 ms oleg122-client.virtnet: 1 84104 1.30 MB/sec execute 286 sec latency 17.232 ms oleg122-client.virtnet: 1 84643 1.31 MB/sec execute 287 sec latency 18.231 ms oleg122-client.virtnet: 1 85257 1.32 MB/sec execute 288 sec latency 18.620 ms oleg122-client.virtnet: 1 85569 1.32 MB/sec execute 289 sec latency 15.827 ms oleg122-client.virtnet: 1 85923 1.32 MB/sec execute 290 sec latency 20.049 ms oleg122-client.virtnet: 1 86243 1.32 MB/sec execute 291 sec latency 17.893 ms oleg122-client.virtnet: 1 86696 1.33 MB/sec execute 292 sec latency 12.517 ms oleg122-client.virtnet: 1 86994 1.32 MB/sec execute 293 sec latency 19.469 ms oleg122-client.virtnet: 1 87382 1.32 MB/sec execute 294 sec latency 11.431 ms oleg122-client.virtnet: 1 88073 1.34 MB/sec execute 295 sec latency 17.610 ms oleg122-client.virtnet: 1 88688 1.35 MB/sec execute 296 sec latency 18.503 ms oleg122-client.virtnet: 1 88985 1.35 MB/sec execute 297 sec latency 15.927 ms oleg122-client.virtnet: 1 89214 1.35 MB/sec execute 298 sec latency 15.708 ms oleg122-client.virtnet: 1 89587 1.35 MB/sec execute 299 sec latency 24.609 ms oleg122-client.virtnet: 1 cleanup 300 sec oleg122-client.virtnet: 0 cleanup 300 sec oleg122-client.virtnet: oleg122-client.virtnet: Operation Count AvgLat MaxLat oleg122-client.virtnet: ---------------------------------------- oleg122-client.virtnet: NTCreateX 13031 12.992 19965.135 oleg122-client.virtnet: Close 9577 1.566 6.425 oleg122-client.virtnet: Rename 555 12.246 21.462 oleg122-client.virtnet: Unlink 2649 2.675 5.604 oleg122-client.virtnet: Qpathinfo 11814 2.794 3746.075 oleg122-client.virtnet: Qfileinfo 2052 0.200 2.392 oleg122-client.virtnet: Qfsinfo 2173 0.320 6.589 oleg122-client.virtnet: Sfileinfo 1079 43.637 19861.132 oleg122-client.virtnet: Find 4569 0.597 15.011 oleg122-client.virtnet: WriteX 6442 1.254 8.442 oleg122-client.virtnet: ReadX 20361 0.030 6.219 oleg122-client.virtnet: LockX 42 1.546 4.371 oleg122-client.virtnet: UnlockX 42 1.538 2.153 oleg122-client.virtnet: Flush 927 8.970 521.851 oleg122-client.virtnet: oleg122-client.virtnet: Throughput 1.34515 MB/sec 1 clients 1 procs max_latency=19965.151 ms oleg122-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg122-client.virtnet at Thu Dec 5 07:14:33 EST 2024 with return code 0 oleg122-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg122-client.virtnet oleg122-client.virtnet: /mnt/lustre/d70b.replay-single/oleg122-client.virtnet /mnt/lustre/d70b.replay-single/oleg122-client.virtnet oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg122-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg122-client.virtnet: removed directory: 'clients/client0' oleg122-client.virtnet: removed directory: 'clients' oleg122-client.virtnet: removed 'client.txt' oleg122-client.virtnet: /mnt/lustre/d70b.replay-single/oleg122-client.virtnet oleg122-client.virtnet: dbench successfully finished