== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 21:57:40 (1745459860) Starting client oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre Started clients oleg424-client.virtnet: 192.168.204.124@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg424-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13336 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2524 1285164 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26160 3556140 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6568 3593164 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 32728 7149304 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg424-server Stopping /mnt/lustre-mds1 (opts:) on oleg424-server 21:57:49 (1745459869) shut down facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds1 to oleg424-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 21:58:03 (1745459883) targets are mounted 21:58:03 (1745459883) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26552 3568644 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6336 3591028 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 32888 7159672 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg424-server Stopping /mnt/lustre-mds2 (opts:) on oleg424-server oleg424-client.virtnet: looking for dbench program oleg424-client.virtnet: /usr/bin/dbench oleg424-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg424-client.virtnet oleg424-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg424-client.virtnet' oleg424-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg424-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg424-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg424-client.virtnet at Wed Apr 23 21:57:41 EDT 2025 oleg424-client.virtnet: waiting for dbench pid 13361 oleg424-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg424-client.virtnet: oleg424-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg424-client.virtnet: failed to create barrier semaphore oleg424-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg424-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg424-client.virtnet: releasing clients oleg424-client.virtnet: 1 511 17.73 MB/sec warmup 1 sec latency 14.376 ms oleg424-client.virtnet: 1 1192 11.52 MB/sec warmup 2 sec latency 9.378 ms oleg424-client.virtnet: 1 1686 8.74 MB/sec warmup 3 sec latency 14.555 ms oleg424-client.virtnet: 1 1884 6.59 MB/sec warmup 4 sec latency 465.042 ms oleg424-client.virtnet: 1 2350 5.58 MB/sec warmup 5 sec latency 490.646 ms oleg424-client.virtnet: 1 2929 5.58 MB/sec warmup 6 sec latency 17.326 ms oleg424-client.virtnet: 1 3341 5.05 MB/sec warmup 7 sec latency 503.404 ms oleg424-client.virtnet: 1 3341 4.42 MB/sec warmup 8 sec latency 1503.555 ms oleg424-client.virtnet: 1 3341 3.93 MB/sec warmup 9 sec latency 2503.705 ms oleg424-client.virtnet: 1 3341 3.54 MB/sec warmup 10 sec latency 3503.923 ms oleg424-client.virtnet: 1 3341 3.22 MB/sec warmup 11 sec latency 4504.172 ms oleg424-client.virtnet: 1 3341 2.95 MB/sec warmup 12 sec latency 5504.383 ms oleg424-client.virtnet: 1 3341 2.72 MB/sec warmup 13 sec latency 6504.616 ms oleg424-client.virtnet: 1 3341 2.53 MB/sec warmup 14 sec latency 7504.899 ms oleg424-client.virtnet: 1 3341 2.36 MB/sec warmup 15 sec latency 8505.098 ms oleg424-client.virtnet: 1 3341 2.21 MB/sec warmup 16 sec latency 9505.278 ms oleg424-client.virtnet: 1 3341 2.08 MB/sec warmup 17 sec latency 10505.437 ms oleg424-client.virtnet: 1 3341 1.97 MB/sec warmup 18 sec latency 11505.576 ms oleg424-client.virtnet: 1 3341 1.86 MB/sec warmup 19 sec latency 12505.685 ms oleg424-client.virtnet: 1 3341 1.77 MB/sec warmup 20 sec latency 13505.858 ms oleg424-client.virtnet: 1 3341 1.68 MB/sec warmup 21 sec latency 14505.970 ms oleg424-client.virtnet: 1 3341 1.61 MB/sec warmup 22 sec latency 15506.148 ms oleg424-client.virtnet: 1 3341 1.54 MB/sec warmup 23 sec latency 16506.415 ms oleg424-client.virtnet: 1 3341 1.47 MB/sec warmup 24 sec latency 17506.704 ms oleg424-client.virtnet: 1 3341 1.42 MB/sec warmup 25 sec latency 18506.974 ms oleg424-client.virtnet: 1 3694 1.49 MB/sec warmup 26 sec latency 18909.241 ms oleg424-client.virtnet: 1 4152 1.48 MB/sec warmup 27 sec latency 9.932 ms oleg424-client.virtnet: 1 4600 1.50 MB/sec warmup 28 sec latency 14.253 ms oleg424-client.virtnet: 1 5135 1.56 MB/sec warmup 29 sec latency 16.537 ms oleg424-client.virtnet: 1 5759 1.53 MB/sec warmup 30 sec latency 12.213 ms oleg424-client.virtnet: 1 6487 1.70 MB/sec warmup 31 sec latency 13.130 ms oleg424-client.virtnet: 1 7364 1.84 MB/sec warmup 32 sec latency 12.893 ms oleg424-client.virtnet: 1 7849 1.80 MB/sec warmup 33 sec latency 121:58:15 (1745459895) shut down facet: mds2 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds2 to oleg424-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 21:58:29 (1745459909) targets are mounted 21:58:29 (1745459909) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36588 3569700 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13048 3592980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7162680 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg424-server Stopping /mnt/lustre-mds1 (opts:) on oleg424-server 21:58:40 (1745459920) shut down facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds1 to oleg424-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 21:58:54 (1745459934) targets are mounted 21:58:54 (1745459934) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 0.257 ms oleg424-client.virtnet: 1 8369 1.80 MB/sec warmup 34 sec latency 12.274 ms oleg424-client.virtnet: 1 8916 1.84 MB/sec warmup 35 sec latency 11.678 ms oleg424-client.virtnet: 1 9522 1.84 MB/sec warmup 36 sec latency 11.981 ms oleg424-client.virtnet: 1 10671 2.07 MB/sec warmup 37 sec latency 12.508 ms oleg424-client.virtnet: 1 11244 2.06 MB/sec warmup 38 sec latency 7.336 ms oleg424-client.virtnet: 1 11833 2.06 MB/sec warmup 39 sec latency 10.003 ms oleg424-client.virtnet: 1 12344 2.09 MB/sec warmup 40 sec latency 14.858 ms oleg424-client.virtnet: 1 12800 2.05 MB/sec warmup 41 sec latency 11.646 ms oleg424-client.virtnet: 1 13260 2.06 MB/sec warmup 42 sec latency 14.785 ms oleg424-client.virtnet: 1 14259 2.23 MB/sec warmup 43 sec latency 12.392 ms oleg424-client.virtnet: 1 14733 2.21 MB/sec warmup 44 sec latency 11.451 ms oleg424-client.virtnet: 1 15339 2.21 MB/sec warmup 45 sec latency 10.329 ms oleg424-client.virtnet: 1 15946 2.23 MB/sec warmup 46 sec latency 10.551 ms oleg424-client.virtnet: 1 16598 2.22 MB/sec warmup 47 sec latency 13.208 ms oleg424-client.virtnet: 1 16755 2.20 MB/sec warmup 48 sec latency 789.486 ms oleg424-client.virtnet: 1 16755 2.15 MB/sec warmup 49 sec latency 1789.654 ms oleg424-client.virtnet: 1 16755 2.11 MB/sec warmup 50 sec latency 2789.867 ms oleg424-client.virtnet: 1 16755 2.07 MB/sec warmup 51 sec latency 3790.026 ms oleg424-client.virtnet: 1 17775 2.21 MB/sec warmup 52 sec latency 3888.036 ms oleg424-client.virtnet: 1 18211 2.20 MB/sec warmup 53 sec latency 14.293 ms oleg424-client.virtnet: 1 18687 2.17 MB/sec warmup 54 sec latency 20.328 ms oleg424-client.virtnet: 1 19403 2.21 MB/sec warmup 55 sec latency 10.930 ms oleg424-client.virtnet: 1 20123 2.20 MB/sec warmup 56 sec latency 9.064 ms oleg424-client.virtnet: 1 21012 2.28 MB/sec warmup 57 sec latency 44.051 ms oleg424-client.virtnet: 1 21736 2.34 MB/sec warmup 58 sec latency 9.485 ms oleg424-client.virtnet: 1 21775 2.30 MB/sec warmup 59 sec latency 921.963 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 1 sec latency 2922.275 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 2 sec latency 3922.557 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 3 sec latency 4922.815 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 4 sec latency 5923.139 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 5 sec latency 6923.414 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 6 sec latency 7923.612 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 7 sec latency 8923.868 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 8 sec latency 9924.138 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 9 sec latency 10924.393 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 10 sec latency 11924.543 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 11 sec latency 12924.709 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 12 sec latency 13924.899 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 13 sec latency 14925.153 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 14 sec latency 15925.381 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 15 sec latency 16925.580 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 16 sec latency 17925.800 ms oleg424-client.virtnet: 1 21775 0.00 MB/sec execute 17 sec latency 18926.029 ms oleg424-client.virtnet: 1 22144 0.03 MB/sec execute 18 sec latency 18963.425 ms oleg424-client.virtnet: 1 22577 0.11 MB/sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36408 3570216 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13088 3593048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7163264 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg424-server Stopping /mnt/lustre-mds2 (opts:) on oleg424-server 21:59:07 (1745459947) shut down facet: mds2 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds2 to oleg424-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 21:59:21 (1745459961) targets are mounted 21:59:21 (1745459961) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36540 3569892 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13096 3591908 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7161800 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg424-server Stopping /mnt/lustre-mds1 (opts:) on oleg424-server 21:59:32 (1745459972) shut down facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds1 to oleg424-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 execute 19 sec latency 15.243 ms oleg424-client.virtnet: 1 23091 0.27 MB/sec execute 20 sec latency 13.451 ms oleg424-client.virtnet: 1 23762 0.34 MB/sec execute 21 sec latency 10.869 ms oleg424-client.virtnet: 1 24600 0.65 MB/sec execute 22 sec latency 14.118 ms oleg424-client.virtnet: 1 25123 0.82 MB/sec execute 23 sec latency 12.167 ms oleg424-client.virtnet: 1 25450 0.80 MB/sec execute 24 sec latency 12.830 ms oleg424-client.virtnet: 1 25789 0.79 MB/sec execute 25 sec latency 23.147 ms oleg424-client.virtnet: 1 26506 0.94 MB/sec execute 26 sec latency 8.907 ms oleg424-client.virtnet: 1 26920 0.92 MB/sec execute 27 sec latency 10.974 ms oleg424-client.virtnet: 1 27355 0.96 MB/sec execute 28 sec latency 14.860 ms oleg424-client.virtnet: 1 28443 1.26 MB/sec execute 29 sec latency 11.892 ms oleg424-client.virtnet: 1 28943 1.28 MB/sec execute 30 sec latency 10.149 ms oleg424-client.virtnet: 1 29621 1.30 MB/sec execute 31 sec latency 9.897 ms oleg424-client.virtnet: 1 30307 1.36 MB/sec execute 32 sec latency 8.656 ms oleg424-client.virtnet: 1 31176 1.49 MB/sec execute 33 sec latency 8.699 ms oleg424-client.virtnet: 1 32230 1.68 MB/sec execute 34 sec latency 8.578 ms oleg424-client.virtnet: 1 32851 1.66 MB/sec execute 35 sec latency 9.864 ms oleg424-client.virtnet: 1 33527 1.74 MB/sec execute 36 sec latency 9.498 ms oleg424-client.virtnet: 1 34176 1.71 MB/sec execute 37 sec latency 9.838 ms oleg424-client.virtnet: 1 35237 1.89 MB/sec execute 38 sec latency 9.649 ms oleg424-client.virtnet: 1 35942 1.96 MB/sec execute 39 sec latency 8.338 ms oleg424-client.virtnet: 1 36568 1.96 MB/sec execute 40 sec latency 9.220 ms oleg424-client.virtnet: 1 37243 2.00 MB/sec execute 41 sec latency 9.479 ms oleg424-client.virtnet: 1 37949 1.99 MB/sec execute 42 sec latency 8.318 ms oleg424-client.virtnet: 1 38267 2.03 MB/sec execute 43 sec latency 713.337 ms oleg424-client.virtnet: 1 39123 2.14 MB/sec execute 44 sec latency 979.761 ms oleg424-client.virtnet: 1 39712 2.13 MB/sec execute 45 sec latency 8.583 ms oleg424-client.virtnet: 1 40324 2.13 MB/sec execute 46 sec latency 9.781 ms oleg424-client.virtnet: 1 41063 2.15 MB/sec execute 47 sec latency 9.425 ms oleg424-client.virtnet: 1 41823 2.22 MB/sec execute 48 sec latency 62.678 ms oleg424-client.virtnet: 1 42715 2.31 MB/sec execute 49 sec latency 307.778 ms oleg424-client.virtnet: 1 43309 2.30 MB/sec execute 50 sec latency 8.415 ms oleg424-client.virtnet: 1 43471 2.26 MB/sec execute 51 sec latency 715.542 ms oleg424-client.virtnet: 1 43471 2.22 MB/sec execute 52 sec latency 1715.820 ms oleg424-client.virtnet: 1 43471 2.18 MB/sec execute 53 sec latency 2716.034 ms oleg424-client.virtnet: 1 43471 2.14 MB/sec execute 54 sec latency 3716.186 ms oleg424-client.virtnet: 1 43471 2.10 MB/sec execute 55 sec latency 4716.341 ms oleg424-client.virtnet: 1 43471 2.06 MB/sec execute 56 sec latency 5716.491 ms oleg424-client.virtnet: 1 43471 2.02 MB/sec execute 57 sec latency 6716.684 ms oleg424-client.virtnet: 1 43471 1.99 MB/sec execute 58 sec latency 7716.857 ms oleg424-client.virtnet: 1 43471 1.95 MB/sec execute 59 sec latency 8717.041 ms oleg424-client.virtnet: 1 43471 1.92 MB/sec execute 60 sec latency 9717.204 ms oleg424-client.virtnet: 1 43471 1.89 MB/sec execute 61 sec latency 10717.334 ms oleg424-client.virtnet: 1 43471 1.86 MB/sec execute 62 sec latency 11717.434 ms oleg424-client.virtnet: 1 43471 1.83 MB/sec execute 63 sec latency 12717.606 ms oleg424-client.virtnet: 1 43471 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 21:59:47 (1745459987) targets are mounted 21:59:47 (1745459987) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36408 3570132 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13084 3592844 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7162976 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg424-server Stopping /mnt/lustre-mds2 (opts:) on oleg424-server 21:59:58 (1745459998) shut down facet: mds2 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds2 to oleg424-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 22:00:12 (1745460012) targets are mounted 22:00:12 (1745460012) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36528 3569700 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13084 3593820 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7163520 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg424-server Stopping /mnt/lustre-mds1 (opts:) on oleg424-server 22:00:24 (1745460024) shut down facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server 1.80 MB/sec execute 64 sec latency 13717.746 ms oleg424-client.virtnet: 1 43471 1.77 MB/sec execute 65 sec latency 14717.915 ms oleg424-client.virtnet: 1 43471 1.75 MB/sec execute 66 sec latency 15718.060 ms oleg424-client.virtnet: 1 43471 1.72 MB/sec execute 67 sec latency 16718.205 ms oleg424-client.virtnet: 1 43471 1.70 MB/sec execute 68 sec latency 17718.367 ms oleg424-client.virtnet: 1 43471 1.67 MB/sec execute 69 sec latency 18718.479 ms oleg424-client.virtnet: 1 44020 1.71 MB/sec execute 70 sec latency 18766.467 ms oleg424-client.virtnet: 1 44633 1.69 MB/sec execute 71 sec latency 8.427 ms oleg424-client.virtnet: 1 45526 1.77 MB/sec execute 72 sec latency 10.214 ms oleg424-client.virtnet: 1 46442 1.83 MB/sec execute 73 sec latency 8.049 ms oleg424-client.virtnet: 1 47028 1.82 MB/sec execute 74 sec latency 9.493 ms oleg424-client.virtnet: 1 47607 1.85 MB/sec execute 75 sec latency 10.183 ms oleg424-client.virtnet: 1 48041 1.83 MB/sec execute 76 sec latency 13.872 ms oleg424-client.virtnet: 1 48779 1.85 MB/sec execute 77 sec latency 9.781 ms oleg424-client.virtnet: 1 49940 1.96 MB/sec execute 78 sec latency 8.400 ms oleg424-client.virtnet: 1 50520 1.94 MB/sec execute 79 sec latency 11.871 ms oleg424-client.virtnet: 1 51256 1.98 MB/sec execute 80 sec latency 8.755 ms oleg424-client.virtnet: 1 51850 1.96 MB/sec execute 81 sec latency 11.428 ms oleg424-client.virtnet: 1 52894 2.03 MB/sec execute 82 sec latency 8.501 ms oleg424-client.virtnet: 1 53653 2.07 MB/sec execute 83 sec latency 9.966 ms oleg424-client.virtnet: 1 54234 2.06 MB/sec execute 84 sec latency 13.042 ms oleg424-client.virtnet: 1 54965 2.09 MB/sec execute 85 sec latency 10.657 ms oleg424-client.virtnet: 1 55685 2.08 MB/sec execute 86 sec latency 8.147 ms oleg424-client.virtnet: 1 56851 2.18 MB/sec execute 87 sec latency 8.440 ms oleg424-client.virtnet: 1 57315 2.17 MB/sec execute 88 sec latency 10.320 ms oleg424-client.virtnet: 1 57866 2.17 MB/sec execute 89 sec latency 11.551 ms oleg424-client.virtnet: 1 58475 2.18 MB/sec execute 90 sec latency 10.391 ms oleg424-client.virtnet: 1 59210 2.18 MB/sec execute 91 sec latency 8.483 ms oleg424-client.virtnet: 1 60361 2.27 MB/sec execute 92 sec latency 9.194 ms oleg424-client.virtnet: 1 60523 2.26 MB/sec execute 93 sec latency 754.389 ms oleg424-client.virtnet: 1 60523 2.23 MB/sec execute 94 sec latency 1754.532 ms oleg424-client.virtnet: 1 60751 2.21 MB/sec execute 95 sec latency 2202.609 ms oleg424-client.virtnet: 1 61366 2.21 MB/sec execute 96 sec latency 10.861 ms oleg424-client.virtnet: 1 62060 2.22 MB/sec execute 97 sec latency 9.862 ms oleg424-client.virtnet: 1 62778 2.22 MB/sec execute 98 sec latency 9.923 ms oleg424-client.virtnet: 1 63864 2.30 MB/sec execute 99 sec latency 11.638 ms oleg424-client.virtnet: 1 64406 2.29 MB/sec execute 100 sec latency 7.910 ms oleg424-client.virtnet: 1 65037 2.29 MB/sec execute 101 sec latency 12.245 ms oleg424-client.virtnet: 1 65466 2.30 MB/sec execute 102 sec latency 382.384 ms oleg424-client.virtnet: 1 65466 2.28 MB/sec execute 103 sec latency 1382.528 ms oleg424-client.virtnet: 1 65466 2.26 MB/sec execute 104 sec latency 2382.711 ms oleg424-client.virtnet: 1 65466 2.23 MB/sec execute 105 sec latency 3382.849 ms oleg424-client.virtnet: 1 65466 2.21 MB/sec execute 106 sec latency 4382.981 ms oleg424-client.virtnet: 1 65466 2.19 MB/sec execute 107 sec latency 5383.144 ms oleg424-client.virtnet: 1 65466 2.17 MB/sec execute 108 sec latency 6383.276 ms oleg424-client.virtnFailover mds1 to oleg424-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 22:00:38 (1745460038) targets are mounted 22:00:38 (1745460038) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36540 3567552 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13228 3592596 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49768 7160148 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg424-server Stopping /mnt/lustre-mds2 (opts:) on oleg424-server 22:00:49 (1745460049) shut down facet: mds2 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds2 to oleg424-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 22:01:03 (1745460063) targets are mounted 22:01:03 (1745460063) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37044 3569268 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12960 3593060 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50004 7162328 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg424-server Stopping /mnt/lustre-mds1 (opts:) on oleg424-server 22:01:14 (1745460074) shut down facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server et: 1 65466 2.15 MB/sec execute 109 sec latency 7383.420 ms oleg424-client.virtnet: 1 65466 2.13 MB/sec execute 110 sec latency 8383.583 ms oleg424-client.virtnet: 1 65466 2.11 MB/sec execute 111 sec latency 9383.767 ms oleg424-client.virtnet: 1 65466 2.09 MB/sec execute 112 sec latency 10384.010 ms oleg424-client.virtnet: 1 65466 2.08 MB/sec execute 113 sec latency 11384.157 ms oleg424-client.virtnet: 1 65466 2.06 MB/sec execute 114 sec latency 12384.339 ms oleg424-client.virtnet: 1 65466 2.04 MB/sec execute 115 sec latency 13384.572 ms oleg424-client.virtnet: 1 65466 2.02 MB/sec execute 116 sec latency 14384.702 ms oleg424-client.virtnet: 1 65466 2.01 MB/sec execute 117 sec latency 15384.857 ms oleg424-client.virtnet: 1 65466 1.99 MB/sec execute 118 sec latency 16384.989 ms oleg424-client.virtnet: 1 65466 1.97 MB/sec execute 119 sec latency 17385.104 ms oleg424-client.virtnet: 1 65466 1.96 MB/sec execute 120 sec latency 18385.266 ms oleg424-client.virtnet: 1 65858 1.94 MB/sec execute 121 sec latency 18762.410 ms oleg424-client.virtnet: 1 66633 1.97 MB/sec execute 122 sec latency 9.795 ms oleg424-client.virtnet: 1 67637 2.02 MB/sec execute 123 sec latency 9.694 ms oleg424-client.virtnet: 1 68191 2.01 MB/sec execute 124 sec latency 10.043 ms oleg424-client.virtnet: 1 68873 2.03 MB/sec execute 125 sec latency 10.577 ms oleg424-client.virtnet: 1 69457 2.02 MB/sec execute 126 sec latency 9.425 ms oleg424-client.virtnet: 1 70386 2.06 MB/sec execute 127 sec latency 8.575 ms oleg424-client.virtnet: 1 71256 2.09 MB/sec execute 128 sec latency 10.661 ms oleg424-client.virtnet: 1 71858 2.08 MB/sec execute 129 sec latency 9.248 ms oleg424-client.virtnet: 1 72585 2.10 MB/sec execute 130 sec latency 8.522 ms oleg424-client.virtnet: 1 73325 2.10 MB/sec execute 131 sec latency 8.711 ms oleg424-client.virtnet: 1 74391 2.14 MB/sec execute 132 sec latency 9.006 ms oleg424-client.virtnet: 1 75045 2.16 MB/sec execute 133 sec latency 7.646 ms oleg424-client.virtnet: 1 75711 2.16 MB/sec execute 134 sec latency 10.048 ms oleg424-client.virtnet: 1 76377 2.16 MB/sec execute 135 sec latency 9.381 ms oleg424-client.virtnet: 1 77126 2.17 MB/sec execute 136 sec latency 8.608 ms oleg424-client.virtnet: 1 78238 2.22 MB/sec execute 137 sec latency 8.444 ms oleg424-client.virtnet: 1 78828 2.22 MB/sec execute 138 sec latency 12.387 ms oleg424-client.virtnet: 1 79496 2.24 MB/sec execute 139 sec latency 8.771 ms oleg424-client.virtnet: 1 80107 2.23 MB/sec execute 140 sec latency 9.065 ms oleg424-client.virtnet: 1 81005 2.26 MB/sec execute 141 sec latency 8.637 ms oleg424-client.virtnet: 1 81928 2.29 MB/sec execute 142 sec latency 8.482 ms oleg424-client.virtnet: 1 82145 2.27 MB/sec execute 143 sec latency 561.687 ms oleg424-client.virtnet: 1 82145 2.26 MB/sec execute 144 sec latency 1561.915 ms oleg424-client.virtnet: 1 82145 2.24 MB/sec execute 145 sec latency 2562.096 ms oleg424-client.virtnet: 1 82538 2.23 MB/sec execute 146 sec latency 2944.920 ms oleg424-client.virtnet: 1 83215 2.25 MB/sec execute 147 sec latency 9.605 ms oleg424-client.virtnet: 1 83801 2.24 MB/sec execute 148 sec latency 10.435 ms oleg424-client.virtnet: 1 84736 2.27 MB/sec execute 149 sec latency 10.082 ms oleg424-client.virtnet: 1 85543 2.29 MB/sec execute 150 sec latency 10.069 ms oleg424-client.virtnet: 1 86143 2.29 MB/sec execute 151 sec latency 10.005 ms oleg424-client.virtnet: 1 86842 2.30 MB/sec execute 152 sec latency 10.012 ms oleg424-client.virtnet: 1 86965 2.28 MB/sec execute 153 sec latency 756.9Failover mds1 to oleg424-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 22:01:28 (1745460088) targets are mounted 22:01:28 (1745460088) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36736 3569676 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13096 3592688 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49832 7162364 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg424-server Stopping /mnt/lustre-mds2 (opts:) on oleg424-server 22:01:40 (1745460100) shut down facet: mds2 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds2 to oleg424-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 22:01:54 (1745460114) targets are mounted 22:01:54 (1745460114) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 12 ms oleg424-client.virtnet: 1 86965 2.27 MB/sec execute 154 sec latency 1757.083 ms oleg424-client.virtnet: 1 86965 2.26 MB/sec execute 155 sec latency 2757.246 ms oleg424-client.virtnet: 1 86965 2.24 MB/sec execute 156 sec latency 3757.401 ms oleg424-client.virtnet: 1 86965 2.23 MB/sec execute 157 sec latency 4757.631 ms oleg424-client.virtnet: 1 86965 2.21 MB/sec execute 158 sec latency 5757.834 ms oleg424-client.virtnet: 1 86965 2.20 MB/sec execute 159 sec latency 6757.989 ms oleg424-client.virtnet: 1 86965 2.18 MB/sec execute 160 sec latency 7758.132 ms oleg424-client.virtnet: 1 86965 2.17 MB/sec execute 161 sec latency 8758.279 ms oleg424-client.virtnet: 1 86965 2.16 MB/sec execute 162 sec latency 9758.424 ms oleg424-client.virtnet: 1 86965 2.14 MB/sec execute 163 sec latency 10758.599 ms oleg424-client.virtnet: 1 86965 2.13 MB/sec execute 164 sec latency 11758.746 ms oleg424-client.virtnet: 1 86965 2.12 MB/sec execute 165 sec latency 12758.870 ms oleg424-client.virtnet: 1 86965 2.11 MB/sec execute 166 sec latency 13759.012 ms oleg424-client.virtnet: 1 86965 2.09 MB/sec execute 167 sec latency 14759.195 ms oleg424-client.virtnet: 1 86965 2.08 MB/sec execute 168 sec latency 15759.402 ms oleg424-client.virtnet: 1 86965 2.07 MB/sec execute 169 sec latency 16759.635 ms oleg424-client.virtnet: 1 86965 2.06 MB/sec execute 170 sec latency 17759.942 ms oleg424-client.virtnet: 1 87160 2.04 MB/sec execute 171 sec latency 18452.648 ms oleg424-client.virtnet: 1 87752 2.05 MB/sec execute 172 sec latency 13.191 ms oleg424-client.virtnet: 1 88859 2.09 MB/sec execute 173 sec latency 10.498 ms oleg424-client.virtnet: 1 89414 2.09 MB/sec execute 174 sec latency 11.627 ms oleg424-client.virtnet: 1 90113 2.10 MB/sec execute 175 sec latency 9.380 ms oleg424-client.virtnet: 1 90712 2.10 MB/sec execute 176 sec latency 9.824 ms oleg424-client.virtnet: 1 91485 2.11 MB/sec execute 177 sec latency 10.854 ms oleg424-client.virtnet: 1 92429 2.14 MB/sec execute 178 sec latency 12.423 ms oleg424-client.virtnet: 1 92951 2.14 MB/sec execute 179 sec latency 10.570 ms oleg424-client.virtnet: 1 93521 2.14 MB/sec execute 180 sec latency 9.641 ms oleg424-client.virtnet: 1 94062 2.14 MB/sec execute 181 sec latency 16.473 ms oleg424-client.virtnet: 1 94463 2.13 MB/sec execute 182 sec latency 11.519 ms oleg424-client.virtnet: 1 95043 2.15 MB/sec execute 183 sec latency 15.015 ms oleg424-client.virtnet: 1 95941 2.18 MB/sec execute 184 sec latency 12.225 ms oleg424-client.virtnet: 1 96538 2.17 MB/sec execute 185 sec latency 9.887 ms oleg424-client.virtnet: 1 97091 2.17 MB/sec execute 186 sec latency 11.120 ms oleg424-client.virtnet: 1 97809 2.18 MB/sec execute 187 sec latency 9.480 ms oleg424-client.virtnet: 1 98691 2.20 MB/sec execute 188 sec latency 9.358 ms oleg424-client.virtnet: 1 99444 2.22 MB/sec execute 189 sec latency 13.752 ms oleg424-client.virtnet: 1 99866 2.21 MB/sec execute 190 sec latency 13.160 ms oleg424-client.virtnet: 1 100456 2.21 MB/sec execute 191 sec latency 11.209 ms oleg424-client.virtnet: 1 101090 2.22 MB/sec execute 192 sec latency 12.922 ms oleg424-client.virtnet: 1 101422 2.21 MB/sec execute 193 sec latency 492.329 ms oleg424-client.virtnet: 1 101422 2.20 MB/sec execute 194 sec latency 1492.490 ms oleg424-client.virtnet: 1 101422 2.19 MB/sec execute 195 sec latency 2492.682 ms oleg424-client.virtnet: 1 101422 2.18 MB/sec execute 196 sec latency 3492.856 ms oleg424-client.virtnet: 1 102015 2.18 MB/sec execute 197 sec latency 3653.886 ms oleg424-client.virtnet: 1 103077UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36540 3569892 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13108 3590860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7160752 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg424-server Stopping /mnt/lustre-mds1 (opts:) on oleg424-server 22:02:05 (1745460125) shut down facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds1 to oleg424-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 22:02:19 (1745460139) targets are mounted 22:02:19 (1745460139) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36568 3569640 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12932 3592940 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162580 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg424-server Stopping /mnt/lustre-mds2 (opts:) on oleg424-server 22:02:31 (1745460151) shut down facet: mds2 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds2 to oleg424-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 22:02:44 (1745460164) targets are mounted 22:02:44 (1745460164) facet_failover done 2.22 MB/sec execute 198 sec latency 9.911 ms oleg424-client.virtnet: 1 103651 2.21 MB/sec execute 199 sec latency 14.345 ms oleg424-client.virtnet: 1 104322 2.22 MB/sec execute 200 sec latency 9.778 ms oleg424-client.virtnet: 1 104800 2.22 MB/sec execute 201 sec latency 205.442 ms oleg424-client.virtnet: 1 105484 2.22 MB/sec execute 202 sec latency 314.672 ms oleg424-client.virtnet: 1 106597 2.25 MB/sec execute 203 sec latency 8.835 ms oleg424-client.virtnet: 1 106707 2.25 MB/sec execute 204 sec latency 834.445 ms oleg424-client.virtnet: 1 106707 2.24 MB/sec execute 205 sec latency 1834.616 ms oleg424-client.virtnet: 1 106707 2.23 MB/sec execute 206 sec latency 2834.803 ms oleg424-client.virtnet: 1 106707 2.22 MB/sec execute 207 sec latency 3835.040 ms oleg424-client.virtnet: 1 106707 2.20 MB/sec execute 208 sec latency 4835.255 ms oleg424-client.virtnet: 1 106707 2.19 MB/sec execute 209 sec latency 5835.513 ms oleg424-client.virtnet: 1 106707 2.18 MB/sec execute 210 sec latency 6835.808 ms oleg424-client.virtnet: 1 106707 2.17 MB/sec execute 211 sec latency 7836.050 ms oleg424-client.virtnet: 1 106707 2.16 MB/sec execute 212 sec latency 8836.263 ms oleg424-client.virtnet: 1 106707 2.15 MB/sec execute 213 sec latency 9836.493 ms oleg424-client.virtnet: 1 106707 2.14 MB/sec execute 214 sec latency 10836.662 ms oleg424-client.virtnet: 1 106707 2.13 MB/sec execute 215 sec latency 11836.818 ms oleg424-client.virtnet: 1 106707 2.12 MB/sec execute 216 sec latency 12837.015 ms oleg424-client.virtnet: 1 106707 2.11 MB/sec execute 217 sec latency 13837.227 ms oleg424-client.virtnet: 1 106707 2.10 MB/sec execute 218 sec latency 14837.430 ms oleg424-client.virtnet: 1 106707 2.09 MB/sec execute 219 sec latency 15837.606 ms oleg424-client.virtnet: 1 106707 2.08 MB/sec execute 220 sec latency 16837.834 ms oleg424-client.virtnet: 1 106707 2.08 MB/sec execute 221 sec latency 17838.051 ms oleg424-client.virtnet: 1 106855 2.07 MB/sec execute 222 sec latency 18508.806 ms oleg424-client.virtnet: 1 107379 2.06 MB/sec execute 223 sec latency 9.179 ms oleg424-client.virtnet: 1 108102 2.07 MB/sec execute 224 sec latency 8.344 ms oleg424-client.virtnet: 1 108808 2.07 MB/sec execute 225 sec latency 10.370 ms oleg424-client.virtnet: 1 109971 2.11 MB/sec execute 226 sec latency 8.473 ms oleg424-client.virtnet: 1 110534 2.11 MB/sec execute 227 sec latency 8.463 ms oleg424-client.virtnet: 1 111184 2.11 MB/sec execute 228 sec latency 9.556 ms oleg424-client.virtnet: 1 111874 2.11 MB/sec execute 229 sec latency 8.680 ms oleg424-client.virtnet: 1 112773 2.13 MB/sec execute 230 sec latency 8.086 ms oleg424-client.virtnet: 1 113832 2.15 MB/sec execute 231 sec latency 7.885 ms oleg424-client.virtnet: 1 114456 2.15 MB/sec execute 232 sec latency 8.724 ms oleg424-client.virtnet: 1 115204 2.16 MB/sec execute 233 sec latency 8.110 ms oleg424-client.virtnet: 1 115961 2.16 MB/sec execute 234 sec latency 9.009 ms oleg424-client.virtnet: 1 117125 2.19 MB/sec execute 235 sec latency 8.699 ms oleg424-client.virtnet: 1 117756 2.19 MB/sec execute 236 sec latency 7.314 ms oleg424-client.virtnet: 1 118369 2.19 MB/sec execute 237 sec latency 12.102 ms oleg424-client.virtnet: 1 119070 2.19 MB/sec execute 238 sec latency 9.425 ms oleg424-client.virtnet: 1 120007 2.21 MB/sec execute 239 sec latency 8.196 ms oleg424-client.virtnet: 1 120955 2.23 MB/sec execute 240 sec latency 8.342 ms oleg424-client.virtnet: 1 121565 2.22 MB/sec execute 241 sec latency 11.439 ms oleg424-client.virtnet: 1 122293 2.23 MB/sec execute 242 sec latency 8.205 ms oleg424-clioleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec ent.virtnet: 1 122672 2.23 MB/sec execute 243 sec latency 419.225 ms oleg424-client.virtnet: 1 122672 2.22 MB/sec execute 244 sec latency 1419.394 ms oleg424-client.virtnet: 1 122672 2.21 MB/sec execute 245 sec latency 2419.585 ms oleg424-client.virtnet: 1 122672 2.20 MB/sec execute 246 sec latency 3419.751 ms oleg424-client.virtnet: 1 123062 2.20 MB/sec execute 247 sec latency 3908.807 ms oleg424-client.virtnet: 1 124215 2.23 MB/sec execute 248 sec latency 9.876 ms oleg424-client.virtnet: 1 124837 2.23 MB/sec execute 249 sec latency 7.387 ms oleg424-client.virtnet: 1 125461 2.23 MB/sec execute 250 sec latency 11.860 ms oleg424-client.virtnet: 1 126192 2.23 MB/sec execute 251 sec latency 9.405 ms oleg424-client.virtnet: 1 127115 2.25 MB/sec execute 252 sec latency 8.711 ms oleg424-client.virtnet: 1 128055 2.27 MB/sec execute 253 sec latency 8.436 ms oleg424-client.virtnet: 1 128677 2.26 MB/sec execute 254 sec latency 10.177 ms oleg424-client.virtnet: 1 129420 2.27 MB/sec execute 255 sec latency 8.743 ms oleg424-client.virtnet: 1 130151 2.27 MB/sec execute 256 sec latency 7.844 ms oleg424-client.virtnet: 1 131348 2.30 MB/sec execute 257 sec latency 8.564 ms oleg424-client.virtnet: 1 131958 2.30 MB/sec execute 258 sec latency 11.218 ms oleg424-client.virtnet: 1 132579 2.30 MB/sec execute 259 sec latency 11.528 ms oleg424-client.virtnet: 1 133269 2.30 MB/sec execute 260 sec latency 9.290 ms oleg424-client.virtnet: 1 134150 2.31 MB/sec execute 261 sec latency 8.875 ms oleg424-client.virtnet: 1 135082 2.33 MB/sec execute 262 sec latency 8.913 ms oleg424-client.virtnet: 1 135642 2.33 MB/sec execute 263 sec latency 9.834 ms oleg424-client.virtnet: 1 136315 2.34 MB/sec execute 264 sec latency 9.007 ms oleg424-client.virtnet: 1 136683 2.33 MB/sec execute 265 sec latency 18.546 ms oleg424-client.virtnet: 1 137205 2.33 MB/sec execute 266 sec latency 9.502 ms oleg424-client.virtnet: 1 138090 2.35 MB/sec execute 267 sec latency 12.595 ms oleg424-client.virtnet: 1 138808 2.35 MB/sec execute 268 sec latency 9.454 ms oleg424-client.virtnet: 1 139452 2.35 MB/sec execute 269 sec latency 10.852 ms oleg424-client.virtnet: 1 140166 2.36 MB/sec execute 270 sec latency 9.183 ms oleg424-client.virtnet: 1 140828 2.36 MB/sec execute 271 sec latency 14.648 ms oleg424-client.virtnet: 1 141876 2.38 MB/sec execute 272 sec latency 13.023 ms oleg424-client.virtnet: 1 142403 2.38 MB/sec execute 273 sec latency 8.499 ms oleg424-client.virtnet: 1 143046 2.38 MB/sec execute 274 sec latency 11.282 ms oleg424-client.virtnet: 1 143710 2.38 MB/sec execute 275 sec latency 8.995 ms oleg424-client.virtnet: 1 144401 2.38 MB/sec execute 276 sec latency 10.008 ms oleg424-client.virtnet: 1 145532 2.41 MB/sec execute 277 sec latency 12.155 ms oleg424-client.virtnet: 1 146060 2.41 MB/sec execute 278 sec latency 8.489 ms oleg424-client.virtnet: 1 146638 2.41 MB/sec execute 279 sec latency 11.409 ms oleg424-client.virtnet: 1 147276 2.41 MB/sec execute 280 sec latency 9.086 ms oleg424-client.virtnet: 1 148028 2.41 MB/sec execute 281 sec latency 7.777 ms oleg424-client.virtnet: 1 149139 2.44 MB/sec execute 282 sec latency 8.344 ms oleg424-client.virtnet: 1 149524 2.43 MB/sec execute 283 sec latency 11.490 ms oleg424-client.virtnet: 1 150233 2.43 MB/sec execute 284 sec latency 8.823 ms oleg424-client.virtnet: 1 150986 2.43 MB/sec execute 285 sec latency 8.379 ms oleg424-client.virtnet: 1 151752 2.45 MB/sec execute 286 sec latency 10.871 ms oleg424-client.virtnet: 1 152799 2.46 MB/sec execute 287 sec latency 8.282 ms oleg424-client.virtnet: 1 153408 2.46 MB/sec execute 288 sec latency 8.579 ms oleg424-client.virtnet: 1 154122 2.47 MB/sec execute 289 sec latency 10.648 ms oleg424-client.virtnet: 1 154713 2.46 MB/sec execute 290 sec latency 9.321 ms oleg424-client.virtnet: 1 155164 2.46 MB/sec execute 291 sec latency 17.033 ms oleg424-client.virtnet: 1 156243 2.48 MB/sec execute 292 sec latency 11.136 ms oleg424-client.virtnet: 1 156873 2.48 MB/sec execute 293 sec latency 10.885 ms oleg424-client.virtnet: 1 157600 2.49 MB/sec execute 294 sec latency 8.392 ms oleg424-client.virtnet: 1 158287 2.48 MB/sec execute 295 sec latency 9.045 ms oleg424-client.virtnet: 1 159436 2.50 MB/sec execute 296 sec latency 8.331 ms oleg424-client.virtnet: 1 160090 2.51 MB/sec execute 297 sec latency 8.224 ms oleg424-client.virtnet: 1 160724 2.51 MB/sec execute 298 sec latency 11.246 ms oleg424-client.virtnet: 1 161397 2.51 MB/sec execute 299 sec latency 10.004 ms oleg424-client.virtnet: 1 cleanup 300 sec oleg424-client.virtnet: 0 cleanup 300 sec oleg424-client.virtnet: oleg424-client.virtnet: Operation Count AvgLat MaxLat oleg424-client.virtnet: ---------------------------------------- oleg424-client.virtnet: NTCreateX 24297 7.632 18963.393 oleg424-client.virtnet: Close 17865 1.921 18762.383 oleg424-client.virtnet: Rename 1028 6.986 18.537 oleg424-client.virtnet: Unlink 4904 1.548 8.163 oleg424-client.virtnet: Qpathinfo 22067 1.468 3908.798 oleg424-client.virtnet: Qfileinfo 3843 0.122 1.781 oleg424-client.virtnet: Qfsinfo 4022 0.722 2202.600 oleg424-client.virtnet: Sfileinfo 1994 4.083 14.325 oleg424-client.virtnet: Find 8511 0.623 2944.912 oleg424-client.virtnet: WriteX 12005 0.714 7.061 oleg424-client.virtnet: ReadX 37920 0.016 3.634 oleg424-client.virtnet: LockX 78 0.868 1.390 oleg424-client.virtnet: UnlockX 78 0.959 4.438 oleg424-client.virtnet: Flush 1710 4.686 314.658 oleg424-client.virtnet: oleg424-client.virtnet: Throughput 2.51186 MB/sec 1 clients 1 procs max_latency=18963.425 ms oleg424-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg424-client.virtnet at Wed Apr 23 22:03:42 EDT 2025 with return code 0 oleg424-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg424-client.virtnet oleg424-client.virtnet: /mnt/lustre/d70b.replay-single/oleg424-client.virtnet /mnt/lustre/d70b.replay-single/oleg424-client.virtnet oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg424-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg424-client.virtnet: removed directory: 'clients/client0' oleg424-client.virtnet: removed directory: 'clients' oleg424-client.virtnet: removed 'client.txt' oleg424-client.virtnet: /mnt/lustre/d70b.replay-single/oleg424-client.virtnet oleg424-client.virtnet: dbench successfully finished