== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 01:27:45 (1708842465) Starting client oleg234-client.virtnet: -o user_xattr,flock oleg234-server@tcp:/lustre /mnt/lustre Started clients oleg234-client.virtnet: 192.168.202.134@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg234-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=21782 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2468 1285220 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26424 3580596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1832 3592500 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28256 7173096 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 01:27:54 (1708842474) shut down Failover mds1 to oleg234-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 01:28:09 (1708842489) targets are mounted 01:28:09 (1708842489) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26716 3569612 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2068 3592868 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28784 7162480 1% /mnt/lustre oleg234-client.virtnet: looking for dbench program oleg234-client.virtnet: /usr/bin/dbench oleg234-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg234-client.virtnet oleg234-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg234-client.virtnet' oleg234-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg234-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg234-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg234-client.virtnet at Sun Feb 25 01:27:46 EST 2024 oleg234-client.virtnet: waiting for dbench pid 21806 oleg234-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg234-client.virtnet: oleg234-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg234-client.virtnet: failed to create barrier semaphore oleg234-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg234-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg234-client.virtnet: releasing clients oleg234-client.virtnet: 1 316 11.48 MB/sec warmup 1 sec latency 21.375 ms oleg234-client.virtnet: 1 692 10.59 MB/sec warmup 2 sec latency 18.143 ms oleg234-client.virtnet: 1 990 7.30 MB/sec warmup 3 sec latency 17.057 ms oleg234-client.virtnet: 1 1493 6.51 MB/sec warmup 4 sec latency 15.359 ms oleg234-client.virtnet: 1 1888 5.28 MB/sec warmup 5 sec latency 16.580 ms oleg234-client.virtnet: 1 2353 4.65 MB/sec warmup 6 sec latency 12.740 ms oleg234-client.virtnet: 1 2882 4.60 MB/sec warmup 7 sec latency 15.167 ms oleg234-client.virtnet: 1 3541 4.80 MB/sec warmup 8 sec latency 14.817 ms oleg234-client.virtnet: 1 3864 4.42 MB/sec warmup 9 sec latency 15.046 ms oleg234-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 150.499 ms oleg234-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 1150.702 ms oleg234-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 2150.899 ms oleg234-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 3151.062 ms oleg234-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 4151.243 ms oleg234-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 5151.453 ms oleg234-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 6151.603 ms oleg234-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 7151.786 ms oleg234-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 8152.014 ms oleg234-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 9152.182 ms oleg234-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 10152.345 ms oleg234-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 11152.522 ms oleg234-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 12152.703 ms oleg234-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 13152.924 ms oleg234-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 14153.114 ms oleg234-client.virtnet: 1 4083 1.60 MB/sec warmup 25 sec latency 15153.396 ms oleg234-client.virtnet: 1 4083 1.54 MB/sec warmup 26 sec latency 16153.699 ms oleg234-client.virtnet: 1 4306 1.50 MB/sec warmup 27 sec latency 16599.679 ms oleg234-client.virtnet: 1 4644 1.50 MB/sec warmup 28 sec latency 16.076 ms oleg234-client.virtnet: 1 5059 1.56 MB/sec warmup 29 sec latency 15.242 ms oleg234-client.virtnet: 1 5368 1.51 MB/sec warmup 30 sec latency 17.467 ms oleg234-client.virtnet: 1 5785 1.48 MB/sec warmup 31 sec latency 10.581 ms oleg234-client.virtnet: 1 6324 1.60 MB/sec warmup 32 sec latency 119.395 ms oleg234-client.virtnet: 1 6765 1.62 MB/sec warmup 33 sec latency 455.21test_70b fail mds2 2 times Failing mds2 on oleg234-server Stopping /mnt/lustre-mds2 (opts:) on oleg234-server 01:28:22 (1708842502) shut down Failover mds2 to oleg234-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 01:28:37 (1708842517) targets are mounted 01:28:37 (1708842517) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2024 1285664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37092 3569744 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13192 3591520 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50284 7161264 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 01:28:50 (1708842530) shut down Failover mds1 to oleg234-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 7 ms oleg234-client.virtnet: 1 7336 1.73 MB/sec warmup 34 sec latency 13.736 ms oleg234-client.virtnet: 1 7535 1.69 MB/sec warmup 35 sec latency 161.601 ms oleg234-client.virtnet: 1 7535 1.64 MB/sec warmup 36 sec latency 1161.844 ms oleg234-client.virtnet: 1 7535 1.60 MB/sec warmup 37 sec latency 2162.116 ms oleg234-client.virtnet: 1 7535 1.56 MB/sec warmup 38 sec latency 3162.396 ms oleg234-client.virtnet: 1 7535 1.52 MB/sec warmup 39 sec latency 4162.594 ms oleg234-client.virtnet: 1 7535 1.48 MB/sec warmup 40 sec latency 5162.806 ms oleg234-client.virtnet: 1 7535 1.44 MB/sec warmup 41 sec latency 6163.116 ms oleg234-client.virtnet: 1 7535 1.41 MB/sec warmup 42 sec latency 7163.422 ms oleg234-client.virtnet: 1 7535 1.38 MB/sec warmup 43 sec latency 8163.689 ms oleg234-client.virtnet: 1 7535 1.34 MB/sec warmup 44 sec latency 9163.883 ms oleg234-client.virtnet: 1 7535 1.31 MB/sec warmup 45 sec latency 10164.165 ms oleg234-client.virtnet: 1 7535 1.29 MB/sec warmup 46 sec latency 11164.353 ms oleg234-client.virtnet: 1 7535 1.26 MB/sec warmup 47 sec latency 12164.546 ms oleg234-client.virtnet: 1 7535 1.23 MB/sec warmup 48 sec latency 13164.731 ms oleg234-client.virtnet: 1 7535 1.21 MB/sec warmup 49 sec latency 14164.919 ms oleg234-client.virtnet: 1 7535 1.18 MB/sec warmup 50 sec latency 15165.192 ms oleg234-client.virtnet: 1 7535 1.16 MB/sec warmup 51 sec latency 16165.451 ms oleg234-client.virtnet: 1 7535 1.14 MB/sec warmup 52 sec latency 17165.649 ms oleg234-client.virtnet: 1 7535 1.12 MB/sec warmup 53 sec latency 18165.843 ms oleg234-client.virtnet: 1 7535 1.10 MB/sec warmup 54 sec latency 19166.119 ms oleg234-client.virtnet: 1 7651 1.08 MB/sec warmup 55 sec latency 19706.659 ms oleg234-client.virtnet: 1 7967 1.07 MB/sec warmup 56 sec latency 18.473 ms oleg234-client.virtnet: 1 8285 1.07 MB/sec warmup 57 sec latency 16.262 ms oleg234-client.virtnet: 1 8676 1.11 MB/sec warmup 58 sec latency 16.015 ms oleg234-client.virtnet: 1 8969 1.09 MB/sec warmup 59 sec latency 17.693 ms oleg234-client.virtnet: 1 9857 4.12 MB/sec execute 1 sec latency 16.333 ms oleg234-client.virtnet: 1 10466 3.81 MB/sec execute 2 sec latency 16.055 ms oleg234-client.virtnet: 1 10887 3.98 MB/sec execute 3 sec latency 15.472 ms oleg234-client.virtnet: 1 11130 3.06 MB/sec execute 4 sec latency 15.235 ms oleg234-client.virtnet: 1 11463 2.55 MB/sec execute 5 sec latency 21.019 ms oleg234-client.virtnet: 1 11792 2.38 MB/sec execute 6 sec latency 25.351 ms oleg234-client.virtnet: 1 12208 2.48 MB/sec execute 7 sec latency 15.789 ms oleg234-client.virtnet: 1 12515 2.21 MB/sec execute 8 sec latency 16.561 ms oleg234-client.virtnet: 1 12954 2.12 MB/sec execute 9 sec latency 11.900 ms oleg234-client.virtnet: 1 13435 2.33 MB/sec execute 10 sec latency 15.611 ms oleg234-client.virtnet: 1 14157 2.69 MB/sec execute 11 sec latency 15.554 ms oleg234-client.virtnet: 1 14502 2.59 MB/sec execute 12 sec latency 16.005 ms oleg234-client.virtnet: 1 14789 2.41 MB/sec execute 13 sec latency 15.229 ms oleg234-client.virtnet: 1 15110 2.29 MB/sec execute 14 sec latency 17.832 ms oleg234-client.virtnet: 1 15596 2.43 MB/sec execute 15 sec latency 15.241 ms oleg234-client.virtnet: 1 15907 2.29 MB/sec execute 16 sec latency 17.664 ms oleg234-client.virtnet: 1 16422 2.19 MB/sec execute 17 sec latency 10.574 ms oleg234-client.virtnet: 1 16957 2.35 MB/sec execute 18 sec latency 15.013 ms oleg234-client.virtnet: 1 17695 2.57 MB/sec execuStarted lustre-MDT0000 01:29:05 (1708842545) targets are mounted 01:29:05 (1708842545) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2052 1285636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36836 3568596 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13200 3592668 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50036 7161264 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg234-server Stopping /mnt/lustre-mds2 (opts:) on oleg234-server 01:29:18 (1708842558) shut down Failover mds2 to oleg234-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 01:29:34 (1708842574) targets are mounted 01:29:34 (1708842574) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2024 1285664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36904 3568816 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13248 3592684 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50152 7161500 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 01:29:47 (1708842587) shut down te 19 sec latency 15.070 ms oleg234-client.virtnet: 1 18012 2.51 MB/sec execute 20 sec latency 15.452 ms oleg234-client.virtnet: 1 18405 2.41 MB/sec execute 21 sec latency 14.156 ms oleg234-client.virtnet: 1 19133 2.52 MB/sec execute 22 sec latency 9.046 ms oleg234-client.virtnet: 1 19533 2.43 MB/sec execute 23 sec latency 16.546 ms oleg234-client.virtnet: 1 19943 2.35 MB/sec execute 24 sec latency 10.986 ms oleg234-client.virtnet: 1 20287 2.33 MB/sec execute 25 sec latency 15.729 ms oleg234-client.virtnet: 1 20970 2.45 MB/sec execute 26 sec latency 15.792 ms oleg234-client.virtnet: 1 21491 2.53 MB/sec execute 27 sec latency 15.667 ms oleg234-client.virtnet: 1 21782 2.49 MB/sec execute 28 sec latency 15.492 ms oleg234-client.virtnet: 1 22067 2.41 MB/sec execute 29 sec latency 183.928 ms oleg234-client.virtnet: 1 22503 2.39 MB/sec execute 30 sec latency 15.255 ms oleg234-client.virtnet: 1 22902 2.41 MB/sec execute 31 sec latency 15.453 ms oleg234-client.virtnet: 1 23051 2.34 MB/sec execute 32 sec latency 517.488 ms oleg234-client.virtnet: 1 23051 2.27 MB/sec execute 33 sec latency 1517.673 ms oleg234-client.virtnet: 1 23051 2.20 MB/sec execute 34 sec latency 2517.970 ms oleg234-client.virtnet: 1 23051 2.14 MB/sec execute 35 sec latency 3518.216 ms oleg234-client.virtnet: 1 23051 2.08 MB/sec execute 36 sec latency 4518.465 ms oleg234-client.virtnet: 1 23051 2.03 MB/sec execute 37 sec latency 5518.663 ms oleg234-client.virtnet: 1 23051 1.97 MB/sec execute 38 sec latency 6518.874 ms oleg234-client.virtnet: 1 23051 1.92 MB/sec execute 39 sec latency 7519.022 ms oleg234-client.virtnet: 1 23051 1.87 MB/sec execute 40 sec latency 8519.199 ms oleg234-client.virtnet: 1 23051 1.83 MB/sec execute 41 sec latency 9519.478 ms oleg234-client.virtnet: 1 23051 1.78 MB/sec execute 42 sec latency 10519.715 ms oleg234-client.virtnet: 1 23051 1.74 MB/sec execute 43 sec latency 11519.934 ms oleg234-client.virtnet: 1 23051 1.70 MB/sec execute 44 sec latency 12520.169 ms oleg234-client.virtnet: 1 23051 1.67 MB/sec execute 45 sec latency 13520.410 ms oleg234-client.virtnet: 1 23051 1.63 MB/sec execute 46 sec latency 14520.574 ms oleg234-client.virtnet: 1 23051 1.59 MB/sec execute 47 sec latency 15520.768 ms oleg234-client.virtnet: 1 23051 1.56 MB/sec execute 48 sec latency 16520.973 ms oleg234-client.virtnet: 1 23051 1.53 MB/sec execute 49 sec latency 17521.183 ms oleg234-client.virtnet: 1 23051 1.50 MB/sec execute 50 sec latency 18521.402 ms oleg234-client.virtnet: 1 23051 1.47 MB/sec execute 51 sec latency 19521.618 ms oleg234-client.virtnet: 1 23207 1.44 MB/sec execute 52 sec latency 19999.828 ms oleg234-client.virtnet: 1 23663 1.44 MB/sec execute 53 sec latency 12.977 ms oleg234-client.virtnet: 1 24094 1.49 MB/sec execute 54 sec latency 15.642 ms oleg234-client.virtnet: 1 24830 1.58 MB/sec execute 55 sec latency 15.278 ms oleg234-client.virtnet: 1 25143 1.58 MB/sec execute 56 sec latency 15.422 ms oleg234-client.virtnet: 1 25381 1.56 MB/sec execute 57 sec latency 15.513 ms oleg234-client.virtnet: 1 25710 1.54 MB/sec execute 58 sec latency 15.844 ms oleg234-client.virtnet: 1 26054 1.54 MB/sec execute 59 sec latency 16.604 ms oleg234-client.virtnet: 1 26474 1.57 MB/sec execute 60 sec latency 15.171 ms oleg234-client.virtnet: 1 26866 1.55 MB/sec execute 61 sec latency 16.576 ms oleg234-client.virtnet: 1 27265 1.54 MB/sec execute 62 sec latency 21.308 ms oleg234-client.virtnet: 1 27787 1.61 MB/sec execute 63 sec latency 21.633 ms oleg234-clFailover mds1 to oleg234-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 01:30:03 (1708842603) targets are mounted 01:30:03 (1708842603) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36840 3569448 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13316 3592712 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50156 7162160 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg234-server Stopping /mnt/lustre-mds2 (opts:) on oleg234-server 01:30:16 (1708842616) shut down Failover mds2 to oleg234-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 01:30:31 (1708842631) targets are mounted 01:30:31 (1708842631) facet_failover done ient.virtnet: 1 28417 1.66 MB/sec execute 64 sec latency 15.727 ms oleg234-client.virtnet: 1 28714 1.66 MB/sec execute 65 sec latency 15.412 ms oleg234-client.virtnet: 1 28971 1.63 MB/sec execute 66 sec latency 15.565 ms oleg234-client.virtnet: 1 29315 1.62 MB/sec execute 67 sec latency 20.326 ms oleg234-client.virtnet: 1 29652 1.62 MB/sec execute 68 sec latency 15.460 ms oleg234-client.virtnet: 1 30069 1.64 MB/sec execute 69 sec latency 17.042 ms oleg234-client.virtnet: 1 30424 1.62 MB/sec execute 70 sec latency 18.899 ms oleg234-client.virtnet: 1 30835 1.62 MB/sec execute 71 sec latency 15.924 ms oleg234-client.virtnet: 1 31385 1.67 MB/sec execute 72 sec latency 16.084 ms oleg234-client.virtnet: 1 31999 1.72 MB/sec execute 73 sec latency 15.682 ms oleg234-client.virtnet: 1 32292 1.71 MB/sec execute 74 sec latency 15.803 ms oleg234-client.virtnet: 1 32626 1.69 MB/sec execute 75 sec latency 16.592 ms oleg234-client.virtnet: 1 32916 1.68 MB/sec execute 76 sec latency 21.572 ms oleg234-client.virtnet: 1 33351 1.71 MB/sec execute 77 sec latency 15.417 ms oleg234-client.virtnet: 1 33659 1.70 MB/sec execute 78 sec latency 17.640 ms oleg234-client.virtnet: 1 34029 1.68 MB/sec execute 79 sec latency 16.848 ms oleg234-client.virtnet: 1 34420 1.68 MB/sec execute 80 sec latency 18.010 ms oleg234-client.virtnet: 1 34980 1.72 MB/sec execute 81 sec latency 15.691 ms oleg234-client.virtnet: 1 35616 1.77 MB/sec execute 82 sec latency 15.115 ms oleg234-client.virtnet: 1 35903 1.76 MB/sec execute 83 sec latency 15.927 ms oleg234-client.virtnet: 1 36229 1.74 MB/sec execute 84 sec latency 23.270 ms oleg234-client.virtnet: 1 36513 1.73 MB/sec execute 85 sec latency 16.756 ms oleg234-client.virtnet: 1 36961 1.76 MB/sec execute 86 sec latency 15.666 ms oleg234-client.virtnet: 1 37287 1.74 MB/sec execute 87 sec latency 34.782 ms oleg234-client.virtnet: 1 37710 1.73 MB/sec execute 88 sec latency 10.981 ms oleg234-client.virtnet: 1 37998 1.73 MB/sec execute 89 sec latency 198.919 ms oleg234-client.virtnet: 1 37998 1.71 MB/sec execute 90 sec latency 1199.184 ms oleg234-client.virtnet: 1 37998 1.69 MB/sec execute 91 sec latency 2199.494 ms oleg234-client.virtnet: 1 37998 1.67 MB/sec execute 92 sec latency 3199.754 ms oleg234-client.virtnet: 1 37998 1.65 MB/sec execute 93 sec latency 4200.069 ms oleg234-client.virtnet: 1 37998 1.64 MB/sec execute 94 sec latency 5200.373 ms oleg234-client.virtnet: 1 37998 1.62 MB/sec execute 95 sec latency 6200.597 ms oleg234-client.virtnet: 1 37998 1.60 MB/sec execute 96 sec latency 7200.826 ms oleg234-client.virtnet: 1 37998 1.59 MB/sec execute 97 sec latency 8201.169 ms oleg234-client.virtnet: 1 37998 1.57 MB/sec execute 98 sec latency 9201.397 ms oleg234-client.virtnet: 1 37998 1.55 MB/sec execute 99 sec latency 10201.629 ms oleg234-client.virtnet: 1 37998 1.54 MB/sec execute 100 sec latency 11201.866 ms oleg234-client.virtnet: 1 37998 1.52 MB/sec execute 101 sec latency 12202.113 ms oleg234-client.virtnet: 1 37998 1.51 MB/sec execute 102 sec latency 13202.429 ms oleg234-client.virtnet: 1 37998 1.49 MB/sec execute 103 sec latency 14202.688 ms oleg234-client.virtnet: 1 37998 1.48 MB/sec execute 104 sec latency 15202.850 ms oleg234-client.virtnet: 1 37998 1.47 MB/sec execute 105 sec latency 16203.042 ms oleg234-client.virtnet: 1 37998 1.45 MB/sec execute 106 sec latency 17203.344 ms oleg234-client.virtnet: 1 37998 1.44 MB/sec execute 107 sec latency 18203.663 ms oleg234-client.virtnet: 1 37998 1.42 MB/sec oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2024 1285664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36864 3569140 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13292 3592136 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50156 7161276 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 01:30:43 (1708842643) shut down Failover mds1 to oleg234-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 01:30:58 (1708842658) targets are mounted 01:30:58 (1708842658) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36756 3569428 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13352 3590580 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50108 7160008 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg234-server Stopping /mnt/lustre-mds2 (opts:) on oleg234-server 01:31:11 (1708842671) shut down execute 108 sec latency 19203.958 ms oleg234-client.virtnet: 1 38302 1.44 MB/sec execute 109 sec latency 19717.780 ms oleg234-client.virtnet: 1 39078 1.49 MB/sec execute 110 sec latency 17.970 ms oleg234-client.virtnet: 1 39502 1.49 MB/sec execute 111 sec latency 10.829 ms oleg234-client.virtnet: 1 40046 1.48 MB/sec execute 112 sec latency 10.375 ms oleg234-client.virtnet: 1 40713 1.51 MB/sec execute 113 sec latency 10.139 ms oleg234-client.virtnet: 1 41445 1.51 MB/sec execute 114 sec latency 9.210 ms oleg234-client.virtnet: 1 42451 1.56 MB/sec execute 115 sec latency 9.337 ms oleg234-client.virtnet: 1 43021 1.59 MB/sec execute 116 sec latency 10.603 ms oleg234-client.virtnet: 1 43546 1.58 MB/sec execute 117 sec latency 12.416 ms oleg234-client.virtnet: 1 44224 1.61 MB/sec execute 118 sec latency 10.914 ms oleg234-client.virtnet: 1 44758 1.60 MB/sec execute 119 sec latency 11.444 ms oleg234-client.virtnet: 1 45316 1.63 MB/sec execute 120 sec latency 13.300 ms oleg234-client.virtnet: 1 46124 1.67 MB/sec execute 121 sec latency 13.472 ms oleg234-client.virtnet: 1 46457 1.67 MB/sec execute 122 sec latency 13.410 ms oleg234-client.virtnet: 1 46767 1.65 MB/sec execute 123 sec latency 15.570 ms oleg234-client.virtnet: 1 47072 1.65 MB/sec execute 124 sec latency 19.248 ms oleg234-client.virtnet: 1 47405 1.64 MB/sec execute 125 sec latency 15.754 ms oleg234-client.virtnet: 1 47813 1.66 MB/sec execute 126 sec latency 15.357 ms oleg234-client.virtnet: 1 48165 1.65 MB/sec execute 127 sec latency 17.283 ms oleg234-client.virtnet: 1 48572 1.65 MB/sec execute 128 sec latency 14.901 ms oleg234-client.virtnet: 1 49127 1.67 MB/sec execute 129 sec latency 15.741 ms oleg234-client.virtnet: 1 49777 1.70 MB/sec execute 130 sec latency 12.636 ms oleg234-client.virtnet: 1 50075 1.70 MB/sec execute 131 sec latency 20.715 ms oleg234-client.virtnet: 1 50401 1.69 MB/sec execute 132 sec latency 16.363 ms oleg234-client.virtnet: 1 50683 1.68 MB/sec execute 133 sec latency 15.496 ms oleg234-client.virtnet: 1 51115 1.70 MB/sec execute 134 sec latency 15.460 ms oleg234-client.virtnet: 1 51427 1.69 MB/sec execute 135 sec latency 17.521 ms oleg234-client.virtnet: 1 51842 1.68 MB/sec execute 136 sec latency 12.811 ms oleg234-client.virtnet: 1 52192 1.68 MB/sec execute 137 sec latency 15.962 ms oleg234-client.virtnet: 1 52876 1.71 MB/sec execute 138 sec latency 16.277 ms oleg234-client.virtnet: 1 53380 1.73 MB/sec execute 139 sec latency 15.938 ms oleg234-client.virtnet: 1 53645 1.73 MB/sec execute 140 sec latency 16.806 ms oleg234-client.virtnet: 1 53960 1.72 MB/sec execute 141 sec latency 15.827 ms oleg234-client.virtnet: 1 54068 1.71 MB/sec execute 142 sec latency 626.532 ms oleg234-client.virtnet: 1 54489 1.71 MB/sec execute 143 sec latency 14.268 ms oleg234-client.virtnet: 1 54929 1.72 MB/sec execute 144 sec latency 16.119 ms oleg234-client.virtnet: 1 55099 1.70 MB/sec execute 145 sec latency 489.889 ms oleg234-client.virtnet: 1 55099 1.69 MB/sec execute 146 sec latency 1490.228 ms oleg234-client.virtnet: 1 55099 1.68 MB/sec execute 147 sec latency 2490.523 ms oleg234-client.virtnet: 1 55099 1.67 MB/sec execute 148 sec latency 3490.764 ms oleg234-client.virtnet: 1 55099 1.66 MB/sec execute 149 sec latency 4491.039 ms oleg234-client.virtnet: 1 55099 1.65 MB/sec execute 150 sec latency 5491.322 ms oleg234-client.virtnet: 1 55099 1.64 MB/sec execute 151 sec latency 6491.570 ms oleg234-client.virtnet: 1 55099 1.63 MB/sec execute 152 sec latency 7491.869 ms oleg234-client.virtnet: 1 5509Failover mds2 to oleg234-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 01:31:27 (1708842687) targets are mounted 01:31:27 (1708842687) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2024 1285664 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36680 3567496 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13476 3592436 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50156 7159932 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 01:31:39 (1708842699) shut down Failover mds1 to oleg234-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 01:31:55 (1708842715) targets are mounted 01:31:55 (1708842715) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2048 1285640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36852 3568528 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13316 3592740 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50168 7161268 1% /mnt/lustre 9 1.62 MB/sec execute 153 sec latency 8492.083 ms oleg234-client.virtnet: 1 55099 1.60 MB/sec execute 154 sec latency 9492.433 ms oleg234-client.virtnet: 1 55099 1.59 MB/sec execute 155 sec latency 10492.670 ms oleg234-client.virtnet: 1 55099 1.58 MB/sec execute 156 sec latency 11492.932 ms oleg234-client.virtnet: 1 55099 1.57 MB/sec execute 157 sec latency 12493.159 ms oleg234-client.virtnet: 1 55099 1.56 MB/sec execute 158 sec latency 13493.407 ms oleg234-client.virtnet: 1 55099 1.55 MB/sec execute 159 sec latency 14493.593 ms oleg234-client.virtnet: 1 55099 1.54 MB/sec execute 160 sec latency 15493.783 ms oleg234-client.virtnet: 1 55099 1.53 MB/sec execute 161 sec latency 16494.005 ms oleg234-client.virtnet: 1 55099 1.53 MB/sec execute 162 sec latency 17494.200 ms oleg234-client.virtnet: 1 55099 1.52 MB/sec execute 163 sec latency 18494.400 ms oleg234-client.virtnet: 1 55099 1.51 MB/sec execute 164 sec latency 19494.584 ms oleg234-client.virtnet: 1 55429 1.50 MB/sec execute 165 sec latency 19732.674 ms oleg234-client.virtnet: 1 56012 1.52 MB/sec execute 166 sec latency 16.025 ms oleg234-client.virtnet: 1 56618 1.53 MB/sec execute 167 sec latency 15.775 ms oleg234-client.virtnet: 1 57025 1.55 MB/sec execute 168 sec latency 16.103 ms oleg234-client.virtnet: 1 57252 1.54 MB/sec execute 169 sec latency 16.076 ms oleg234-client.virtnet: 1 57694 1.54 MB/sec execute 170 sec latency 15.085 ms oleg234-client.virtnet: 1 58031 1.54 MB/sec execute 171 sec latency 15.222 ms oleg234-client.virtnet: 1 58440 1.55 MB/sec execute 172 sec latency 17.141 ms oleg234-client.virtnet: 1 58806 1.54 MB/sec execute 173 sec latency 17.860 ms oleg234-client.virtnet: 1 59216 1.54 MB/sec execute 174 sec latency 14.823 ms oleg234-client.virtnet: 1 59965 1.57 MB/sec execute 175 sec latency 15.483 ms oleg234-client.virtnet: 1 60475 1.58 MB/sec execute 176 sec latency 13.487 ms oleg234-client.virtnet: 1 60751 1.58 MB/sec execute 177 sec latency 15.255 ms oleg234-client.virtnet: 1 61078 1.57 MB/sec execute 178 sec latency 26.808 ms oleg234-client.virtnet: 1 61401 1.57 MB/sec execute 179 sec latency 15.762 ms oleg234-client.virtnet: 1 61806 1.58 MB/sec execute 180 sec latency 15.869 ms oleg234-client.virtnet: 1 62284 1.58 MB/sec execute 181 sec latency 12.860 ms oleg234-client.virtnet: 1 62727 1.58 MB/sec execute 182 sec latency 15.269 ms oleg234-client.virtnet: 1 63255 1.60 MB/sec execute 183 sec latency 15.780 ms oleg234-client.virtnet: 1 63884 1.62 MB/sec execute 184 sec latency 15.265 ms oleg234-client.virtnet: 1 64194 1.61 MB/sec execute 185 sec latency 15.751 ms oleg234-client.virtnet: 1 64497 1.61 MB/sec execute 186 sec latency 15.911 ms oleg234-client.virtnet: 1 64828 1.60 MB/sec execute 187 sec latency 17.442 ms oleg234-client.virtnet: 1 65267 1.62 MB/sec execute 188 sec latency 15.019 ms oleg234-client.virtnet: 1 65587 1.61 MB/sec execute 189 sec latency 17.389 ms oleg234-client.virtnet: 1 65953 1.60 MB/sec execute 190 sec latency 11.481 ms oleg234-client.virtnet: 1 66326 1.60 MB/sec execute 191 sec latency 15.814 ms oleg234-client.virtnet: 1 66881 1.62 MB/sec execute 192 sec latency 16.021 ms oleg234-client.virtnet: 1 67525 1.64 MB/sec execute 193 sec latency 14.789 ms oleg234-client.virtnet: 1 67801 1.64 MB/sec execute 194 sec latency 15.681 ms oleg234-client.virtnet: 1 68118 1.63 MB/sec execute 195 sec latency 21.435 ms oleg234-client.virtnet: 1 68400 1.63 MB/sec execute 196 sec latency 19.400 ms oleg234-client.virtnet: 1 68846 1.64 MB/sec execute 197 sec latency 16.070 ms otest_70b fail mds2 10 times Failing mds2 on oleg234-server Stopping /mnt/lustre-mds2 (opts:) on oleg234-server 01:32:08 (1708842728) shut down Failover mds2 to oleg234-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 01:32:22 (1708842742) targets are mounted 01:32:22 (1708842742) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1285304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2032 1285656 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36984 3569472 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13196 3593052 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50180 7162524 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 01:32:33 (1708842753) shut down Failover mds1 to oleg234-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 01:32:48 (1708842768) targets are mounted 01:32:48 (1708842768) facet_failover done leg234-client.virtnet: 1 69058 1.63 MB/sec execute 198 sec latency 379.179 ms oleg234-client.virtnet: 1 69335 1.63 MB/sec execute 199 sec latency 597.158 ms oleg234-client.virtnet: 1 69774 1.63 MB/sec execute 200 sec latency 11.747 ms oleg234-client.virtnet: 1 70298 1.64 MB/sec execute 201 sec latency 177.795 ms oleg234-client.virtnet: 1 70298 1.63 MB/sec execute 202 sec latency 1178.011 ms oleg234-client.virtnet: 1 70298 1.62 MB/sec execute 203 sec latency 2178.197 ms oleg234-client.virtnet: 1 70298 1.61 MB/sec execute 204 sec latency 3178.466 ms oleg234-client.virtnet: 1 70298 1.61 MB/sec execute 205 sec latency 4178.628 ms oleg234-client.virtnet: 1 70298 1.60 MB/sec execute 206 sec latency 5178.783 ms oleg234-client.virtnet: 1 70298 1.59 MB/sec execute 207 sec latency 6178.916 ms oleg234-client.virtnet: 1 70298 1.58 MB/sec execute 208 sec latency 7179.062 ms oleg234-client.virtnet: 1 70298 1.58 MB/sec execute 209 sec latency 8179.221 ms oleg234-client.virtnet: 1 70298 1.57 MB/sec execute 210 sec latency 9179.424 ms oleg234-client.virtnet: 1 70298 1.56 MB/sec execute 211 sec latency 10179.553 ms oleg234-client.virtnet: 1 70298 1.55 MB/sec execute 212 sec latency 11179.723 ms oleg234-client.virtnet: 1 70298 1.55 MB/sec execute 213 sec latency 12179.910 ms oleg234-client.virtnet: 1 70298 1.54 MB/sec execute 214 sec latency 13180.113 ms oleg234-client.virtnet: 1 70298 1.53 MB/sec execute 215 sec latency 14180.300 ms oleg234-client.virtnet: 1 70298 1.53 MB/sec execute 216 sec latency 15180.501 ms oleg234-client.virtnet: 1 70298 1.52 MB/sec execute 217 sec latency 16180.730 ms oleg234-client.virtnet: 1 70298 1.51 MB/sec execute 218 sec latency 17180.950 ms oleg234-client.virtnet: 1 70298 1.50 MB/sec execute 219 sec latency 18181.110 ms oleg234-client.virtnet: 1 70442 1.50 MB/sec execute 220 sec latency 18969.321 ms oleg234-client.virtnet: 1 71353 1.53 MB/sec execute 221 sec latency 8.165 ms oleg234-client.virtnet: 1 71932 1.52 MB/sec execute 222 sec latency 10.797 ms oleg234-client.virtnet: 1 72696 1.54 MB/sec execute 223 sec latency 8.793 ms oleg234-client.virtnet: 1 73464 1.54 MB/sec execute 224 sec latency 8.233 ms oleg234-client.virtnet: 1 74638 1.58 MB/sec execute 225 sec latency 9.433 ms oleg234-client.virtnet: 1 75205 1.58 MB/sec execute 226 sec latency 9.567 ms oleg234-client.virtnet: 1 75797 1.58 MB/sec execute 227 sec latency 24.427 ms oleg234-client.virtnet: 1 76604 1.59 MB/sec execute 228 sec latency 8.817 ms oleg234-client.virtnet: 1 77542 1.61 MB/sec execute 229 sec latency 8.270 ms oleg234-client.virtnet: 1 78452 1.63 MB/sec execute 230 sec latency 8.553 ms oleg234-client.virtnet: 1 78994 1.63 MB/sec execute 231 sec latency 16.632 ms oleg234-client.virtnet: 1 79664 1.64 MB/sec execute 232 sec latency 14.922 ms oleg234-client.virtnet: 1 80461 1.64 MB/sec execute 233 sec latency 8.577 ms oleg234-client.virtnet: 1 81602 1.68 MB/sec execute 234 sec latency 10.258 ms oleg234-client.virtnet: 1 82126 1.68 MB/sec execute 235 sec latency 10.572 ms oleg234-client.virtnet: 1 82803 1.68 MB/sec execute 236 sec latency 10.817 ms oleg234-client.virtnet: 1 83529 1.69 MB/sec execute 237 sec latency 10.500 ms oleg234-client.virtnet: 1 84409 1.70 MB/sec execute 238 sec latency 9.500 ms oleg234-client.virtnet: 1 85343 1.73 MB/sec execute 239 sec latency 11.157 ms oleg234-client.virtnet: 1 85616 1.72 MB/sec execute 240 sec latency 15.755 ms oleg234-client.virtnet: 1 86002 1.72 MB/sec execute 241 sec latency 13.838 ms oleg234-client.virtnet: 1 86601 1.73 MB/soleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec ec execute 242 sec latency 12.512 ms oleg234-client.virtnet: 1 87267 1.73 MB/sec execute 243 sec latency 10.633 ms oleg234-client.virtnet: 1 87935 1.74 MB/sec execute 244 sec latency 11.058 ms oleg234-client.virtnet: 1 88951 1.77 MB/sec execute 245 sec latency 10.408 ms oleg234-client.virtnet: 1 89484 1.76 MB/sec execute 246 sec latency 13.101 ms oleg234-client.virtnet: 1 90001 1.76 MB/sec execute 247 sec latency 13.224 ms oleg234-client.virtnet: 1 90825 1.77 MB/sec execute 248 sec latency 9.280 ms oleg234-client.virtnet: 1 91736 1.79 MB/sec execute 249 sec latency 12.666 ms oleg234-client.virtnet: 1 92379 1.80 MB/sec execute 250 sec latency 15.010 ms oleg234-client.virtnet: 1 92725 1.80 MB/sec execute 251 sec latency 14.075 ms oleg234-client.virtnet: 1 93248 1.80 MB/sec execute 252 sec latency 15.018 ms oleg234-client.virtnet: 1 93994 1.81 MB/sec execute 253 sec latency 10.757 ms oleg234-client.virtnet: 1 94688 1.81 MB/sec execute 254 sec latency 15.005 ms oleg234-client.virtnet: 1 95236 1.82 MB/sec execute 255 sec latency 16.264 ms oleg234-client.virtnet: 1 96023 1.84 MB/sec execute 256 sec latency 15.507 ms oleg234-client.virtnet: 1 96489 1.83 MB/sec execute 257 sec latency 12.261 ms oleg234-client.virtnet: 1 97021 1.83 MB/sec execute 258 sec latency 20.734 ms oleg234-client.virtnet: 1 97783 1.84 MB/sec execute 259 sec latency 8.587 ms oleg234-client.virtnet: 1 98737 1.86 MB/sec execute 260 sec latency 8.133 ms oleg234-client.virtnet: 1 99659 1.88 MB/sec execute 261 sec latency 9.771 ms oleg234-client.virtnet: 1 100167 1.87 MB/sec execute 262 sec latency 11.891 ms oleg234-client.virtnet: 1 100827 1.88 MB/sec execute 263 sec latency 13.719 ms oleg234-client.virtnet: 1 101423 1.88 MB/sec execute 264 sec latency 9.967 ms oleg234-client.virtnet: 1 102239 1.89 MB/sec execute 265 sec latency 12.011 ms oleg234-client.virtnet: 1 103055 1.91 MB/sec execute 266 sec latency 12.739 ms oleg234-client.virtnet: 1 103560 1.91 MB/sec execute 267 sec latency 12.326 ms oleg234-client.virtnet: 1 104184 1.91 MB/sec execute 268 sec latency 10.394 ms oleg234-client.virtnet: 1 104948 1.91 MB/sec execute 269 sec latency 9.194 ms oleg234-client.virtnet: 1 105889 1.93 MB/sec execute 270 sec latency 16.621 ms oleg234-client.virtnet: 1 106731 1.95 MB/sec execute 271 sec latency 9.865 ms oleg234-client.virtnet: 1 107282 1.95 MB/sec execute 272 sec latency 10.601 ms oleg234-client.virtnet: 1 107999 1.95 MB/sec execute 273 sec latency 11.375 ms oleg234-client.virtnet: 1 108615 1.95 MB/sec execute 274 sec latency 10.717 ms oleg234-client.virtnet: 1 109346 1.97 MB/sec execute 275 sec latency 14.549 ms oleg234-client.virtnet: 1 110233 1.98 MB/sec execute 276 sec latency 10.590 ms oleg234-client.virtnet: 1 110490 1.98 MB/sec execute 277 sec latency 15.616 ms oleg234-client.virtnet: 1 111019 1.97 MB/sec execute 278 sec latency 17.161 ms oleg234-client.virtnet: 1 111776 1.98 MB/sec execute 279 sec latency 8.426 ms oleg234-client.virtnet: 1 112317 1.98 MB/sec execute 280 sec latency 11.109 ms oleg234-client.virtnet: 1 112876 1.99 MB/sec execute 281 sec latency 15.686 ms oleg234-client.virtnet: 1 113510 2.00 MB/sec execute 282 sec latency 15.208 ms oleg234-client.virtnet: 1 113891 2.00 MB/sec execute 283 sec latency 14.045 ms oleg234-client.virtnet: 1 114220 2.00 MB/sec execute 284 sec latency 17.693 ms oleg234-client.virtnet: 1 114632 2.00 MB/sec execute 285 sec latency 21.004 ms oleg234-client.virtnet: 1 115389 2.00 MB/sec execute 286 sec latency 10.357 ms oleg234-client.virtnet: 1 116316 2.01 MB/sec execute 287 sec latency 7.580 ms oleg234-client.virtnet: 1 117363 2.03 MB/sec execute 288 sec latency 8.089 ms oleg234-client.virtnet: 1 117963 2.03 MB/sec execute 289 sec latency 9.886 ms oleg234-client.virtnet: 1 118578 2.04 MB/sec execute 290 sec latency 12.233 ms oleg234-client.virtnet: 1 119277 2.03 MB/sec execute 291 sec latency 9.159 ms oleg234-client.virtnet: 1 120073 2.05 MB/sec execute 292 sec latency 11.437 ms oleg234-client.virtnet: 1 120859 2.06 MB/sec execute 293 sec latency 12.129 ms oleg234-client.virtnet: 1 121249 2.06 MB/sec execute 294 sec latency 13.937 ms oleg234-client.virtnet: 1 121576 2.05 MB/sec execute 295 sec latency 19.826 ms oleg234-client.virtnet: 1 122059 2.06 MB/sec execute 296 sec latency 15.193 ms oleg234-client.virtnet: 1 122453 2.06 MB/sec execute 297 sec latency 15.011 ms oleg234-client.virtnet: 1 123010 2.05 MB/sec execute 298 sec latency 13.468 ms oleg234-client.virtnet: 1 124008 2.07 MB/sec execute 299 sec latency 14.136 ms oleg234-client.virtnet: 1 cleanup 300 sec oleg234-client.virtnet: 0 cleanup 300 sec oleg234-client.virtnet: oleg234-client.virtnet: Operation Count AvgLat MaxLat oleg234-client.virtnet: ---------------------------------------- oleg234-client.virtnet: NTCreateX 19910 9.015 19999.815 oleg234-client.virtnet: Close 14595 1.336 7.796 oleg234-client.virtnet: Rename 847 32.128 19717.742 oleg234-client.virtnet: Unlink 4034 2.325 6.220 oleg234-client.virtnet: Qpathinfo 18019 1.654 15.548 oleg234-client.virtnet: Qfileinfo 3149 0.157 2.633 oleg234-client.virtnet: Qfsinfo 3327 0.268 25.333 oleg234-client.virtnet: Sfileinfo 1615 4.668 13.354 oleg234-client.virtnet: Find 6982 0.475 14.918 oleg234-client.virtnet: WriteX 9838 1.030 8.403 oleg234-client.virtnet: ReadX 31389 0.021 4.630 oleg234-client.virtnet: LockX 66 1.478 2.609 oleg234-client.virtnet: UnlockX 66 1.546 3.376 oleg234-client.virtnet: Flush 1389 7.507 626.507 oleg234-client.virtnet: oleg234-client.virtnet: Throughput 2.07196 MB/sec 1 clients 1 procs max_latency=19999.828 ms oleg234-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg234-client.virtnet at Sun Feb 25 01:33:46 EST 2024 with return code 0 oleg234-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg234-client.virtnet oleg234-client.virtnet: /mnt/lustre/d70b.replay-single/oleg234-client.virtnet /mnt/lustre/d70b.replay-single/oleg234-client.virtnet oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg234-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg234-client.virtnet: removed directory: 'clients/client0' oleg234-client.virtnet: removed directory: 'clients' oleg234-client.virtnet: removed 'client.txt' oleg234-client.virtnet: /mnt/lustre/d70b.replay-single/oleg234-client.virtnet oleg234-client.virtnet: dbench successfully finished