== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 05:32:07 (1740479527) Starting client oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre Started clients oleg422-client.virtnet: 192.168.204.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg422-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13583 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1285608 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26160 3556140 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5800 3594212 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 31960 7150352 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg422-server Stopping /mnt/lustre-mds1 (opts:) on oleg422-server 05:32:14 (1740479534) shut down facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds1 to oleg422-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 05:32:29 (1740479549) targets are mounted 05:32:29 (1740479549) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1285236 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26588 3566492 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5508 3594304 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 32096 7160796 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg422-server Stopping /mnt/lustre-mds2 (opts:) on oleg422-server 05:32:40 (1740479560) shut down facet: mds2 facet_host: oleg422-server facet_failover_host: oleg422-server oleg422-client.virtnet: looking for dbench program oleg422-client.virtnet: /usr/bin/dbench oleg422-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg422-client.virtnet oleg422-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg422-client.virtnet' oleg422-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg422-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg422-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg422-client.virtnet at Tue Feb 25 05:32:08 EST 2025 oleg422-client.virtnet: waiting for dbench pid 13608 oleg422-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg422-client.virtnet: oleg422-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg422-client.virtnet: failed to create barrier semaphore oleg422-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg422-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg422-client.virtnet: releasing clients oleg422-client.virtnet: 1 671 21.17 MB/sec warmup 1 sec latency 11.521 ms oleg422-client.virtnet: 1 1387 13.00 MB/sec warmup 2 sec latency 9.188 ms oleg422-client.virtnet: 1 1980 8.82 MB/sec warmup 3 sec latency 9.408 ms oleg422-client.virtnet: 1 2406 7.01 MB/sec warmup 4 sec latency 53.431 ms oleg422-client.virtnet: 1 3275 7.07 MB/sec warmup 5 sec latency 16.558 ms oleg422-client.virtnet: 1 3939 6.68 MB/sec warmup 6 sec latency 11.025 ms oleg422-client.virtnet: 1 3961 5.72 MB/sec warmup 7 sec latency 951.052 ms oleg422-client.virtnet: 1 3961 5.01 MB/sec warmup 8 sec latency 1951.278 ms oleg422-client.virtnet: 1 3961 4.45 MB/sec warmup 9 sec latency 2951.551 ms oleg422-client.virtnet: 1 3961 4.01 MB/sec warmup 10 sec latency 3951.815 ms oleg422-client.virtnet: 1 3961 3.64 MB/sec warmup 11 sec latency 4952.008 ms oleg422-client.virtnet: 1 3961 3.34 MB/sec warmup 12 sec latency 5952.321 ms oleg422-client.virtnet: 1 3961 3.08 MB/sec warmup 13 sec latency 6952.570 ms oleg422-client.virtnet: 1 3961 2.86 MB/sec warmup 14 sec latency 7952.834 ms oleg422-client.virtnet: 1 3961 2.67 MB/sec warmup 15 sec latency 8953.182 ms oleg422-client.virtnet: 1 3961 2.50 MB/sec warmup 16 sec latency 9953.416 ms oleg422-client.virtnet: 1 3961 2.36 MB/sec warmup 17 sec latency 10953.590 ms oleg422-client.virtnet: 1 3961 2.23 MB/sec warmup 18 sec latency 11953.686 ms oleg422-client.virtnet: 1 3961 2.11 MB/sec warmup 19 sec latency 12953.805 ms oleg422-client.virtnet: 1 3961 2.00 MB/sec warmup 20 sec latency 13954.034 ms oleg422-client.virtnet: 1 3961 1.91 MB/sec warmup 21 sec latency 14954.197 ms oleg422-client.virtnet: 1 3961 1.82 MB/sec warmup 22 sec latency 15954.364 ms oleg422-client.virtnet: 1 3961 1.74 MB/sec warmup 23 sec latency 16954.547 ms oleg422-client.virtnet: 1 3961 1.67 MB/sec warmup 24 sec latency 17954.712 ms oleg422-client.virtnet: 1 3970 1.60 MB/sec warmup 25 sec latency 18926.111 ms oleg422-client.virtnet: 1 4520 1.57 MB/sec warmup 26 sec latency 11.004 ms oleg422-client.virtnet: 1 5237 1.68 MB/sec warmup 27 sec latency 8.644 ms oleg422-client.virtnet: 1 5976 1.69 MB/sec warmup 28 sec latency 8.867 ms oleg422-client.virtnet: 1 7174 1.99 MB/sec warmup 29 sec latency 8.260 ms oleg422-client.virtnet: 1 7787 1.98 MB/sec warmup 30 sec latency 10.754 ms oleg422-client.virtnet: 1 8515 2.07 MB/sec warmup 31 sec latency 10.539 ms oleg422-client.virtnet: 1 9131 2.02 MB/sec warmup 32 sec latency 12.051 ms oleg422-client.virtnet: 1 10088 2.17 MB/sec warmup 33 sec latency 7.985 ms Failover mds2 to oleg422-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0001 05:32:54 (1740479574) targets are mounted 05:32:54 (1740479574) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37736 3566352 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12088 3593940 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49824 7160292 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg422-server Stopping /mnt/lustre-mds1 (opts:) on oleg422-server 05:33:06 (1740479586) shut down facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds1 to oleg422-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 05:33:19 (1740479599) targets are mounted 05:33:20 (1740479600) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg422-client.virtnet: 1 11017 2.30 MB/sec warmup 34 sec latency 8.178 ms oleg422-client.virtnet: 1 11625 2.26 MB/sec warmup 35 sec latency 13.420 ms oleg422-client.virtnet: 1 12346 2.32 MB/sec warmup 36 sec latency 8.636 ms oleg422-client.virtnet: 1 13077 2.31 MB/sec warmup 37 sec latency 8.490 ms oleg422-client.virtnet: 1 14221 2.52 MB/sec warmup 38 sec latency 11.094 ms oleg422-client.virtnet: 1 14680 2.50 MB/sec warmup 39 sec latency 11.712 ms oleg422-client.virtnet: 1 15305 2.49 MB/sec warmup 40 sec latency 10.654 ms oleg422-client.virtnet: 1 15986 2.51 MB/sec warmup 41 sec latency 10.111 ms oleg422-client.virtnet: 1 16683 2.50 MB/sec warmup 42 sec latency 8.182 ms oleg422-client.virtnet: 1 17853 2.68 MB/sec warmup 43 sec latency 8.749 ms oleg422-client.virtnet: 1 18465 2.65 MB/sec warmup 44 sec latency 9.405 ms oleg422-client.virtnet: 1 19198 2.70 MB/sec warmup 45 sec latency 8.275 ms oleg422-client.virtnet: 1 19531 2.65 MB/sec warmup 46 sec latency 422.527 ms oleg422-client.virtnet: 1 19531 2.59 MB/sec warmup 47 sec latency 1422.695 ms oleg422-client.virtnet: 1 19531 2.54 MB/sec warmup 48 sec latency 2422.848 ms oleg422-client.virtnet: 1 19531 2.49 MB/sec warmup 49 sec latency 3423.072 ms oleg422-client.virtnet: 1 19531 2.44 MB/sec warmup 50 sec latency 4423.281 ms oleg422-client.virtnet: 1 19991 2.40 MB/sec warmup 51 sec latency 4554.006 ms oleg422-client.virtnet: 1 21041 2.52 MB/sec warmup 52 sec latency 9.998 ms oleg422-client.virtnet: 1 21777 2.56 MB/sec warmup 53 sec latency 7.524 ms oleg422-client.virtnet: 1 22384 2.55 MB/sec warmup 54 sec latency 11.387 ms oleg422-client.virtnet: 1 23061 2.56 MB/sec warmup 55 sec latency 10.243 ms oleg422-client.virtnet: 1 23752 2.55 MB/sec warmup 56 sec latency 57.701 ms oleg422-client.virtnet: 1 24900 2.69 MB/sec warmup 57 sec latency 9.032 ms oleg422-client.virtnet: 1 25140 2.66 MB/sec warmup 58 sec latency 612.333 ms oleg422-client.virtnet: 1 25140 2.62 MB/sec warmup 59 sec latency 1612.505 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 1 sec latency 3612.851 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 2 sec latency 4613.093 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 3 sec latency 5613.302 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 4 sec latency 6613.484 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 5 sec latency 7613.867 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 6 sec latency 8614.179 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 7 sec latency 9614.373 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 8 sec latency 10614.509 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 9 sec latency 11614.655 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 10 sec latency 12614.856 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 11 sec latency 13615.042 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 12 sec latency 14615.241 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 13 sec latency 15615.372 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 14 sec latency 16615.547 ms oleg422-client.virtnet: 1 25140 0.00 MB/sec execute 15 sec latency 17615.703 ms oleg422-client.virtnet: 1 25186 0.00 MB/sec execute 16 sec latency 18501.606 ms oleg422-client.virtnet: 1 25705 0.05 MB/sec execute 17 sec latency 9.058 ms oleg422-client.virtnet: 1 26387 0.30 MB/sec execute 18 sec latency 10.595 ms oleg422-client.virtnet: 1 27038 0.32 MB/sec execute 19 sec lUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1285276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3568808 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12064 3593772 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162580 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg422-server Stopping /mnt/lustre-mds2 (opts:) on oleg422-server 05:33:31 (1740479611) shut down facet: mds2 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds2 to oleg422-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0001 05:33:45 (1740479625) targets are mounted 05:33:45 (1740479625) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37564 3568868 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12072 3591896 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160764 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg422-server Stopping /mnt/lustre-mds1 (opts:) on oleg422-server 05:33:56 (1740479636) shut down facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds1 to oleg422-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 05:34:10 (1740479650) targets are mounted 05:34:10 (1740479650) facet_failover done atency 11.856 ms oleg422-client.virtnet: 1 28081 0.68 MB/sec execute 20 sec latency 8.028 ms oleg422-client.virtnet: 1 28838 0.92 MB/sec execute 21 sec latency 7.995 ms oleg422-client.virtnet: 1 29409 0.93 MB/sec execute 22 sec latency 12.073 ms oleg422-client.virtnet: 1 30088 1.07 MB/sec execute 23 sec latency 10.196 ms oleg422-client.virtnet: 1 30813 1.10 MB/sec execute 24 sec latency 9.857 ms oleg422-client.virtnet: 1 31907 1.47 MB/sec execute 25 sec latency 9.284 ms oleg422-client.virtnet: 1 32449 1.48 MB/sec execute 26 sec latency 8.306 ms oleg422-client.virtnet: 1 32954 1.46 MB/sec execute 27 sec latency 13.887 ms oleg422-client.virtnet: 1 33623 1.56 MB/sec execute 28 sec latency 10.462 ms oleg422-client.virtnet: 1 34339 1.57 MB/sec execute 29 sec latency 8.214 ms oleg422-client.virtnet: 1 35492 1.87 MB/sec execute 30 sec latency 9.205 ms oleg422-client.virtnet: 1 36063 1.86 MB/sec execute 31 sec latency 7.138 ms oleg422-client.virtnet: 1 36722 1.87 MB/sec execute 32 sec latency 9.530 ms oleg422-client.virtnet: 1 37416 1.91 MB/sec execute 33 sec latency 9.090 ms oleg422-client.virtnet: 1 38278 2.02 MB/sec execute 34 sec latency 8.842 ms oleg422-client.virtnet: 1 39307 2.18 MB/sec execute 35 sec latency 9.408 ms oleg422-client.virtnet: 1 39873 2.15 MB/sec execute 36 sec latency 16.049 ms oleg422-client.virtnet: 1 40291 2.13 MB/sec execute 37 sec latency 341.591 ms oleg422-client.virtnet: 1 40291 2.08 MB/sec execute 38 sec latency 1341.795 ms oleg422-client.virtnet: 1 40291 2.02 MB/sec execute 39 sec latency 2341.957 ms oleg422-client.virtnet: 1 40291 1.97 MB/sec execute 40 sec latency 3342.133 ms oleg422-client.virtnet: 1 40291 1.92 MB/sec execute 41 sec latency 4342.254 ms oleg422-client.virtnet: 1 40949 1.96 MB/sec execute 42 sec latency 4347.666 ms oleg422-client.virtnet: 1 41825 2.04 MB/sec execute 43 sec latency 8.434 ms oleg422-client.virtnet: 1 42878 2.17 MB/sec execute 44 sec latency 7.798 ms oleg422-client.virtnet: 1 43459 2.14 MB/sec execute 45 sec latency 12.299 ms oleg422-client.virtnet: 1 44085 2.20 MB/sec execute 46 sec latency 107.859 ms oleg422-client.virtnet: 1 44698 2.16 MB/sec execute 47 sec latency 163.050 ms oleg422-client.virtnet: 1 45620 2.26 MB/sec execute 48 sec latency 8.445 ms oleg422-client.virtnet: 1 45871 2.25 MB/sec execute 49 sec latency 881.654 ms oleg422-client.virtnet: 1 45871 2.21 MB/sec execute 50 sec latency 1881.899 ms oleg422-client.virtnet: 1 45871 2.16 MB/sec execute 51 sec latency 2882.133 ms oleg422-client.virtnet: 1 45871 2.12 MB/sec execute 52 sec latency 3882.357 ms oleg422-client.virtnet: 1 45871 2.08 MB/sec execute 53 sec latency 4882.534 ms oleg422-client.virtnet: 1 45871 2.04 MB/sec execute 54 sec latency 5882.703 ms oleg422-client.virtnet: 1 45871 2.01 MB/sec execute 55 sec latency 6882.870 ms oleg422-client.virtnet: 1 45871 1.97 MB/sec execute 56 sec latency 7883.105 ms oleg422-client.virtnet: 1 45871 1.93 MB/sec execute 57 sec latency 8883.334 ms oleg422-client.virtnet: 1 45871 1.90 MB/sec execute 58 sec latency 9883.527 ms oleg422-client.virtnet: 1 45871 1.87 MB/sec execute 59 sec latency 10883.652 ms oleg422-client.virtnet: 1 45871 1.84 MB/sec execute 60 sec latency 11883.769 ms oleg422-client.virtnet: 1 45871 1.81 MB/sec execute 61 sec latency 12883.949 ms oleg422-client.virtnet: 1 45871 1.78 MB/sec execute 62 sec latency 13884.085 ms oleg422-client.virtnet: 1 45871 1.75 MB/sec execute 63 sec latency 14884.235 ms oleg422-client.virtnet: 1 45871 oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38012 3568516 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11900 3591956 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49912 7160472 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg422-server Stopping /mnt/lustre-mds2 (opts:) on oleg422-server 05:34:22 (1740479662) shut down facet: mds2 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds2 to oleg422-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0001 05:34:35 (1740479675) targets are mounted 05:34:35 (1740479675) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37652 3568608 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11908 3593960 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49560 7162568 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg422-server Stopping /mnt/lustre-mds1 (opts:) on oleg422-server 05:34:47 (1740479687) shut down facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server 1.72 MB/sec execute 64 sec latency 15884.452 ms oleg422-client.virtnet: 1 45871 1.70 MB/sec execute 65 sec latency 16884.638 ms oleg422-client.virtnet: 1 45871 1.67 MB/sec execute 66 sec latency 17884.850 ms oleg422-client.virtnet: 1 46283 1.70 MB/sec execute 67 sec latency 18350.436 ms oleg422-client.virtnet: 1 46851 1.69 MB/sec execute 68 sec latency 8.646 ms oleg422-client.virtnet: 1 47550 1.74 MB/sec execute 69 sec latency 8.411 ms oleg422-client.virtnet: 1 48177 1.72 MB/sec execute 70 sec latency 9.614 ms oleg422-client.virtnet: 1 49109 1.80 MB/sec execute 71 sec latency 8.584 ms oleg422-client.virtnet: 1 50015 1.86 MB/sec execute 72 sec latency 7.887 ms oleg422-client.virtnet: 1 50622 1.85 MB/sec execute 73 sec latency 10.502 ms oleg422-client.virtnet: 1 51343 1.88 MB/sec execute 74 sec latency 8.174 ms oleg422-client.virtnet: 1 52078 1.88 MB/sec execute 75 sec latency 9.393 ms oleg422-client.virtnet: 1 53242 1.99 MB/sec execute 76 sec latency 8.530 ms oleg422-client.virtnet: 1 53840 1.99 MB/sec execute 77 sec latency 7.888 ms oleg422-client.virtnet: 1 54503 1.99 MB/sec execute 78 sec latency 8.899 ms oleg422-client.virtnet: 1 55242 2.01 MB/sec execute 79 sec latency 9.025 ms oleg422-client.virtnet: 1 56167 2.07 MB/sec execute 80 sec latency 8.822 ms oleg422-client.virtnet: 1 57108 2.12 MB/sec execute 81 sec latency 8.187 ms oleg422-client.virtnet: 1 57709 2.11 MB/sec execute 82 sec latency 10.402 ms oleg422-client.virtnet: 1 58418 2.14 MB/sec execute 83 sec latency 8.922 ms oleg422-client.virtnet: 1 59142 2.13 MB/sec execute 84 sec latency 8.957 ms oleg422-client.virtnet: 1 60307 2.23 MB/sec execute 85 sec latency 8.512 ms oleg422-client.virtnet: 1 60884 2.23 MB/sec execute 86 sec latency 7.972 ms oleg422-client.virtnet: 1 61331 2.21 MB/sec execute 87 sec latency 270.445 ms oleg422-client.virtnet: 1 61331 2.19 MB/sec execute 88 sec latency 1270.629 ms oleg422-client.virtnet: 1 61331 2.16 MB/sec execute 89 sec latency 2270.799 ms oleg422-client.virtnet: 1 61331 2.14 MB/sec execute 90 sec latency 3271.045 ms oleg422-client.virtnet: 1 61331 2.11 MB/sec execute 91 sec latency 4271.234 ms oleg422-client.virtnet: 1 61741 2.14 MB/sec execute 92 sec latency 4762.814 ms oleg422-client.virtnet: 1 62336 2.12 MB/sec execute 93 sec latency 9.893 ms oleg422-client.virtnet: 1 63212 2.16 MB/sec execute 94 sec latency 9.043 ms oleg422-client.virtnet: 1 64129 2.21 MB/sec execute 95 sec latency 9.633 ms oleg422-client.virtnet: 1 64667 2.20 MB/sec execute 96 sec latency 9.691 ms oleg422-client.virtnet: 1 65023 2.19 MB/sec execute 97 sec latency 414.866 ms oleg422-client.virtnet: 1 65704 2.20 MB/sec execute 98 sec latency 9.128 ms oleg422-client.virtnet: 1 66258 2.20 MB/sec execute 99 sec latency 274.118 ms oleg422-client.virtnet: 1 66258 2.18 MB/sec execute 100 sec latency 1274.390 ms oleg422-client.virtnet: 1 66258 2.15 MB/sec execute 101 sec latency 2274.619 ms oleg422-client.virtnet: 1 66258 2.13 MB/sec execute 102 sec latency 3274.797 ms oleg422-client.virtnet: 1 66258 2.11 MB/sec execute 103 sec latency 4274.961 ms oleg422-client.virtnet: 1 66258 2.09 MB/sec execute 104 sec latency 5275.126 ms oleg422-client.virtnet: 1 66258 2.07 MB/sec execute 105 sec latency 6275.322 ms oleg422-client.virtnet: 1 66258 2.05 MB/sec execute 106 sec latency 7275.476 ms oleg422-client.virtnet: 1 66258 2.03 MB/sec execute 107 sec latency 8275.636 ms oleg422-client.virtnet: 1 66258 2.01 MB/sec execute 108 sec latency 9275.802 ms oleg422-client.virtnet:Failover mds1 to oleg422-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:01 (1740479701) targets are mounted 05:35:01 (1740479701) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1285276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37776 3568668 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12384 3593832 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50160 7162500 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg422-server Stopping /mnt/lustre-mds2 (opts:) on oleg422-server 05:35:12 (1740479712) shut down facet: mds2 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds2 to oleg422-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0001 05:35:27 (1740479727) targets are mounted 05:35:27 (1740479727) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3569072 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12060 3593932 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49496 7163004 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg422-server Stopping /mnt/lustre-mds1 (opts:) on oleg422-server 05:35:38 (1740479738) shut down facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server 1 66258 2.00 MB/sec execute 109 sec latency 10275.963 ms oleg422-client.virtnet: 1 66258 1.98 MB/sec execute 110 sec latency 11276.164 ms oleg422-client.virtnet: 1 66258 1.96 MB/sec execute 111 sec latency 12276.331 ms oleg422-client.virtnet: 1 66258 1.94 MB/sec execute 112 sec latency 13276.467 ms oleg422-client.virtnet: 1 66258 1.93 MB/sec execute 113 sec latency 14276.617 ms oleg422-client.virtnet: 1 66258 1.91 MB/sec execute 114 sec latency 15276.802 ms oleg422-client.virtnet: 1 66258 1.89 MB/sec execute 115 sec latency 16277.004 ms oleg422-client.virtnet: 1 66258 1.88 MB/sec execute 116 sec latency 17277.172 ms oleg422-client.virtnet: 1 66258 1.86 MB/sec execute 117 sec latency 18277.276 ms oleg422-client.virtnet: 1 67362 1.93 MB/sec execute 118 sec latency 18284.193 ms oleg422-client.virtnet: 1 67898 1.93 MB/sec execute 119 sec latency 9.042 ms oleg422-client.virtnet: 1 68546 1.93 MB/sec execute 120 sec latency 9.667 ms oleg422-client.virtnet: 1 69221 1.94 MB/sec execute 121 sec latency 9.389 ms oleg422-client.virtnet: 1 69977 1.95 MB/sec execute 122 sec latency 8.489 ms oleg422-client.virtnet: 1 71107 2.01 MB/sec execute 123 sec latency 8.631 ms oleg422-client.virtnet: 1 71692 2.01 MB/sec execute 124 sec latency 11.110 ms oleg422-client.virtnet: 1 72404 2.03 MB/sec execute 125 sec latency 8.513 ms oleg422-client.virtnet: 1 73070 2.02 MB/sec execute 126 sec latency 8.507 ms oleg422-client.virtnet: 1 74005 2.06 MB/sec execute 127 sec latency 8.645 ms oleg422-client.virtnet: 1 74869 2.09 MB/sec execute 128 sec latency 9.018 ms oleg422-client.virtnet: 1 75476 2.08 MB/sec execute 129 sec latency 8.681 ms oleg422-client.virtnet: 1 76204 2.10 MB/sec execute 130 sec latency 8.874 ms oleg422-client.virtnet: 1 76939 2.10 MB/sec execute 131 sec latency 8.427 ms oleg422-client.virtnet: 1 78109 2.16 MB/sec execute 132 sec latency 9.075 ms oleg422-client.virtnet: 1 78641 2.16 MB/sec execute 133 sec latency 10.015 ms oleg422-client.virtnet: 1 79120 2.16 MB/sec execute 134 sec latency 14.812 ms oleg422-client.virtnet: 1 79625 2.17 MB/sec execute 135 sec latency 14.912 ms oleg422-client.virtnet: 1 80159 2.15 MB/sec execute 136 sec latency 9.947 ms oleg422-client.virtnet: 1 80537 2.15 MB/sec execute 137 sec latency 16.488 ms oleg422-client.virtnet: 1 81358 2.18 MB/sec execute 138 sec latency 16.944 ms oleg422-client.virtnet: 1 81909 2.20 MB/sec execute 139 sec latency 14.839 ms oleg422-client.virtnet: 1 81979 2.19 MB/sec execute 140 sec latency 729.444 ms oleg422-client.virtnet: 1 81979 2.17 MB/sec execute 141 sec latency 1729.616 ms oleg422-client.virtnet: 1 81979 2.16 MB/sec execute 142 sec latency 2729.828 ms oleg422-client.virtnet: 1 82073 2.14 MB/sec execute 143 sec latency 3503.749 ms oleg422-client.virtnet: 1 82631 2.13 MB/sec execute 144 sec latency 11.843 ms oleg422-client.virtnet: 1 83201 2.15 MB/sec execute 145 sec latency 16.697 ms oleg422-client.virtnet: 1 83508 2.13 MB/sec execute 146 sec latency 19.436 ms oleg422-client.virtnet: 1 84382 2.16 MB/sec execute 147 sec latency 8.599 ms oleg422-client.virtnet: 1 85381 2.20 MB/sec execute 148 sec latency 8.963 ms oleg422-client.virtnet: 1 85923 2.19 MB/sec execute 149 sec latency 11.638 ms oleg422-client.virtnet: 1 86570 2.20 MB/sec execute 150 sec latency 10.148 ms oleg422-client.virtnet: 1 86595 2.19 MB/sec execute 151 sec latency 932.610 ms oleg422-client.virtnet: 1 86595 2.17 MB/sec execute 152 sec latency 1932.775 ms oleg422-client.virtnet: 1 86595 2.16 MB/sec execute 153 sec latency 2933.042 mFailover mds1 to oleg422-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:52 (1740479752) targets are mounted 05:35:52 (1740479752) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37516 3568724 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12184 3591768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49700 7160492 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg422-server Stopping /mnt/lustre-mds2 (opts:) on oleg422-server 05:36:04 (1740479764) shut down facet: mds2 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds2 to oleg422-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0001 05:36:17 (1740479777) targets are mounted 05:36:17 (1740479777) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37564 3568928 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12152 3591872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49716 7160800 1% /mnt/lustre s oleg422-client.virtnet: 1 86595 2.15 MB/sec execute 154 sec latency 3933.190 ms oleg422-client.virtnet: 1 86595 2.13 MB/sec execute 155 sec latency 4933.337 ms oleg422-client.virtnet: 1 86595 2.12 MB/sec execute 156 sec latency 5933.525 ms oleg422-client.virtnet: 1 86595 2.10 MB/sec execute 157 sec latency 6933.694 ms oleg422-client.virtnet: 1 86595 2.09 MB/sec execute 158 sec latency 7933.977 ms oleg422-client.virtnet: 1 86595 2.08 MB/sec execute 159 sec latency 8934.164 ms oleg422-client.virtnet: 1 86595 2.07 MB/sec execute 160 sec latency 9934.363 ms oleg422-client.virtnet: 1 86595 2.05 MB/sec execute 161 sec latency 10934.520 ms oleg422-client.virtnet: 1 86595 2.04 MB/sec execute 162 sec latency 11934.663 ms oleg422-client.virtnet: 1 86595 2.03 MB/sec execute 163 sec latency 12934.811 ms oleg422-client.virtnet: 1 86595 2.01 MB/sec execute 164 sec latency 13934.940 ms oleg422-client.virtnet: 1 86595 2.00 MB/sec execute 165 sec latency 14935.094 ms oleg422-client.virtnet: 1 86595 1.99 MB/sec execute 166 sec latency 15935.263 ms oleg422-client.virtnet: 1 86595 1.98 MB/sec execute 167 sec latency 16935.408 ms oleg422-client.virtnet: 1 86595 1.97 MB/sec execute 168 sec latency 17935.610 ms oleg422-client.virtnet: 1 86829 1.96 MB/sec execute 169 sec latency 18528.607 ms oleg422-client.virtnet: 1 87531 1.95 MB/sec execute 170 sec latency 9.093 ms oleg422-client.virtnet: 1 88667 2.00 MB/sec execute 171 sec latency 9.791 ms oleg422-client.virtnet: 1 89234 2.00 MB/sec execute 172 sec latency 7.690 ms oleg422-client.virtnet: 1 89870 2.00 MB/sec execute 173 sec latency 9.849 ms oleg422-client.virtnet: 1 90494 2.01 MB/sec execute 174 sec latency 13.693 ms oleg422-client.virtnet: 1 91154 2.01 MB/sec execute 175 sec latency 10.373 ms oleg422-client.virtnet: 1 92330 2.06 MB/sec execute 176 sec latency 9.427 ms oleg422-client.virtnet: 1 92727 2.05 MB/sec execute 177 sec latency 13.027 ms oleg422-client.virtnet: 1 93248 2.05 MB/sec execute 178 sec latency 12.974 ms oleg422-client.virtnet: 1 93663 2.06 MB/sec execute 179 sec latency 17.310 ms oleg422-client.virtnet: 1 94262 2.05 MB/sec execute 180 sec latency 14.175 ms oleg422-client.virtnet: 1 95171 2.07 MB/sec execute 181 sec latency 12.611 ms oleg422-client.virtnet: 1 96102 2.10 MB/sec execute 182 sec latency 10.388 ms oleg422-client.virtnet: 1 96712 2.10 MB/sec execute 183 sec latency 10.025 ms oleg422-client.virtnet: 1 97410 2.11 MB/sec execute 184 sec latency 9.148 ms oleg422-client.virtnet: 1 98119 2.11 MB/sec execute 185 sec latency 10.692 ms oleg422-client.virtnet: 1 99189 2.14 MB/sec execute 186 sec latency 8.773 ms oleg422-client.virtnet: 1 99846 2.15 MB/sec execute 187 sec latency 8.156 ms oleg422-client.virtnet: 1 100504 2.15 MB/sec execute 188 sec latency 9.157 ms oleg422-client.virtnet: 1 101100 2.15 MB/sec execute 189 sec latency 124.185 ms oleg422-client.virtnet: 1 101100 2.14 MB/sec execute 190 sec latency 1124.325 ms oleg422-client.virtnet: 1 101100 2.13 MB/sec execute 191 sec latency 2124.432 ms oleg422-client.virtnet: 1 101100 2.12 MB/sec execute 192 sec latency 3124.612 ms oleg422-client.virtnet: 1 101100 2.11 MB/sec execute 193 sec latency 4124.865 ms oleg422-client.virtnet: 1 101251 2.10 MB/sec execute 194 sec latency 4778.901 ms oleg422-client.virtnet: 1 102074 2.12 MB/sec execute 195 sec latency 8.987 ms oleg422-client.virtnet: 1 103150 2.15 MB/sec execute 196 sec latency 8.609 ms oleg422-client.virtnet: 1 103727 2.14 MB/sec execute 197 sec latency 9.852 ms oleg422-client.virtnet: 1 104444 2.15 MBtest_70b fail mds1 11 times Failing mds1 on oleg422-server Stopping /mnt/lustre-mds1 (opts:) on oleg422-server 05:36:29 (1740479789) shut down facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds1 to oleg422-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 05:36:43 (1740479803) targets are mounted 05:36:43 (1740479803) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1972 1285716 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37432 3569308 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12060 3594932 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49492 7164240 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg422-server Stopping /mnt/lustre-mds2 (opts:) on oleg422-server 05:36:55 (1740479815) shut down facet: mds2 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds2 to oleg422-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0001 05:37:09 (1740479829) targets are mounted 05:37:09 (1740479829) facet_failover done /sec execute 198 sec latency 8.084 ms oleg422-client.virtnet: 1 104948 2.14 MB/sec execute 199 sec latency 12.429 ms oleg422-client.virtnet: 1 105787 2.16 MB/sec execute 200 sec latency 10.034 ms oleg422-client.virtnet: 1 106604 2.18 MB/sec execute 201 sec latency 189.253 ms oleg422-client.virtnet: 1 106604 2.17 MB/sec execute 202 sec latency 1189.407 ms oleg422-client.virtnet: 1 106604 2.16 MB/sec execute 203 sec latency 2189.573 ms oleg422-client.virtnet: 1 106604 2.15 MB/sec execute 204 sec latency 3189.752 ms oleg422-client.virtnet: 1 106604 2.14 MB/sec execute 205 sec latency 4189.947 ms oleg422-client.virtnet: 1 106604 2.13 MB/sec execute 206 sec latency 5190.138 ms oleg422-client.virtnet: 1 106604 2.12 MB/sec execute 207 sec latency 6190.345 ms oleg422-client.virtnet: 1 106604 2.11 MB/sec execute 208 sec latency 7190.530 ms oleg422-client.virtnet: 1 106604 2.10 MB/sec execute 209 sec latency 8190.750 ms oleg422-client.virtnet: 1 106604 2.09 MB/sec execute 210 sec latency 9190.922 ms oleg422-client.virtnet: 1 106604 2.08 MB/sec execute 211 sec latency 10191.073 ms oleg422-client.virtnet: 1 106604 2.07 MB/sec execute 212 sec latency 11191.191 ms oleg422-client.virtnet: 1 106604 2.06 MB/sec execute 213 sec latency 12191.383 ms oleg422-client.virtnet: 1 106604 2.05 MB/sec execute 214 sec latency 13191.596 ms oleg422-client.virtnet: 1 106604 2.04 MB/sec execute 215 sec latency 14191.720 ms oleg422-client.virtnet: 1 106604 2.03 MB/sec execute 216 sec latency 15191.938 ms oleg422-client.virtnet: 1 106604 2.02 MB/sec execute 217 sec latency 16192.191 ms oleg422-client.virtnet: 1 106604 2.01 MB/sec execute 218 sec latency 17192.474 ms oleg422-client.virtnet: 1 106604 2.00 MB/sec execute 219 sec latency 18192.736 ms oleg422-client.virtnet: 1 106797 2.00 MB/sec execute 220 sec latency 18635.176 ms oleg422-client.virtnet: 1 107151 1.99 MB/sec execute 221 sec latency 19.034 ms oleg422-client.virtnet: 1 107427 1.99 MB/sec execute 222 sec latency 17.711 ms oleg422-client.virtnet: 1 107835 2.00 MB/sec execute 223 sec latency 16.929 ms oleg422-client.virtnet: 1 108327 1.99 MB/sec execute 224 sec latency 15.383 ms oleg422-client.virtnet: 1 108870 1.99 MB/sec execute 225 sec latency 12.371 ms oleg422-client.virtnet: 1 109944 2.02 MB/sec execute 226 sec latency 11.189 ms oleg422-client.virtnet: 1 110360 2.02 MB/sec execute 227 sec latency 11.593 ms oleg422-client.virtnet: 1 110889 2.02 MB/sec execute 228 sec latency 9.761 ms oleg422-client.virtnet: 1 111570 2.03 MB/sec execute 229 sec latency 9.618 ms oleg422-client.virtnet: 1 112261 2.02 MB/sec execute 230 sec latency 9.551 ms oleg422-client.virtnet: 1 113359 2.05 MB/sec execute 231 sec latency 8.733 ms oleg422-client.virtnet: 1 113983 2.06 MB/sec execute 232 sec latency 11.137 ms oleg422-client.virtnet: 1 114523 2.06 MB/sec execute 233 sec latency 12.096 ms oleg422-client.virtnet: 1 115247 2.07 MB/sec execute 234 sec latency 9.242 ms oleg422-client.virtnet: 1 115985 2.07 MB/sec execute 235 sec latency 8.601 ms oleg422-client.virtnet: 1 117177 2.10 MB/sec execute 236 sec latency 8.828 ms oleg422-client.virtnet: 1 117796 2.10 MB/sec execute 237 sec latency 8.810 ms oleg422-client.virtnet: 1 118496 2.11 MB/sec execute 238 sec latency 8.798 ms oleg422-client.virtnet: 1 119050 2.10 MB/sec execute 239 sec latency 10.203 ms oleg422-client.virtnet: 1 119969 2.12 MB/sec execute 240 sec latency 8.804 ms oleg422-client.virtnet: 1 120761 2.14 MB/sec execute 241 sec latency 12.063 ms oleg422-client.virtnet: 1 121008 2.13 MB/sec execute 242 sec latency 160.890 ms oleg422-clieoleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec nt.virtnet: 1 121008 2.13 MB/sec execute 243 sec latency 1161.103 ms oleg422-client.virtnet: 1 121008 2.12 MB/sec execute 244 sec latency 2161.289 ms oleg422-client.virtnet: 1 121008 2.11 MB/sec execute 245 sec latency 3161.563 ms oleg422-client.virtnet: 1 121461 2.10 MB/sec execute 246 sec latency 3332.652 ms oleg422-client.virtnet: 1 122192 2.11 MB/sec execute 247 sec latency 10.997 ms oleg422-client.virtnet: 1 122867 2.11 MB/sec execute 248 sec latency 8.908 ms oleg422-client.virtnet: 1 123983 2.13 MB/sec execute 249 sec latency 8.642 ms oleg422-client.virtnet: 1 124524 2.14 MB/sec execute 250 sec latency 15.314 ms oleg422-client.virtnet: 1 124932 2.14 MB/sec execute 251 sec latency 14.695 ms oleg422-client.virtnet: 1 125507 2.14 MB/sec execute 252 sec latency 16.870 ms oleg422-client.virtnet: 1 125868 2.14 MB/sec execute 253 sec latency 15.233 ms oleg422-client.virtnet: 1 126340 2.13 MB/sec execute 254 sec latency 15.562 ms oleg422-client.virtnet: 1 126819 2.13 MB/sec execute 255 sec latency 16.323 ms oleg422-client.virtnet: 1 127766 2.16 MB/sec execute 256 sec latency 12.816 ms oleg422-client.virtnet: 1 128097 2.16 MB/sec execute 257 sec latency 15.848 ms oleg422-client.virtnet: 1 128436 2.15 MB/sec execute 258 sec latency 14.922 ms oleg422-client.virtnet: 1 129137 2.16 MB/sec execute 259 sec latency 9.422 ms oleg422-client.virtnet: 1 129793 2.16 MB/sec execute 260 sec latency 8.716 ms oleg422-client.virtnet: 1 130710 2.17 MB/sec execute 261 sec latency 8.400 ms oleg422-client.virtnet: 1 131602 2.19 MB/sec execute 262 sec latency 8.672 ms oleg422-client.virtnet: 1 132190 2.19 MB/sec execute 263 sec latency 12.566 ms oleg422-client.virtnet: 1 132530 2.18 MB/sec execute 264 sec latency 17.143 ms oleg422-client.virtnet: 1 132988 2.19 MB/sec execute 265 sec latency 17.152 ms oleg422-client.virtnet: 1 133472 2.18 MB/sec execute 266 sec latency 10.927 ms oleg422-client.virtnet: 1 133885 2.18 MB/sec execute 267 sec latency 17.411 ms oleg422-client.virtnet: 1 134663 2.20 MB/sec execute 268 sec latency 16.949 ms oleg422-client.virtnet: 1 135067 2.20 MB/sec execute 269 sec latency 14.853 ms oleg422-client.virtnet: 1 135305 2.20 MB/sec execute 270 sec latency 15.589 ms oleg422-client.virtnet: 1 135626 2.19 MB/sec execute 271 sec latency 20.118 ms oleg422-client.virtnet: 1 135954 2.19 MB/sec execute 272 sec latency 17.237 ms oleg422-client.virtnet: 1 136324 2.19 MB/sec execute 273 sec latency 17.173 ms oleg422-client.virtnet: 1 136615 2.18 MB/sec execute 274 sec latency 18.932 ms oleg422-client.virtnet: 1 137051 2.18 MB/sec execute 275 sec latency 11.388 ms oleg422-client.virtnet: 1 137576 2.19 MB/sec execute 276 sec latency 16.999 ms oleg422-client.virtnet: 1 138451 2.20 MB/sec execute 277 sec latency 12.976 ms oleg422-client.virtnet: 1 138943 2.20 MB/sec execute 278 sec latency 9.716 ms oleg422-client.virtnet: 1 139605 2.20 MB/sec execute 279 sec latency 10.346 ms oleg422-client.virtnet: 1 140351 2.21 MB/sec execute 280 sec latency 8.166 ms oleg422-client.virtnet: 1 141278 2.22 MB/sec execute 281 sec latency 8.147 ms oleg422-client.virtnet: 1 142221 2.24 MB/sec execute 282 sec latency 8.045 ms oleg422-client.virtnet: 1 142825 2.23 MB/sec execute 283 sec latency 8.938 ms oleg422-client.virtnet: 1 143549 2.24 MB/sec execute 284 sec latency 8.653 ms oleg422-client.virtnet: 1 144289 2.24 MB/sec execute 285 sec latency 8.867 ms oleg422-client.virtnet: 1 145447 2.27 MB/sec execute 286 sec latency 8.381 ms oleg422-client.virtnet: 1 145953 2.27 MB/sec execute 287 sec latency 10.749 ms oleg422-client.virtnet: 1 146603 2.27 MB/sec execute 288 sec latency 10.035 ms oleg422-client.virtnet: 1 147274 2.27 MB/sec execute 289 sec latency 9.178 ms oleg422-client.virtnet: 1 148036 2.27 MB/sec execute 290 sec latency 8.789 ms oleg422-client.virtnet: 1 149131 2.30 MB/sec execute 291 sec latency 11.196 ms oleg422-client.virtnet: 1 149621 2.29 MB/sec execute 292 sec latency 9.827 ms oleg422-client.virtnet: 1 149946 2.29 MB/sec execute 293 sec latency 19.630 ms oleg422-client.virtnet: 1 150275 2.28 MB/sec execute 294 sec latency 16.979 ms oleg422-client.virtnet: 1 150702 2.29 MB/sec execute 295 sec latency 15.065 ms oleg422-client.virtnet: 1 151375 2.29 MB/sec execute 296 sec latency 8.599 ms oleg422-client.virtnet: 1 152236 2.30 MB/sec execute 297 sec latency 13.474 ms oleg422-client.virtnet: 1 152892 2.31 MB/sec execute 298 sec latency 11.538 ms oleg422-client.virtnet: 1 153241 2.30 MB/sec execute 299 sec latency 14.852 ms oleg422-client.virtnet: 1 cleanup 300 sec oleg422-client.virtnet: 0 cleanup 300 sec oleg422-client.virtnet: oleg422-client.virtnet: Operation Count AvgLat MaxLat oleg422-client.virtnet: ---------------------------------------- oleg422-client.virtnet: NTCreateX 22257 7.533 18635.163 oleg422-client.virtnet: Close 16310 2.033 18501.573 oleg422-client.virtnet: Rename 940 7.348 19.390 oleg422-client.virtnet: Unlink 4527 3.480 4762.794 oleg422-client.virtnet: Qpathinfo 20151 2.472 18528.590 oleg422-client.virtnet: Qfileinfo 3502 0.133 2.554 oleg422-client.virtnet: Qfsinfo 3697 0.197 4.606 oleg422-client.virtnet: Sfileinfo 1817 4.318 14.042 oleg422-client.virtnet: Find 7785 0.366 10.482 oleg422-client.virtnet: WriteX 10965 0.769 15.213 oleg422-client.virtnet: ReadX 34736 0.017 2.691 oleg422-client.virtnet: LockX 72 0.897 1.870 oleg422-client.virtnet: UnlockX 72 0.932 1.936 oleg422-client.virtnet: Flush 1571 4.726 414.847 oleg422-client.virtnet: oleg422-client.virtnet: Throughput 2.30394 MB/sec 1 clients 1 procs max_latency=18635.176 ms oleg422-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg422-client.virtnet at Tue Feb 25 05:38:08 EST 2025 with return code 0 oleg422-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg422-client.virtnet oleg422-client.virtnet: /mnt/lustre/d70b.replay-single/oleg422-client.virtnet /mnt/lustre/d70b.replay-single/oleg422-client.virtnet oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg422-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg422-client.virtnet: removed directory: 'clients/client0' oleg422-client.virtnet: removed directory: 'clients' oleg422-client.virtnet: removed 'client.txt' oleg422-client.virtnet: /mnt/lustre/d70b.replay-single/oleg422-client.virtnet oleg422-client.virtnet: dbench successfully finished