== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 14:05:00 (1741629900) Starting client oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre Started clients oleg143-client.virtnet: 192.168.201.143@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg143-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13438 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1576 3593280 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26148 3556152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7149432 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg143-server Stopping /mnt/lustre-mds1 (opts:) on oleg143-server 14:05:07 (1741629907) shut down facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds1 to oleg143-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 14:05:21 (1741629921) targets are mounted 14:05:21 (1741629921) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1744 3594004 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26412 3568812 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28156 7162816 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg143-server Stopping /mnt/lustre-mds2 (opts:) on oleg143-server 14:05:33 (1741629933) shut down facet: mds2 facet_host: oleg143-server facet_failover_host: oleg143-server oleg143-client.virtnet: looking for dbench program oleg143-client.virtnet: /usr/bin/dbench oleg143-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg143-client.virtnet oleg143-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg143-client.virtnet' oleg143-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg143-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg143-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg143-client.virtnet at Mon Mar 10 14:05:00 EDT 2025 oleg143-client.virtnet: waiting for dbench pid 13463 oleg143-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg143-client.virtnet: oleg143-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg143-client.virtnet: failed to create barrier semaphore oleg143-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg143-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg143-client.virtnet: releasing clients oleg143-client.virtnet: 1 616 20.98 MB/sec warmup 1 sec latency 15.163 ms oleg143-client.virtnet: 1 1259 11.55 MB/sec warmup 2 sec latency 8.518 ms oleg143-client.virtnet: 1 1978 8.82 MB/sec warmup 3 sec latency 9.837 ms oleg143-client.virtnet: 1 2593 7.25 MB/sec warmup 4 sec latency 11.272 ms oleg143-client.virtnet: 1 3689 7.75 MB/sec warmup 5 sec latency 11.092 ms oleg143-client.virtnet: 1 4076 6.68 MB/sec warmup 6 sec latency 14.091 ms oleg143-client.virtnet: 1 4083 5.73 MB/sec warmup 7 sec latency 980.893 ms oleg143-client.virtnet: 1 4083 5.01 MB/sec warmup 8 sec latency 1981.071 ms oleg143-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 2981.204 ms oleg143-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 3981.372 ms oleg143-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 4981.527 ms oleg143-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 5981.678 ms oleg143-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 6981.801 ms oleg143-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 7982.132 ms oleg143-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 8982.264 ms oleg143-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 9982.430 ms oleg143-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 10982.549 ms oleg143-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 11982.656 ms oleg143-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 12982.774 ms oleg143-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 13982.887 ms oleg143-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 14983.039 ms oleg143-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 15983.377 ms oleg143-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 16983.705 ms oleg143-client.virtnet: 1 4117 1.67 MB/sec warmup 24 sec latency 17915.996 ms oleg143-client.virtnet: 1 4550 1.64 MB/sec warmup 25 sec latency 14.854 ms oleg143-client.virtnet: 1 5030 1.74 MB/sec warmup 26 sec latency 16.398 ms oleg143-client.virtnet: 1 5377 1.68 MB/sec warmup 27 sec latency 20.724 ms oleg143-client.virtnet: 1 5913 1.68 MB/sec warmup 28 sec latency 14.278 ms oleg143-client.virtnet: 1 6489 1.81 MB/sec warmup 29 sec latency 19.260 ms oleg143-client.virtnet: 1 7066 1.92 MB/sec warmup 30 sec latency 469.695 ms oleg143-client.virtnet: 1 7555 1.91 MB/sec warmup 31 sec latency 10.436 ms oleg143-client.virtnet: 1 7997 1.87 MB/sec warmup 32 sec latency 28.332 ms oleg143-client.virtnet: 1 7997 1.82 MB/sec warmup 33 sec latency 1028.465Failover mds2 to oleg143-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 14:05:48 (1741629948) targets are mounted 14:05:48 (1741629948) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12128 3593816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3568912 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49544 7162728 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg143-server Stopping /mnt/lustre-mds1 (opts:) on oleg143-server 14:06:00 (1741629960) shut down facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds1 to oleg143-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 14:06:15 (1741629975) targets are mounted 14:06:15 (1741629975) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid ms oleg143-client.virtnet: 1 7997 1.76 MB/sec warmup 34 sec latency 2028.800 ms oleg143-client.virtnet: 1 7997 1.71 MB/sec warmup 35 sec latency 3028.959 ms oleg143-client.virtnet: 1 7997 1.66 MB/sec warmup 36 sec latency 4029.402 ms oleg143-client.virtnet: 1 7997 1.62 MB/sec warmup 37 sec latency 5029.746 ms oleg143-client.virtnet: 1 7997 1.58 MB/sec warmup 38 sec latency 6029.894 ms oleg143-client.virtnet: 1 7997 1.54 MB/sec warmup 39 sec latency 7030.192 ms oleg143-client.virtnet: 1 7997 1.50 MB/sec warmup 40 sec latency 8030.337 ms oleg143-client.virtnet: 1 7997 1.46 MB/sec warmup 41 sec latency 9030.465 ms oleg143-client.virtnet: 1 7997 1.43 MB/sec warmup 42 sec latency 10030.861 ms oleg143-client.virtnet: 1 7997 1.39 MB/sec warmup 43 sec latency 11031.203 ms oleg143-client.virtnet: 1 7997 1.36 MB/sec warmup 44 sec latency 12031.376 ms oleg143-client.virtnet: 1 7997 1.33 MB/sec warmup 45 sec latency 13031.760 ms oleg143-client.virtnet: 1 7997 1.30 MB/sec warmup 46 sec latency 14033.791 ms oleg143-client.virtnet: 1 7997 1.27 MB/sec warmup 47 sec latency 15033.963 ms oleg143-client.virtnet: 1 7997 1.25 MB/sec warmup 48 sec latency 16034.291 ms oleg143-client.virtnet: 1 7997 1.22 MB/sec warmup 49 sec latency 17034.455 ms oleg143-client.virtnet: 1 7997 1.20 MB/sec warmup 50 sec latency 18034.955 ms oleg143-client.virtnet: 1 8014 1.17 MB/sec warmup 51 sec latency 18948.153 ms oleg143-client.virtnet: 1 8384 1.18 MB/sec warmup 52 sec latency 18.565 ms oleg143-client.virtnet: 1 8787 1.22 MB/sec warmup 53 sec latency 25.500 ms oleg143-client.virtnet: 1 9254 1.20 MB/sec warmup 54 sec latency 13.575 ms oleg143-client.virtnet: 1 9818 1.28 MB/sec warmup 55 sec latency 17.466 ms oleg143-client.virtnet: 1 10671 1.37 MB/sec warmup 56 sec latency 12.984 ms oleg143-client.virtnet: 1 11077 1.37 MB/sec warmup 57 sec latency 11.794 ms oleg143-client.virtnet: 1 11464 1.36 MB/sec warmup 58 sec latency 14.614 ms oleg143-client.virtnet: 1 11907 1.36 MB/sec warmup 59 sec latency 15.755 ms oleg143-client.virtnet: 1 12992 1.55 MB/sec execute 1 sec latency 13.970 ms oleg143-client.virtnet: 1 13650 3.63 MB/sec execute 2 sec latency 16.077 ms oleg143-client.virtnet: 1 14316 4.15 MB/sec execute 3 sec latency 20.070 ms oleg143-client.virtnet: 1 14649 3.45 MB/sec execute 4 sec latency 15.939 ms oleg143-client.virtnet: 1 15013 2.86 MB/sec execute 5 sec latency 20.636 ms oleg143-client.virtnet: 1 15363 2.64 MB/sec execute 6 sec latency 19.893 ms oleg143-client.virtnet: 1 15777 2.71 MB/sec execute 7 sec latency 18.483 ms oleg143-client.virtnet: 1 16069 2.40 MB/sec execute 8 sec latency 20.874 ms oleg143-client.virtnet: 1 16492 2.29 MB/sec execute 9 sec latency 21.215 ms oleg143-client.virtnet: 1 17018 2.49 MB/sec execute 10 sec latency 19.564 ms oleg143-client.virtnet: 1 17764 2.84 MB/sec execute 11 sec latency 13.912 ms oleg143-client.virtnet: 1 18086 2.74 MB/sec execute 12 sec latency 18.188 ms oleg143-client.virtnet: 1 18301 2.53 MB/sec execute 13 sec latency 21.667 ms oleg143-client.virtnet: 1 18693 2.41 MB/sec execute 14 sec latency 23.056 ms oleg143-client.virtnet: 1 19152 2.53 MB/sec execute 15 sec latency 17.094 ms oleg143-client.virtnet: 1 19461 2.39 MB/sec execute 16 sec latency 28.688 ms oleg143-client.virtnet: 1 19881 2.27 MB/sec execute 17 sec latency 15.530 ms oleg143-client.virtnet: 1 20239 2.23 MB/sec execute 18 sec latency 17.853 ms oleg143-client.virtnet: 1 20977 2.44 MB/sec execute 19 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12096 3591872 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37540 3568892 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160764 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg143-server Stopping /mnt/lustre-mds2 (opts:) on oleg143-server 14:06:28 (1741629988) shut down facet: mds2 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds2 to oleg143-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 14:06:43 (1741630003) targets are mounted 14:06:43 (1741630003) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12220 3592796 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37540 3569040 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49760 7161836 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg143-server Stopping /mnt/lustre-mds1 (opts:) on oleg143-server 14:06:55 (1741630015) shut down facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server sec latency 16.995 ms oleg143-client.virtnet: 1 21531 2.59 MB/sec execute 20 sec latency 21.417 ms oleg143-client.virtnet: 1 21848 2.48 MB/sec execute 21 sec latency 16.537 ms oleg143-client.virtnet: 1 22248 2.40 MB/sec execute 22 sec latency 21.179 ms oleg143-client.virtnet: 1 22546 2.36 MB/sec execute 23 sec latency 25.667 ms oleg143-client.virtnet: 1 22947 2.39 MB/sec execute 24 sec latency 16.915 ms oleg143-client.virtnet: 1 23162 2.30 MB/sec execute 25 sec latency 188.222 ms oleg143-client.virtnet: 1 23483 2.22 MB/sec execute 26 sec latency 20.984 ms oleg143-client.virtnet: 1 23755 2.19 MB/sec execute 27 sec latency 164.297 ms oleg143-client.virtnet: 1 23755 2.12 MB/sec execute 28 sec latency 1164.496 ms oleg143-client.virtnet: 1 23755 2.04 MB/sec execute 29 sec latency 2164.654 ms oleg143-client.virtnet: 1 23755 1.97 MB/sec execute 30 sec latency 3164.861 ms oleg143-client.virtnet: 1 23755 1.91 MB/sec execute 31 sec latency 4165.249 ms oleg143-client.virtnet: 1 23755 1.85 MB/sec execute 32 sec latency 5165.429 ms oleg143-client.virtnet: 1 23755 1.79 MB/sec execute 33 sec latency 6165.600 ms oleg143-client.virtnet: 1 23755 1.74 MB/sec execute 34 sec latency 7165.757 ms oleg143-client.virtnet: 1 23755 1.69 MB/sec execute 35 sec latency 8165.978 ms oleg143-client.virtnet: 1 23755 1.65 MB/sec execute 36 sec latency 9166.327 ms oleg143-client.virtnet: 1 23755 1.60 MB/sec execute 37 sec latency 10169.595 ms oleg143-client.virtnet: 1 23755 1.56 MB/sec execute 38 sec latency 11169.719 ms oleg143-client.virtnet: 1 23755 1.52 MB/sec execute 39 sec latency 12169.892 ms oleg143-client.virtnet: 1 23755 1.48 MB/sec execute 40 sec latency 13170.269 ms oleg143-client.virtnet: 1 23755 1.44 MB/sec execute 41 sec latency 14170.417 ms oleg143-client.virtnet: 1 23755 1.41 MB/sec execute 42 sec latency 15170.604 ms oleg143-client.virtnet: 1 23755 1.38 MB/sec execute 43 sec latency 16170.899 ms oleg143-client.virtnet: 1 23755 1.35 MB/sec execute 44 sec latency 17171.313 ms oleg143-client.virtnet: 1 23755 1.32 MB/sec execute 45 sec latency 18171.472 ms oleg143-client.virtnet: 1 23799 1.30 MB/sec execute 46 sec latency 19039.528 ms oleg143-client.virtnet: 1 24500 1.39 MB/sec execute 47 sec latency 16.397 ms oleg143-client.virtnet: 1 25070 1.48 MB/sec execute 48 sec latency 19.844 ms oleg143-client.virtnet: 1 25430 1.45 MB/sec execute 49 sec latency 15.679 ms oleg143-client.virtnet: 1 25867 1.44 MB/sec execute 50 sec latency 17.017 ms oleg143-client.virtnet: 1 26439 1.50 MB/sec execute 51 sec latency 14.241 ms oleg143-client.virtnet: 1 26933 1.48 MB/sec execute 52 sec latency 13.229 ms oleg143-client.virtnet: 1 27445 1.49 MB/sec execute 53 sec latency 12.922 ms oleg143-client.virtnet: 1 28296 1.60 MB/sec execute 54 sec latency 15.152 ms oleg143-client.virtnet: 1 28769 1.64 MB/sec execute 55 sec latency 14.327 ms oleg143-client.virtnet: 1 29204 1.62 MB/sec execute 56 sec latency 15.017 ms oleg143-client.virtnet: 1 29580 1.62 MB/sec execute 57 sec latency 13.540 ms oleg143-client.virtnet: 1 30003 1.65 MB/sec execute 58 sec latency 20.370 ms oleg143-client.virtnet: 1 30481 1.63 MB/sec execute 59 sec latency 13.223 ms oleg143-client.virtnet: 1 30991 1.64 MB/sec execute 60 sec latency 13.928 ms oleg143-client.virtnet: 1 31907 1.77 MB/sec execute 61 sec latency 16.884 ms oleg143-client.virtnet: 1 32347 1.76 MB/sec execute 62 sec latency 15.759 ms oleg143-client.virtnet: 1 32752 1.74 MB/sec execute 63 sec latency 14.542 ms oleg143-client.virtFailover mds1 to oleg143-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 14:07:10 (1741630030) targets are mounted 14:07:10 (1741630030) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12096 3591872 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37540 3568892 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160764 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg143-server Stopping /mnt/lustre-mds2 (opts:) on oleg143-server 14:07:22 (1741630042) shut down facet: mds2 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds2 to oleg143-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 14:07:36 (1741630056) targets are mounted 14:07:36 (1741630056) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12256 3593264 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37568 3568432 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49824 7161696 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg143-server Stopping /mnt/lustre-mds1 (opts:) on oleg143-server net: 1 33323 1.78 MB/sec execute 64 sec latency 13.408 ms oleg143-client.virtnet: 1 33804 1.77 MB/sec execute 65 sec latency 12.770 ms oleg143-client.virtnet: 1 34362 1.76 MB/sec execute 66 sec latency 11.049 ms oleg143-client.virtnet: 1 35258 1.85 MB/sec execute 67 sec latency 12.566 ms oleg143-client.virtnet: 1 35837 1.89 MB/sec execute 68 sec latency 11.410 ms oleg143-client.virtnet: 1 36292 1.87 MB/sec execute 69 sec latency 14.562 ms oleg143-client.virtnet: 1 36742 1.87 MB/sec execute 70 sec latency 13.213 ms oleg143-client.virtnet: 1 37190 1.88 MB/sec execute 71 sec latency 18.132 ms oleg143-client.virtnet: 1 37656 1.86 MB/sec execute 72 sec latency 16.707 ms oleg143-client.virtnet: 1 38361 1.91 MB/sec execute 73 sec latency 11.077 ms oleg143-client.virtnet: 1 39146 1.98 MB/sec execute 74 sec latency 12.586 ms oleg143-client.virtnet: 1 39517 1.97 MB/sec execute 75 sec latency 13.535 ms oleg143-client.virtnet: 1 39968 1.95 MB/sec execute 76 sec latency 19.460 ms oleg143-client.virtnet: 1 40325 1.95 MB/sec execute 77 sec latency 21.681 ms oleg143-client.virtnet: 1 40659 1.96 MB/sec execute 78 sec latency 20.734 ms oleg143-client.virtnet: 1 40851 1.94 MB/sec execute 79 sec latency 415.913 ms oleg143-client.virtnet: 1 41270 1.92 MB/sec execute 80 sec latency 17.462 ms oleg143-client.virtnet: 1 41934 1.96 MB/sec execute 81 sec latency 160.079 ms oleg143-client.virtnet: 1 41934 1.94 MB/sec execute 82 sec latency 1160.443 ms oleg143-client.virtnet: 1 41934 1.91 MB/sec execute 83 sec latency 2160.632 ms oleg143-client.virtnet: 1 41934 1.89 MB/sec execute 84 sec latency 3160.787 ms oleg143-client.virtnet: 1 41934 1.87 MB/sec execute 85 sec latency 4160.973 ms oleg143-client.virtnet: 1 41934 1.85 MB/sec execute 86 sec latency 5161.436 ms oleg143-client.virtnet: 1 41934 1.83 MB/sec execute 87 sec latency 6161.650 ms oleg143-client.virtnet: 1 41934 1.81 MB/sec execute 88 sec latency 7162.022 ms oleg143-client.virtnet: 1 41934 1.79 MB/sec execute 89 sec latency 8162.263 ms oleg143-client.virtnet: 1 41934 1.77 MB/sec execute 90 sec latency 9162.587 ms oleg143-client.virtnet: 1 41934 1.75 MB/sec execute 91 sec latency 10162.789 ms oleg143-client.virtnet: 1 41934 1.73 MB/sec execute 92 sec latency 11162.976 ms oleg143-client.virtnet: 1 41934 1.71 MB/sec execute 93 sec latency 12163.138 ms oleg143-client.virtnet: 1 41934 1.69 MB/sec execute 94 sec latency 13163.559 ms oleg143-client.virtnet: 1 41934 1.67 MB/sec execute 95 sec latency 14165.451 ms oleg143-client.virtnet: 1 41934 1.66 MB/sec execute 96 sec latency 15165.684 ms oleg143-client.virtnet: 1 41934 1.64 MB/sec execute 97 sec latency 16165.840 ms oleg143-client.virtnet: 1 41934 1.62 MB/sec execute 98 sec latency 17165.998 ms oleg143-client.virtnet: 1 41934 1.61 MB/sec execute 99 sec latency 18166.738 ms oleg143-client.virtnet: 1 42419 1.62 MB/sec execute 100 sec latency 18558.842 ms oleg143-client.virtnet: 1 42926 1.65 MB/sec execute 101 sec latency 14.777 ms oleg143-client.virtnet: 1 43332 1.64 MB/sec execute 102 sec latency 20.787 ms oleg143-client.virtnet: 1 43800 1.64 MB/sec execute 103 sec latency 13.376 ms oleg143-client.virtnet: 1 44237 1.65 MB/sec execute 104 sec latency 13.107 ms oleg143-client.virtnet: 1 44619 1.64 MB/sec execute 105 sec latency 17.445 ms oleg143-client.virtnet: 1 45064 1.65 MB/sec execute 106 sec latency 14.225 ms oleg143-client.virtnet: 1 45804 1.68 MB/sec execute 107 sec latency 18.117 ms oleg143-client.virtnet: 1 46376 1.72 MB/sec execu14:07:49 (1741630069) shut down facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds1 to oleg143-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 14:08:03 (1741630083) targets are mounted 14:08:03 (1741630083) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11912 3594220 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37828 3568296 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49740 7162516 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg143-server Stopping /mnt/lustre-mds2 (opts:) on oleg143-server 14:08:15 (1741630095) shut down facet: mds2 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds2 to oleg143-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 14:08:30 (1741630110) targets are mounted 14:08:30 (1741630110) facet_failover done te 108 sec latency 21.206 ms oleg143-client.virtnet: 1 46711 1.71 MB/sec execute 109 sec latency 15.662 ms oleg143-client.virtnet: 1 47109 1.70 MB/sec execute 110 sec latency 16.566 ms oleg143-client.virtnet: 1 47576 1.72 MB/sec execute 111 sec latency 17.376 ms oleg143-client.virtnet: 1 47944 1.71 MB/sec execute 112 sec latency 16.576 ms oleg143-client.virtnet: 1 48498 1.71 MB/sec execute 113 sec latency 15.365 ms oleg143-client.virtnet: 1 49312 1.75 MB/sec execute 114 sec latency 11.145 ms oleg143-client.virtnet: 1 49966 1.78 MB/sec execute 115 sec latency 12.968 ms oleg143-client.virtnet: 1 50438 1.77 MB/sec execute 116 sec latency 15.399 ms oleg143-client.virtnet: 1 50962 1.77 MB/sec execute 117 sec latency 11.779 ms oleg143-client.virtnet: 1 51475 1.78 MB/sec execute 118 sec latency 16.249 ms oleg143-client.virtnet: 1 52061 1.78 MB/sec execute 119 sec latency 12.233 ms oleg143-client.virtnet: 1 52919 1.82 MB/sec execute 120 sec latency 12.420 ms oleg143-client.virtnet: 1 53581 1.85 MB/sec execute 121 sec latency 9.432 ms oleg143-client.virtnet: 1 54013 1.84 MB/sec execute 122 sec latency 12.887 ms oleg143-client.virtnet: 1 54444 1.84 MB/sec execute 123 sec latency 17.575 ms oleg143-client.virtnet: 1 54953 1.85 MB/sec execute 124 sec latency 13.084 ms oleg143-client.virtnet: 1 55405 1.84 MB/sec execute 125 sec latency 15.597 ms oleg143-client.virtnet: 1 56210 1.88 MB/sec execute 126 sec latency 11.281 ms oleg143-client.virtnet: 1 56941 1.91 MB/sec execute 127 sec latency 14.707 ms oleg143-client.virtnet: 1 57375 1.90 MB/sec execute 128 sec latency 10.819 ms oleg143-client.virtnet: 1 57818 1.90 MB/sec execute 129 sec latency 19.018 ms oleg143-client.virtnet: 1 58404 1.91 MB/sec execute 130 sec latency 12.540 ms oleg143-client.virtnet: 1 58928 1.90 MB/sec execute 131 sec latency 11.486 ms oleg143-client.virtnet: 1 59333 1.90 MB/sec execute 132 sec latency 247.799 ms oleg143-client.virtnet: 1 60154 1.94 MB/sec execute 133 sec latency 14.736 ms oleg143-client.virtnet: 1 60615 1.96 MB/sec execute 134 sec latency 119.067 ms oleg143-client.virtnet: 1 60615 1.94 MB/sec execute 135 sec latency 1119.197 ms oleg143-client.virtnet: 1 60615 1.93 MB/sec execute 136 sec latency 2119.435 ms oleg143-client.virtnet: 1 60615 1.91 MB/sec execute 137 sec latency 3119.606 ms oleg143-client.virtnet: 1 60615 1.90 MB/sec execute 138 sec latency 4119.741 ms oleg143-client.virtnet: 1 60615 1.89 MB/sec execute 139 sec latency 5119.901 ms oleg143-client.virtnet: 1 60615 1.87 MB/sec execute 140 sec latency 6120.201 ms oleg143-client.virtnet: 1 60615 1.86 MB/sec execute 141 sec latency 7121.278 ms oleg143-client.virtnet: 1 60615 1.85 MB/sec execute 142 sec latency 8121.442 ms oleg143-client.virtnet: 1 60615 1.83 MB/sec execute 143 sec latency 9121.580 ms oleg143-client.virtnet: 1 60615 1.82 MB/sec execute 144 sec latency 10121.746 ms oleg143-client.virtnet: 1 60615 1.81 MB/sec execute 145 sec latency 11123.614 ms oleg143-client.virtnet: 1 60615 1.79 MB/sec execute 146 sec latency 12125.683 ms oleg143-client.virtnet: 1 60615 1.78 MB/sec execute 147 sec latency 13125.968 ms oleg143-client.virtnet: 1 60615 1.77 MB/sec execute 148 sec latency 14126.103 ms oleg143-client.virtnet: 1 60615 1.76 MB/sec execute 149 sec latency 15126.387 ms oleg143-client.virtnet: 1 60615 1.75 MB/sec execute 150 sec latency 16126.541 ms oleg143-client.virtnet: 1 60615 1.74 MB/sec execute 151 sec latency 17126.922 ms oleg143-client.virtnet: 1 60615 1.72 MB/sec execute 152 sec latency 18127.244 ms oleg143-client.voleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12072 3594064 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3569236 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163300 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg143-server Stopping /mnt/lustre-mds1 (opts:) on oleg143-server 14:08:42 (1741630122) shut down facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds1 to oleg143-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 14:08:57 (1741630137) targets are mounted 14:08:57 (1741630137) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12128 3591644 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37416 3568272 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49544 7159916 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg143-server Stopping /mnt/lustre-mds2 (opts:) on oleg143-server 14:09:16 (1741630156) shut down facet: mds2 facet_host: oleg143-server facet_failover_host: oleg143-server irtnet: 1 60643 1.71 MB/sec execute 153 sec latency 19042.608 ms oleg143-client.virtnet: 1 61032 1.71 MB/sec execute 154 sec latency 14.299 ms oleg143-client.virtnet: 1 61411 1.71 MB/sec execute 155 sec latency 18.486 ms oleg143-client.virtnet: 1 61852 1.71 MB/sec execute 156 sec latency 17.787 ms oleg143-client.virtnet: 1 62155 1.71 MB/sec execute 157 sec latency 19.891 ms oleg143-client.virtnet: 1 62676 1.70 MB/sec execute 158 sec latency 12.386 ms oleg143-client.virtnet: 1 63257 1.73 MB/sec execute 159 sec latency 18.427 ms oleg143-client.virtnet: 1 64008 1.75 MB/sec execute 160 sec latency 15.679 ms oleg143-client.virtnet: 1 64354 1.75 MB/sec execute 161 sec latency 13.777 ms oleg143-client.virtnet: 1 64670 1.74 MB/sec execute 162 sec latency 20.624 ms oleg143-client.virtnet: 1 65043 1.74 MB/sec execute 163 sec latency 19.078 ms oleg143-client.virtnet: 1 65565 1.75 MB/sec execute 164 sec latency 13.695 ms oleg143-client.virtnet: 1 65956 1.74 MB/sec execute 165 sec latency 14.236 ms oleg143-client.virtnet: 1 66432 1.74 MB/sec execute 166 sec latency 16.144 ms oleg143-client.virtnet: 1 67145 1.77 MB/sec execute 167 sec latency 21.650 ms oleg143-client.virtnet: 1 67666 1.79 MB/sec execute 168 sec latency 15.768 ms oleg143-client.virtnet: 1 68041 1.78 MB/sec execute 169 sec latency 13.554 ms oleg143-client.virtnet: 1 68420 1.77 MB/sec execute 170 sec latency 16.001 ms oleg143-client.virtnet: 1 68942 1.79 MB/sec execute 171 sec latency 14.622 ms oleg143-client.virtnet: 1 69312 1.78 MB/sec execute 172 sec latency 16.171 ms oleg143-client.virtnet: 1 69763 1.78 MB/sec execute 173 sec latency 24.311 ms oleg143-client.virtnet: 1 70193 1.79 MB/sec execute 174 sec latency 17.768 ms oleg143-client.virtnet: 1 70953 1.82 MB/sec execute 175 sec latency 16.113 ms oleg143-client.virtnet: 1 71249 1.81 MB/sec execute 176 sec latency 25.369 ms oleg143-client.virtnet: 1 71520 1.81 MB/sec execute 177 sec latency 22.092 ms oleg143-client.virtnet: 1 71921 1.80 MB/sec execute 178 sec latency 25.926 ms oleg143-client.virtnet: 1 72456 1.81 MB/sec execute 179 sec latency 14.310 ms oleg143-client.virtnet: 1 72841 1.81 MB/sec execute 180 sec latency 13.906 ms oleg143-client.virtnet: 1 73170 1.80 MB/sec execute 181 sec latency 24.181 ms oleg143-client.virtnet: 1 73567 1.80 MB/sec execute 182 sec latency 24.263 ms oleg143-client.virtnet: 1 74347 1.83 MB/sec execute 183 sec latency 20.520 ms oleg143-client.virtnet: 1 74805 1.84 MB/sec execute 184 sec latency 17.364 ms oleg143-client.virtnet: 1 75232 1.83 MB/sec execute 185 sec latency 13.205 ms oleg143-client.virtnet: 1 75467 1.83 MB/sec execute 186 sec latency 355.644 ms oleg143-client.virtnet: 1 75793 1.82 MB/sec execute 187 sec latency 434.743 ms oleg143-client.virtnet: 1 76205 1.83 MB/sec execute 188 sec latency 17.072 ms oleg143-client.virtnet: 1 76363 1.82 MB/sec execute 189 sec latency 465.559 ms oleg143-client.virtnet: 1 76363 1.81 MB/sec execute 190 sec latency 1465.723 ms oleg143-client.virtnet: 1 76363 1.80 MB/sec execute 191 sec latency 2465.959 ms oleg143-client.virtnet: 1 76363 1.79 MB/sec execute 192 sec latency 3466.123 ms oleg143-client.virtnet: 1 76363 1.78 MB/sec execute 193 sec latency 4466.347 ms oleg143-client.virtnet: 1 76363 1.77 MB/sec execute 194 sec latency 5466.471 ms oleg143-client.virtnet: 1 76363 1.77 MB/sec execute 195 sec latency 6466.618 ms oleg143-client.virtnet: 1 76363 1.76 MB/sec execute 196 sec latency 7466.939 ms oleg143-client.virtnet: 1 76363 1.75 MB/sec execute 197 sec latency 8467.08Failover mds2 to oleg143-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 14:09:30 (1741630170) targets are mounted 14:09:30 (1741630170) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12248 3593664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37428 3566748 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49676 7160412 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg143-server Stopping /mnt/lustre-mds1 (opts:) on oleg143-server 14:09:43 (1741630183) shut down facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds1 to oleg143-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 14:09:57 (1741630197) targets are mounted 14:09:57 (1741630197) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 6 ms oleg143-client.virtnet: 1 76363 1.74 MB/sec execute 198 sec latency 9467.307 ms oleg143-client.virtnet: 1 76363 1.73 MB/sec execute 199 sec latency 10467.438 ms oleg143-client.virtnet: 1 76363 1.72 MB/sec execute 200 sec latency 11467.807 ms oleg143-client.virtnet: 1 76363 1.71 MB/sec execute 201 sec latency 12467.927 ms oleg143-client.virtnet: 1 76363 1.70 MB/sec execute 202 sec latency 13468.079 ms oleg143-client.virtnet: 1 76363 1.70 MB/sec execute 203 sec latency 14468.221 ms oleg143-client.virtnet: 1 76363 1.69 MB/sec execute 204 sec latency 15468.377 ms oleg143-client.virtnet: 1 76363 1.68 MB/sec execute 205 sec latency 16468.521 ms oleg143-client.virtnet: 1 76363 1.67 MB/sec execute 206 sec latency 17468.846 ms oleg143-client.virtnet: 1 76363 1.66 MB/sec execute 207 sec latency 18469.028 ms oleg143-client.virtnet: 1 76363 1.66 MB/sec execute 208 sec latency 19469.357 ms oleg143-client.virtnet: 1 76363 1.65 MB/sec execute 209 sec latency 20469.512 ms oleg143-client.virtnet: 1 76363 1.64 MB/sec execute 210 sec latency 21469.665 ms oleg143-client.virtnet: 1 76363 1.63 MB/sec execute 211 sec latency 22469.821 ms oleg143-client.virtnet: 1 76363 1.62 MB/sec execute 212 sec latency 23470.235 ms oleg143-client.virtnet: 1 76363 1.62 MB/sec execute 213 sec latency 24470.425 ms oleg143-client.virtnet: 1 76641 1.61 MB/sec execute 214 sec latency 24772.530 ms oleg143-client.virtnet: 1 77114 1.61 MB/sec execute 215 sec latency 19.467 ms oleg143-client.virtnet: 1 77950 1.64 MB/sec execute 216 sec latency 14.548 ms oleg143-client.virtnet: 1 78409 1.65 MB/sec execute 217 sec latency 15.588 ms oleg143-client.virtnet: 1 78753 1.64 MB/sec execute 218 sec latency 14.928 ms oleg143-client.virtnet: 1 79077 1.64 MB/sec execute 219 sec latency 16.293 ms oleg143-client.virtnet: 1 79616 1.65 MB/sec execute 220 sec latency 11.663 ms oleg143-client.virtnet: 1 80071 1.64 MB/sec execute 221 sec latency 14.428 ms oleg143-client.virtnet: 1 80589 1.65 MB/sec execute 222 sec latency 11.836 ms oleg143-client.virtnet: 1 81694 1.68 MB/sec execute 223 sec latency 11.263 ms oleg143-client.virtnet: 1 82150 1.68 MB/sec execute 224 sec latency 10.176 ms oleg143-client.virtnet: 1 82790 1.69 MB/sec execute 225 sec latency 11.098 ms oleg143-client.virtnet: 1 83428 1.69 MB/sec execute 226 sec latency 12.936 ms oleg143-client.virtnet: 1 84063 1.69 MB/sec execute 227 sec latency 10.976 ms oleg143-client.virtnet: 1 84998 1.72 MB/sec execute 228 sec latency 11.963 ms oleg143-client.virtnet: 1 85573 1.73 MB/sec execute 229 sec latency 11.187 ms oleg143-client.virtnet: 1 86108 1.73 MB/sec execute 230 sec latency 12.684 ms oleg143-client.virtnet: 1 86758 1.74 MB/sec execute 231 sec latency 10.592 ms oleg143-client.virtnet: 1 87324 1.73 MB/sec execute 232 sec latency 11.840 ms oleg143-client.virtnet: 1 88092 1.75 MB/sec execute 233 sec latency 11.672 ms oleg143-client.virtnet: 1 88916 1.77 MB/sec execute 234 sec latency 11.312 ms oleg143-client.virtnet: 1 89380 1.77 MB/sec execute 235 sec latency 11.167 ms oleg143-client.virtnet: 1 89865 1.77 MB/sec execute 236 sec latency 18.315 ms oleg143-client.virtnet: 1 90410 1.77 MB/sec execute 237 sec latency 10.725 ms oleg143-client.virtnet: 1 90921 1.77 MB/sec execute 238 sec latency 12.680 ms oleg143-client.virtnet: 1 91647 1.79 MB/sec execute 239 sec latency 16.670 ms oleg143-client.virtnet: 1 92409 1.80 MB/sec execute 240 sec latency 12.288 ms oleg143-client.virtnet: 1 92790 1.80 MB/sec execute 241 sec latency 15.151 ms oleg143-client.virtnet: 1 93238 1.80 MB/sec execute 242 sec latency 26.090 ms oleg143-client.virtnet: 1 93762 1.81 MB/sec execute 243 sec latency 14.447 ms oleg143-client.virtnet: 1 94256 1.80 MB/sec execute 244 sec latency 11.503 ms oleg143-client.virtnet: 1 94829 1.81 MB/sec execute 245 sec latency 11.411 ms oleg143-client.virtnet: 1 95657 1.83 MB/sec execute 246 sec latency 14.063 ms oleg143-client.virtnet: 1 96202 1.84 MB/sec execute 247 sec latency 15.939 ms oleg143-client.virtnet: 1 96700 1.83 MB/sec execute 248 sec latency 13.532 ms oleg143-client.virtnet: 1 97301 1.84 MB/sec execute 249 sec latency 14.972 ms oleg143-client.virtnet: 1 97832 1.84 MB/sec execute 250 sec latency 17.947 ms oleg143-client.virtnet: 1 98450 1.84 MB/sec execute 251 sec latency 12.541 ms oleg143-client.virtnet: 1 99448 1.87 MB/sec execute 252 sec latency 13.391 ms oleg143-client.virtnet: 1 99884 1.87 MB/sec execute 253 sec latency 10.365 ms oleg143-client.virtnet: 1 100420 1.87 MB/sec execute 254 sec latency 13.144 ms oleg143-client.virtnet: 1 101014 1.87 MB/sec execute 255 sec latency 13.427 ms oleg143-client.virtnet: 1 101542 1.87 MB/sec execute 256 sec latency 11.744 ms oleg143-client.virtnet: 1 102291 1.89 MB/sec execute 257 sec latency 13.816 ms oleg143-client.virtnet: 1 103086 1.90 MB/sec execute 258 sec latency 12.426 ms oleg143-client.virtnet: 1 103607 1.90 MB/sec execute 259 sec latency 13.209 ms oleg143-client.virtnet: 1 104124 1.90 MB/sec execute 260 sec latency 12.576 ms oleg143-client.virtnet: 1 104701 1.90 MB/sec execute 261 sec latency 12.718 ms oleg143-client.virtnet: 1 105300 1.90 MB/sec execute 262 sec latency 8.731 ms oleg143-client.virtnet: 1 106249 1.92 MB/sec execute 263 sec latency 12.515 ms oleg143-client.virtnet: 1 106814 1.94 MB/sec execute 264 sec latency 10.962 ms oleg143-client.virtnet: 1 107268 1.93 MB/sec execute 265 sec latency 15.529 ms oleg143-client.virtnet: 1 107835 1.94 MB/sec execute 266 sec latency 13.526 ms oleg143-client.virtnet: 1 108241 1.93 MB/sec execute 267 sec latency 17.682 ms oleg143-client.virtnet: 1 108805 1.93 MB/sec execute 268 sec latency 10.737 ms oleg143-client.virtnet: 1 109446 1.95 MB/sec execute 269 sec latency 16.540 ms oleg143-client.virtnet: 1 110215 1.96 MB/sec execute 270 sec latency 12.730 ms oleg143-client.virtnet: 1 110564 1.96 MB/sec execute 271 sec latency 15.376 ms oleg143-client.virtnet: 1 110991 1.95 MB/sec execute 272 sec latency 18.785 ms oleg143-client.virtnet: 1 111506 1.96 MB/sec execute 273 sec latency 12.899 ms oleg143-client.virtnet: 1 111822 1.95 MB/sec execute 274 sec latency 20.449 ms oleg143-client.virtnet: 1 112241 1.95 MB/sec execute 275 sec latency 14.867 ms oleg143-client.virtnet: 1 112933 1.97 MB/sec execute 276 sec latency 13.917 ms oleg143-client.virtnet: 1 113693 1.98 MB/sec execute 277 sec latency 11.840 ms oleg143-client.virtnet: 1 114076 1.98 MB/sec execute 278 sec latency 14.994 ms oleg143-client.virtnet: 1 114600 1.98 MB/sec execute 279 sec latency 15.218 ms oleg143-client.virtnet: 1 115194 1.98 MB/sec execute 280 sec latency 10.752 ms oleg143-client.virtnet: 1 115667 1.97 MB/sec execute 281 sec latency 12.776 ms oleg143-client.virtnet: 1 116293 1.99 MB/sec execute 282 sec latency 15.129 ms oleg143-client.virtnet: 1 117225 2.00 MB/sec execute 283 sec latency 11.136 ms oleg143-client.virtnet: 1 117718 2.00 MB/sec execute 284 sec latency 11.209 ms oleg143-client.virtnet: 1 118259 2.00 MB/sec execute 285 sec latency 16.798 ms oleg143-client.virtnet: 1 118852 2.01 MB/sec execute 286 sec latency 13.774 ms oleg143-client.virtnet: 1 119440 2.00 MB/sec execute 287 sec latency 11.621 ms oleg143-client.virtnet: 1 120321 2.02 MB/sec execute 288 sec latency 11.356 ms oleg143-client.virtnet: 1 120960 2.03 MB/sec execute 289 sec latency 11.037 ms oleg143-client.virtnet: 1 121453 2.03 MB/sec execute 290 sec latency 10.587 ms oleg143-client.virtnet: 1 122059 2.04 MB/sec execute 291 sec latency 12.494 ms oleg143-client.virtnet: 1 122537 2.03 MB/sec execute 292 sec latency 12.793 ms oleg143-client.virtnet: 1 123116 2.03 MB/sec execute 293 sec latency 10.993 ms oleg143-client.virtnet: 1 124142 2.06 MB/sec execute 294 sec latency 14.545 ms oleg143-client.virtnet: 1 124609 2.06 MB/sec execute 295 sec latency 11.366 ms oleg143-client.virtnet: 1 125009 2.05 MB/sec execute 296 sec latency 16.340 ms oleg143-client.virtnet: 1 125449 2.05 MB/sec execute 297 sec latency 14.255 ms oleg143-client.virtnet: 1 125977 2.05 MB/sec execute 298 sec latency 13.524 ms oleg143-client.virtnet: 1 126533 2.05 MB/sec execute 299 sec latency 10.593 ms oleg143-client.virtnet: 1 cleanup 300 sec oleg143-client.virtnet: 0 cleanup 300 sec oleg143-client.virtnet: oleg143-client.virtnet: Operation Count AvgLat MaxLat oleg143-client.virtnet: ---------------------------------------- oleg143-client.virtnet: NTCreateX 19831 10.193 24772.518 oleg143-client.virtnet: Close 14573 1.118 7.398 oleg143-client.virtnet: Rename 842 9.738 25.644 oleg143-client.virtnet: Unlink 3990 2.142 8.687 oleg143-client.virtnet: Qpathinfo 18017 1.676 23.200 oleg143-client.virtnet: Qfileinfo 3153 0.182 2.881 oleg143-client.virtnet: Qfsinfo 3287 0.276 12.846 oleg143-client.virtnet: Sfileinfo 1616 5.416 18.540 oleg143-client.virtnet: Find 6948 0.458 12.102 oleg143-client.virtnet: WriteX 9820 0.973 14.188 oleg143-client.virtnet: ReadX 31176 0.024 6.780 oleg143-client.virtnet: LockX 66 1.209 2.951 oleg143-client.virtnet: UnlockX 66 1.273 3.333 oleg143-client.virtnet: Flush 1387 7.070 434.729 oleg143-client.virtnet: oleg143-client.virtnet: Throughput 2.05179 MB/sec 1 clients 1 procs max_latency=24772.530 ms oleg143-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg143-client.virtnet at Mon Mar 10 14:11:01 EDT 2025 with return code 0 oleg143-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg143-client.virtnet oleg143-client.virtnet: /mnt/lustre/d70b.replay-single/oleg143-client.virtnet /mnt/lustre/d70b.replay-single/oleg143-client.virtnet oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg143-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg143-client.virtnet: removed directory: 'clients/client0' oleg143-client.virtnet: removed directory: 'clients' oleg143-client.virtnet: removed 'client.txt' oleg143-client.virtnet: /mnt/lustre/d70b.replay-single/oleg143-client.virtnet oleg143-client.virtnet: dbench successfully finished