== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 05:54:05 (1744538045) Starting client oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg305-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=5800 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3840 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3738624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3686400 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 13312 7425024 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:54:13 (1744538053) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:54:27 (1744538067) targets are mounted 05:54:27 (1744538067) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3840 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3738624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3664896 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 46080 7403520 1% /mnt/lustre oleg305-client.virtnet: looking for dbench program oleg305-client.virtnet: /usr/bin/dbench oleg305-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg305-client.virtnet oleg305-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg305-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg305-client.virtnet' oleg305-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg305-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg305-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg305-client.virtnet at Sun Apr 13 05:54:07 EDT 2025 oleg305-client.virtnet: waiting for dbench pid 5825 oleg305-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg305-client.virtnet: oleg305-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg305-client.virtnet: failed to create barrier semaphore oleg305-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg305-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg305-client.virtnet: releasing clients oleg305-client.virtnet: 1 313 11.48 MB/sec warmup 1 sec latency 25.511 ms oleg305-client.virtnet: 1 664 10.58 MB/sec warmup 2 sec latency 55.748 ms oleg305-client.virtnet: 1 876 7.22 MB/sec warmup 3 sec latency 87.228 ms oleg305-client.virtnet: 1 1172 5.75 MB/sec warmup 4 sec latency 36.730 ms oleg305-client.virtnet: 1 1552 5.22 MB/sec warmup 5 sec latency 38.118 ms oleg305-client.virtnet: 1 1758 4.38 MB/sec warmup 6 sec latency 418.519 ms oleg305-client.virtnet: 1 1758 3.75 MB/sec warmup 7 sec latency 1418.701 ms oleg305-client.virtnet: 1 1758 3.28 MB/sec warmup 8 sec latency 2418.956 ms oleg305-client.virtnet: 1 1758 2.92 MB/sec warmup 9 sec latency 3419.166 ms oleg305-client.virtnet: 1 1758 2.63 MB/sec warmup 10 sec latency 4419.371 ms oleg305-client.virtnet: 1 1758 2.39 MB/sec warmup 11 sec latency 5419.641 ms oleg305-client.virtnet: 1 1758 2.19 MB/sec warmup 12 sec latency 6419.880 ms oleg305-client.virtnet: 1 1758 2.02 MB/sec warmup 13 sec latency 7420.086 ms oleg305-client.virtnet: 1 1758 1.88 MB/sec warmup 14 sec latency 8420.309 ms oleg305-client.virtnet: 1 1758 1.75 MB/sec warmup 15 sec latency 9420.523 ms oleg305-client.virtnet: 1 1758 1.64 MB/sec warmup 16 sec latency 10420.785 ms oleg305-client.virtnet: 1 1758 1.55 MB/sec warmup 17 sec latency 11421.042 ms oleg305-client.virtnet: 1 1758 1.46 MB/sec warmup 18 sec latency 12421.279 ms oleg305-client.virtnet: 1 1758 1.38 MB/sec warmup 19 sec latency 13421.518 ms oleg305-client.virtnet: 1 1758 1.31 MB/sec warmup 20 sec latency 14421.718 ms oleg305-client.virtnet: 1 1758 1.25 MB/sec warmup 21 sec latency 15421.969 ms oleg305-client.virtnet: 1 1758 1.19 MB/sec warmup 22 sec latency 16422.169 ms oleg305-client.virtnet: 1 1758 1.14 MB/sec warmup 23 sec latency 17422.360 ms oleg305-client.virtnet: 1 1758 1.09 MB/sec warmup 24 sec latency 18422.585 ms oleg305-client.virtnet: 1 1758 1.05 MB/sec warmup 25 sec latency 19422.848 ms oleg305-client.virtnet: 1 1838 1.01 MB/sec warmup 26 sec latency 20155.364 ms oleg305-client.virtnet: 1 2180 0.99 MB/sec warmup 27 sec latency 47.523 ms oleg305-client.virtnet: 1 2485 1.03 MB/sec warmup 28 sec latency 40.658 ms oleg305-client.virtnet: 1 3012 1.16 MB/sec warmup 29 sec latency 36.640 ms oleg305-client.virtnet: 1 3628 1.29 MB/sec warmup 30 sec latency 42.258 ms oleg305-client.virtnet: 1 3918 1.29 MB/sec warmup 31 sec latency 15.702 ms oleg305-client.virtnet: 1 4212 1.26 MB/sec warmup 32 sec latency 15.348 test_70b fail mds1 2 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:54:41 (1744538081) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:54:55 (1744538095) targets are mounted 05:54:55 (1744538095) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3718144 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7454720 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:55:06 (1744538106) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:55:20 (1744538120) targets are mounted 05:55:20 (1744538120) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec ms oleg305-client.virtnet: 1 4438 1.23 MB/sec warmup 33 sec latency 50.576 ms oleg305-client.virtnet: 1 4562 1.21 MB/sec warmup 34 sec latency 401.950 ms oleg305-client.virtnet: 1 4562 1.17 MB/sec warmup 35 sec latency 1402.156 ms oleg305-client.virtnet: 1 4562 1.14 MB/sec warmup 36 sec latency 2402.422 ms oleg305-client.virtnet: 1 4562 1.11 MB/sec warmup 37 sec latency 3402.651 ms oleg305-client.virtnet: 1 4562 1.08 MB/sec warmup 38 sec latency 4402.941 ms oleg305-client.virtnet: 1 4562 1.05 MB/sec warmup 39 sec latency 5403.224 ms oleg305-client.virtnet: 1 4562 1.03 MB/sec warmup 40 sec latency 6403.494 ms oleg305-client.virtnet: 1 4562 1.00 MB/sec warmup 41 sec latency 7403.752 ms oleg305-client.virtnet: 1 4562 0.98 MB/sec warmup 42 sec latency 8404.001 ms oleg305-client.virtnet: 1 4562 0.95 MB/sec warmup 43 sec latency 9404.256 ms oleg305-client.virtnet: 1 4562 0.93 MB/sec warmup 44 sec latency 10404.467 ms oleg305-client.virtnet: 1 4562 0.91 MB/sec warmup 45 sec latency 11404.680 ms oleg305-client.virtnet: 1 4562 0.89 MB/sec warmup 46 sec latency 12404.947 ms oleg305-client.virtnet: 1 4562 0.87 MB/sec warmup 47 sec latency 13405.289 ms oleg305-client.virtnet: 1 4562 0.85 MB/sec warmup 48 sec latency 14405.517 ms oleg305-client.virtnet: 1 4562 0.84 MB/sec warmup 49 sec latency 15405.676 ms oleg305-client.virtnet: 1 4562 0.82 MB/sec warmup 50 sec latency 16405.876 ms oleg305-client.virtnet: 1 4667 0.82 MB/sec warmup 51 sec latency 17143.670 ms oleg305-client.virtnet: 1 5047 0.87 MB/sec warmup 52 sec latency 17.825 ms oleg305-client.virtnet: 1 5294 0.86 MB/sec warmup 53 sec latency 50.804 ms oleg305-client.virtnet: 1 5589 0.85 MB/sec warmup 54 sec latency 37.863 ms oleg305-client.virtnet: 1 5955 0.86 MB/sec warmup 55 sec latency 40.309 ms oleg305-client.virtnet: 1 6432 0.92 MB/sec warmup 56 sec latency 54.772 ms oleg305-client.virtnet: 1 6920 0.96 MB/sec warmup 57 sec latency 39.664 ms oleg305-client.virtnet: 1 7328 1.02 MB/sec warmup 58 sec latency 16.304 ms oleg305-client.virtnet: 1 7464 1.00 MB/sec warmup 59 sec latency 426.959 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 1 sec latency 2427.295 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 2 sec latency 3427.495 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 3 sec latency 4427.747 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 4 sec latency 5427.964 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 5 sec latency 6428.163 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 6 sec latency 7428.354 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 7 sec latency 8428.560 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 8 sec latency 9428.794 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 9 sec latency 10428.992 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 10 sec latency 11429.147 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 11 sec latency 12429.344 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 12 sec latency 13429.520 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 13 sec latency 14429.734 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 14 sec latency 15429.912 ms oleg305-client.virtnet: 1 7464 0.00 MB/sec execute 15 sec latency 16430.103 ms oleg305-client.virtnet: 1 7514 0.00 MB/sec execute 16 sec latency 17238.325 ms oleg305-client.virtnet: 1 7778 0.01 MB/sec execute 17 sec latency 34.749 ms oleg305-client.virtnet: 1 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3724288 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7471104 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:55:31 (1744538131) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:55:45 (1744538145) targets are mounted 05:55:45 (1744538145) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3739648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3702784 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7442432 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:55:56 (1744538156) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 7988 0.04 MB/sec execute 18 sec latency 57.940 ms oleg305-client.virtnet: 1 8257 0.11 MB/sec execute 19 sec latency 40.230 ms oleg305-client.virtnet: 1 8613 0.26 MB/sec execute 20 sec latency 45.795 ms oleg305-client.virtnet: 1 8883 0.25 MB/sec execute 21 sec latency 48.102 ms oleg305-client.virtnet: 1 9194 0.26 MB/sec execute 22 sec latency 36.989 ms oleg305-client.virtnet: 1 9520 0.31 MB/sec execute 23 sec latency 41.887 ms oleg305-client.virtnet: 1 9757 0.38 MB/sec execute 24 sec latency 348.767 ms oleg305-client.virtnet: 1 9757 0.36 MB/sec execute 25 sec latency 1348.965 ms oleg305-client.virtnet: 1 9757 0.35 MB/sec execute 26 sec latency 2349.171 ms oleg305-client.virtnet: 1 9757 0.34 MB/sec execute 27 sec latency 3349.403 ms oleg305-client.virtnet: 1 9757 0.33 MB/sec execute 28 sec latency 4349.674 ms oleg305-client.virtnet: 1 9757 0.31 MB/sec execute 29 sec latency 5349.911 ms oleg305-client.virtnet: 1 9757 0.30 MB/sec execute 30 sec latency 6350.120 ms oleg305-client.virtnet: 1 9757 0.29 MB/sec execute 31 sec latency 7350.320 ms oleg305-client.virtnet: 1 9757 0.28 MB/sec execute 32 sec latency 8350.522 ms oleg305-client.virtnet: 1 9757 0.28 MB/sec execute 33 sec latency 9350.835 ms oleg305-client.virtnet: 1 9757 0.27 MB/sec execute 34 sec latency 10351.093 ms oleg305-client.virtnet: 1 9757 0.26 MB/sec execute 35 sec latency 11351.282 ms oleg305-client.virtnet: 1 9757 0.25 MB/sec execute 36 sec latency 12351.471 ms oleg305-client.virtnet: 1 9757 0.25 MB/sec execute 37 sec latency 13351.697 ms oleg305-client.virtnet: 1 9757 0.24 MB/sec execute 38 sec latency 14351.893 ms oleg305-client.virtnet: 1 9757 0.23 MB/sec execute 39 sec latency 15352.165 ms oleg305-client.virtnet: 1 9757 0.23 MB/sec execute 40 sec latency 16352.400 ms oleg305-client.virtnet: 1 10001 0.29 MB/sec execute 41 sec latency 17083.057 ms oleg305-client.virtnet: 1 10480 0.35 MB/sec execute 42 sec latency 45.574 ms oleg305-client.virtnet: 1 10885 0.44 MB/sec execute 43 sec latency 15.479 ms oleg305-client.virtnet: 1 11124 0.43 MB/sec execute 44 sec latency 15.335 ms oleg305-client.virtnet: 1 11386 0.43 MB/sec execute 45 sec latency 58.076 ms oleg305-client.virtnet: 1 11641 0.44 MB/sec execute 46 sec latency 35.802 ms oleg305-client.virtnet: 1 12049 0.51 MB/sec execute 47 sec latency 18.034 ms oleg305-client.virtnet: 1 12315 0.51 MB/sec execute 48 sec latency 39.620 ms oleg305-client.virtnet: 1 12442 0.50 MB/sec execute 49 sec latency 555.760 ms oleg305-client.virtnet: 1 12442 0.49 MB/sec execute 50 sec latency 1555.982 ms oleg305-client.virtnet: 1 12442 0.48 MB/sec execute 51 sec latency 2556.285 ms oleg305-client.virtnet: 1 12442 0.47 MB/sec execute 52 sec latency 3556.534 ms oleg305-client.virtnet: 1 12442 0.46 MB/sec execute 53 sec latency 4556.802 ms oleg305-client.virtnet: 1 12442 0.45 MB/sec execute 54 sec latency 5556.973 ms oleg305-client.virtnet: 1 12442 0.44 MB/sec execute 55 sec latency 6557.199 ms oleg305-client.virtnet: 1 12442 0.44 MB/sec execute 56 sec latency 7557.444 ms oleg305-client.virtnet: 1 12442 0.43 MB/sec execute 57 sec latency 8557.790 ms oleg305-client.virtnet: 1 12442 0.42 MB/sec execute 58 sec latency 9558.051 ms oleg305-client.virtnet: 1 12442 0.41 MB/sec execute 59 sec latency 10558.312 ms oleg305-client.virtnet: 1 12442 0.41 MB/sec execute 60 sec latency 11558.491 ms oleg305-client.virtnet: 1 12442 0.40 MB/sec execute 61 sec latency 12558.734 ms oleg305-client.virtnet: 1 12442 0.39 MB/seoleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:56:10 (1744538170) targets are mounted 05:56:10 (1744538170) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3734528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3699712 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7434240 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:56:21 (1744538181) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:56:35 (1744538195) targets are mounted 05:56:35 (1744538195) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3712000 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7447552 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:56:45 (1744538205) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server c execute 62 sec latency 13558.949 ms oleg305-client.virtnet: 1 12442 0.39 MB/sec execute 63 sec latency 14559.143 ms oleg305-client.virtnet: 1 12442 0.38 MB/sec execute 64 sec latency 15559.367 ms oleg305-client.virtnet: 1 12442 0.38 MB/sec execute 65 sec latency 16559.697 ms oleg305-client.virtnet: 1 12486 0.37 MB/sec execute 66 sec latency 17358.645 ms oleg305-client.virtnet: 1 12825 0.37 MB/sec execute 67 sec latency 45.555 ms oleg305-client.virtnet: 1 13131 0.40 MB/sec execute 68 sec latency 44.398 ms oleg305-client.virtnet: 1 13650 0.46 MB/sec execute 69 sec latency 41.000 ms oleg305-client.virtnet: 1 14221 0.52 MB/sec execute 70 sec latency 44.495 ms oleg305-client.virtnet: 1 14498 0.53 MB/sec execute 71 sec latency 16.719 ms oleg305-client.virtnet: 1 14733 0.53 MB/sec execute 72 sec latency 17.449 ms oleg305-client.virtnet: 1 14996 0.53 MB/sec execute 73 sec latency 51.668 ms oleg305-client.virtnet: 1 15139 0.53 MB/sec execute 74 sec latency 372.056 ms oleg305-client.virtnet: 1 15139 0.52 MB/sec execute 75 sec latency 1372.292 ms oleg305-client.virtnet: 1 15139 0.51 MB/sec execute 76 sec latency 2372.677 ms oleg305-client.virtnet: 1 15139 0.51 MB/sec execute 77 sec latency 3372.927 ms oleg305-client.virtnet: 1 15139 0.50 MB/sec execute 78 sec latency 4373.163 ms oleg305-client.virtnet: 1 15139 0.49 MB/sec execute 79 sec latency 5373.391 ms oleg305-client.virtnet: 1 15139 0.49 MB/sec execute 80 sec latency 6373.617 ms oleg305-client.virtnet: 1 15139 0.48 MB/sec execute 81 sec latency 7373.874 ms oleg305-client.virtnet: 1 15139 0.48 MB/sec execute 82 sec latency 8374.144 ms oleg305-client.virtnet: 1 15139 0.47 MB/sec execute 83 sec latency 9374.376 ms oleg305-client.virtnet: 1 15139 0.47 MB/sec execute 84 sec latency 10374.648 ms oleg305-client.virtnet: 1 15139 0.46 MB/sec execute 85 sec latency 11374.873 ms oleg305-client.virtnet: 1 15139 0.45 MB/sec execute 86 sec latency 12375.065 ms oleg305-client.virtnet: 1 15139 0.45 MB/sec execute 87 sec latency 13375.322 ms oleg305-client.virtnet: 1 15139 0.44 MB/sec execute 88 sec latency 14375.540 ms oleg305-client.virtnet: 1 15139 0.44 MB/sec execute 89 sec latency 15375.718 ms oleg305-client.virtnet: 1 15139 0.43 MB/sec execute 90 sec latency 16375.881 ms oleg305-client.virtnet: 1 15330 0.44 MB/sec execute 91 sec latency 17068.657 ms oleg305-client.virtnet: 1 15861 0.47 MB/sec execute 92 sec latency 43.763 ms oleg305-client.virtnet: 1 16207 0.47 MB/sec execute 93 sec latency 35.544 ms oleg305-client.virtnet: 1 16597 0.48 MB/sec execute 94 sec latency 37.978 ms oleg305-client.virtnet: 1 17114 0.53 MB/sec execute 95 sec latency 48.686 ms oleg305-client.virtnet: 1 17738 0.58 MB/sec execute 96 sec latency 37.841 ms oleg305-client.virtnet: 1 18007 0.59 MB/sec execute 97 sec latency 85.758 ms oleg305-client.virtnet: 1 18244 0.59 MB/sec execute 98 sec latency 156.793 ms oleg305-client.virtnet: 1 18244 0.58 MB/sec execute 99 sec latency 1157.027 ms oleg305-client.virtnet: 1 18244 0.57 MB/sec execute 100 sec latency 2157.238 ms oleg305-client.virtnet: 1 18244 0.57 MB/sec execute 101 sec latency 3157.407 ms oleg305-client.virtnet: 1 18244 0.56 MB/sec execute 102 sec latency 4157.622 ms oleg305-client.virtnet: 1 18244 0.56 MB/sec execute 103 sec latency 5157.877 ms oleg305-client.virtnet: 1 18244 0.55 MB/sec execute 104 sec latency 6158.094 ms oleg305-client.virtnet: 1 18244 0.55 MB/sec execute 105 sec latency 7158.283 ms oleg305-client.virtnet: 1 18244 0.54 MB/sec execute 106 sec Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:56:58 (1744538218) targets are mounted 05:56:58 (1744538218) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3721216 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7472128 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:57:10 (1744538230) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:57:24 (1744538244) targets are mounted 05:57:24 (1744538244) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3741696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 44032 3710976 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7452672 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:57:37 (1744538257) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server latency 8158.487 ms oleg305-client.virtnet: 1 18244 0.54 MB/sec execute 107 sec latency 9158.677 ms oleg305-client.virtnet: 1 18244 0.53 MB/sec execute 108 sec latency 10158.856 ms oleg305-client.virtnet: 1 18244 0.53 MB/sec execute 109 sec latency 11159.034 ms oleg305-client.virtnet: 1 18244 0.52 MB/sec execute 110 sec latency 12159.184 ms oleg305-client.virtnet: 1 18244 0.52 MB/sec execute 111 sec latency 13159.316 ms oleg305-client.virtnet: 1 18244 0.51 MB/sec execute 112 sec latency 14159.444 ms oleg305-client.virtnet: 1 18244 0.51 MB/sec execute 113 sec latency 15159.642 ms oleg305-client.virtnet: 1 18244 0.50 MB/sec execute 114 sec latency 16159.860 ms oleg305-client.virtnet: 1 18244 0.50 MB/sec execute 115 sec latency 17160.025 ms oleg305-client.virtnet: 1 18364 0.49 MB/sec execute 116 sec latency 17925.010 ms oleg305-client.virtnet: 1 18819 0.51 MB/sec execute 117 sec latency 30.196 ms oleg305-client.virtnet: 1 19204 0.53 MB/sec execute 118 sec latency 17.837 ms oleg305-client.virtnet: 1 19463 0.53 MB/sec execute 119 sec latency 44.692 ms oleg305-client.virtnet: 1 19914 0.53 MB/sec execute 120 sec latency 33.166 ms oleg305-client.virtnet: 1 20225 0.53 MB/sec execute 121 sec latency 41.342 ms oleg305-client.virtnet: 1 20751 0.57 MB/sec execute 122 sec latency 33.765 ms oleg305-client.virtnet: 1 21158 0.58 MB/sec execute 123 sec latency 316.118 ms oleg305-client.virtnet: 1 21158 0.58 MB/sec execute 124 sec latency 1316.365 ms oleg305-client.virtnet: 1 21158 0.58 MB/sec execute 125 sec latency 2316.584 ms oleg305-client.virtnet: 1 21158 0.57 MB/sec execute 126 sec latency 3316.880 ms oleg305-client.virtnet: 1 21158 0.57 MB/sec execute 127 sec latency 4317.188 ms oleg305-client.virtnet: 1 21158 0.56 MB/sec execute 128 sec latency 5317.394 ms oleg305-client.virtnet: 1 21158 0.56 MB/sec execute 129 sec latency 6317.569 ms oleg305-client.virtnet: 1 21158 0.55 MB/sec execute 130 sec latency 7317.742 ms oleg305-client.virtnet: 1 21158 0.55 MB/sec execute 131 sec latency 8318.003 ms oleg305-client.virtnet: 1 21158 0.54 MB/sec execute 132 sec latency 9318.230 ms oleg305-client.virtnet: 1 21158 0.54 MB/sec execute 133 sec latency 10318.453 ms oleg305-client.virtnet: 1 21158 0.54 MB/sec execute 134 sec latency 11318.703 ms oleg305-client.virtnet: 1 21158 0.53 MB/sec execute 135 sec latency 12318.925 ms oleg305-client.virtnet: 1 21158 0.53 MB/sec execute 136 sec latency 13319.138 ms oleg305-client.virtnet: 1 21158 0.52 MB/sec execute 137 sec latency 14319.329 ms oleg305-client.virtnet: 1 21158 0.52 MB/sec execute 138 sec latency 15319.576 ms oleg305-client.virtnet: 1 21158 0.52 MB/sec execute 139 sec latency 16319.778 ms oleg305-client.virtnet: 1 21158 0.51 MB/sec execute 140 sec latency 17320.040 ms oleg305-client.virtnet: 1 21158 0.51 MB/sec execute 141 sec latency 18320.156 ms oleg305-client.virtnet: 1 21561 0.54 MB/sec execute 142 sec latency 18330.267 ms oleg305-client.virtnet: 1 21792 0.54 MB/sec execute 143 sec latency 16.428 ms oleg305-client.virtnet: 1 22125 0.53 MB/sec execute 144 sec latency 42.730 ms oleg305-client.virtnet: 1 22329 0.53 MB/sec execute 145 sec latency 47.306 ms oleg305-client.virtnet: 1 22788 0.56 MB/sec execute 146 sec latency 17.811 ms oleg305-client.virtnet: 1 23045 0.56 MB/sec execute 147 sec latency 50.294 ms oleg305-client.virtnet: 1 23342 0.55 MB/sec execute 148 sec latency 38.958 ms oleg305-client.virtnet: 1 23684 0.56 MB/sec execute 149 sec latency 96.283 ms oleg305-client.virtnet: 1 23684 0.56 MB/sec execute 150 sec latency 109Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:57:50 (1744538270) targets are mounted 05:57:50 (1744538270) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3705856 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7451648 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:58:01 (1744538281) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:58:15 (1744538295) targets are mounted 05:58:15 (1744538295) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 6.485 ms oleg305-client.virtnet: 1 23684 0.55 MB/sec execute 151 sec latency 2096.769 ms oleg305-client.virtnet: 1 23684 0.55 MB/sec execute 152 sec latency 3097.006 ms oleg305-client.virtnet: 1 23684 0.55 MB/sec execute 153 sec latency 4097.249 ms oleg305-client.virtnet: 1 23684 0.54 MB/sec execute 154 sec latency 5097.546 ms oleg305-client.virtnet: 1 23684 0.54 MB/sec execute 155 sec latency 6097.824 ms oleg305-client.virtnet: 1 23684 0.54 MB/sec execute 156 sec latency 7098.040 ms oleg305-client.virtnet: 1 23684 0.53 MB/sec execute 157 sec latency 8098.268 ms oleg305-client.virtnet: 1 23684 0.53 MB/sec execute 158 sec latency 9098.529 ms oleg305-client.virtnet: 1 23684 0.52 MB/sec execute 159 sec latency 10098.811 ms oleg305-client.virtnet: 1 23684 0.52 MB/sec execute 160 sec latency 11099.017 ms oleg305-client.virtnet: 1 23684 0.52 MB/sec execute 161 sec latency 12099.207 ms oleg305-client.virtnet: 1 23684 0.52 MB/sec execute 162 sec latency 13099.384 ms oleg305-client.virtnet: 1 23684 0.51 MB/sec execute 163 sec latency 14099.637 ms oleg305-client.virtnet: 1 23684 0.51 MB/sec execute 164 sec latency 15099.859 ms oleg305-client.virtnet: 1 23684 0.51 MB/sec execute 165 sec latency 16100.117 ms oleg305-client.virtnet: 1 23728 0.50 MB/sec execute 166 sec latency 16869.927 ms oleg305-client.virtnet: 1 24204 0.53 MB/sec execute 167 sec latency 45.568 ms oleg305-client.virtnet: 1 24685 0.54 MB/sec execute 168 sec latency 39.969 ms oleg305-client.virtnet: 1 25082 0.56 MB/sec execute 169 sec latency 15.201 ms oleg305-client.virtnet: 1 25321 0.56 MB/sec execute 170 sec latency 15.754 ms oleg305-client.virtnet: 1 25598 0.56 MB/sec execute 171 sec latency 66.857 ms oleg305-client.virtnet: 1 25867 0.56 MB/sec execute 172 sec latency 41.251 ms oleg305-client.virtnet: 1 26272 0.58 MB/sec execute 173 sec latency 18.670 ms oleg305-client.virtnet: 1 26451 0.58 MB/sec execute 174 sec latency 367.201 ms oleg305-client.virtnet: 1 26451 0.58 MB/sec execute 175 sec latency 1367.395 ms oleg305-client.virtnet: 1 26451 0.57 MB/sec execute 176 sec latency 2367.598 ms oleg305-client.virtnet: 1 26451 0.57 MB/sec execute 177 sec latency 3367.837 ms oleg305-client.virtnet: 1 26451 0.57 MB/sec execute 178 sec latency 4368.086 ms oleg305-client.virtnet: 1 26451 0.56 MB/sec execute 179 sec latency 5368.307 ms oleg305-client.virtnet: 1 26451 0.56 MB/sec execute 180 sec latency 6368.494 ms oleg305-client.virtnet: 1 26451 0.56 MB/sec execute 181 sec latency 7368.680 ms oleg305-client.virtnet: 1 26451 0.55 MB/sec execute 182 sec latency 8368.869 ms oleg305-client.virtnet: 1 26451 0.55 MB/sec execute 183 sec latency 9369.101 ms oleg305-client.virtnet: 1 26451 0.55 MB/sec execute 184 sec latency 10369.390 ms oleg305-client.virtnet: 1 26451 0.54 MB/sec execute 185 sec latency 11369.630 ms oleg305-client.virtnet: 1 26451 0.54 MB/sec execute 186 sec latency 12369.836 ms oleg305-client.virtnet: 1 26451 0.54 MB/sec execute 187 sec latency 13370.131 ms oleg305-client.virtnet: 1 26451 0.54 MB/sec execute 188 sec latency 14370.299 ms oleg305-client.virtnet: 1 26451 0.53 MB/sec execute 189 sec latency 15370.482 ms oleg305-client.virtnet: 1 26451 0.53 MB/sec execute 190 sec latency 16370.689 ms oleg305-client.virtnet: 1 26465 0.53 MB/sec execute 191 sec latency 17287.135 ms oleg305-client.virtnet: 1 26742 0.53 MB/sec execute 192 sec latency 43.504 ms oleg305-client.virtnet: 1 27077 0.53 MB/sec execute 193 sec latency 41.963 ms oleg305-client.virtnet: 1 27372 0.53 MB/sec execute 194 sec latency 41.320 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3723264 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3704832 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7428096 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:58:26 (1744538306) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:58:40 (1744538320) targets are mounted 05:58:40 (1744538320) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3741696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3712000 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7453696 1% /mnt/lustre test_70b fail mds1 12 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 05:58:52 (1744538332) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 05:59:05 (1744538345) targets are mounted 05:59:05 (1744538345) facet_failover done oleg305-client.virtnet: 1 28060 0.56 MB/sec execute 195 sec latency 43.838 ms oleg305-client.virtnet: 1 28553 0.58 MB/sec execute 196 sec latency 19.295 ms oleg305-client.virtnet: 1 28826 0.58 MB/sec execute 197 sec latency 20.410 ms oleg305-client.virtnet: 1 29156 0.58 MB/sec execute 198 sec latency 34.605 ms oleg305-client.virtnet: 1 29304 0.58 MB/sec execute 199 sec latency 481.872 ms oleg305-client.virtnet: 1 29304 0.58 MB/sec execute 200 sec latency 1482.084 ms oleg305-client.virtnet: 1 29304 0.57 MB/sec execute 201 sec latency 2482.331 ms oleg305-client.virtnet: 1 29304 0.57 MB/sec execute 202 sec latency 3482.567 ms oleg305-client.virtnet: 1 29304 0.57 MB/sec execute 203 sec latency 4482.799 ms oleg305-client.virtnet: 1 29304 0.57 MB/sec execute 204 sec latency 5483.026 ms oleg305-client.virtnet: 1 29304 0.56 MB/sec execute 205 sec latency 6483.245 ms oleg305-client.virtnet: 1 29304 0.56 MB/sec execute 206 sec latency 7483.427 ms oleg305-client.virtnet: 1 29304 0.56 MB/sec execute 207 sec latency 8483.627 ms oleg305-client.virtnet: 1 29304 0.56 MB/sec execute 208 sec latency 9483.883 ms oleg305-client.virtnet: 1 29304 0.55 MB/sec execute 209 sec latency 10484.094 ms oleg305-client.virtnet: 1 29304 0.55 MB/sec execute 210 sec latency 11484.321 ms oleg305-client.virtnet: 1 29304 0.55 MB/sec execute 211 sec latency 12484.570 ms oleg305-client.virtnet: 1 29304 0.54 MB/sec execute 212 sec latency 13484.845 ms oleg305-client.virtnet: 1 29304 0.54 MB/sec execute 213 sec latency 14485.090 ms oleg305-client.virtnet: 1 29304 0.54 MB/sec execute 214 sec latency 15485.268 ms oleg305-client.virtnet: 1 29304 0.54 MB/sec execute 215 sec latency 16485.499 ms oleg305-client.virtnet: 1 29304 0.53 MB/sec execute 216 sec latency 17485.729 ms oleg305-client.virtnet: 1 29553 0.54 MB/sec execute 217 sec latency 17590.482 ms oleg305-client.virtnet: 1 29921 0.55 MB/sec execute 218 sec latency 39.077 ms oleg305-client.virtnet: 1 30194 0.55 MB/sec execute 219 sec latency 39.374 ms oleg305-client.virtnet: 1 30515 0.55 MB/sec execute 220 sec latency 35.409 ms oleg305-client.virtnet: 1 30860 0.55 MB/sec execute 221 sec latency 50.715 ms oleg305-client.virtnet: 1 31382 0.57 MB/sec execute 222 sec latency 39.906 ms oleg305-client.virtnet: 1 31938 0.59 MB/sec execute 223 sec latency 35.992 ms oleg305-client.virtnet: 1 32227 0.60 MB/sec execute 224 sec latency 16.303 ms oleg305-client.virtnet: 1 32244 0.59 MB/sec execute 225 sec latency 922.011 ms oleg305-client.virtnet: 1 32244 0.59 MB/sec execute 226 sec latency 1922.219 ms oleg305-client.virtnet: 1 32244 0.59 MB/sec execute 227 sec latency 2922.511 ms oleg305-client.virtnet: 1 32244 0.59 MB/sec execute 228 sec latency 3922.738 ms oleg305-client.virtnet: 1 32244 0.58 MB/sec execute 229 sec latency 4922.992 ms oleg305-client.virtnet: 1 32244 0.58 MB/sec execute 230 sec latency 5923.217 ms oleg305-client.virtnet: 1 32244 0.58 MB/sec execute 231 sec latency 6923.463 ms oleg305-client.virtnet: 1 32244 0.58 MB/sec execute 232 sec latency 7923.693 ms oleg305-client.virtnet: 1 32244 0.57 MB/sec execute 233 sec latency 8923.995 ms oleg305-client.virtnet: 1 32244 0.57 MB/sec execute 234 sec latency 9924.251 ms oleg305-client.virtnet: 1 32244 0.57 MB/sec execute 235 sec latency 10924.532 ms oleg305-client.virtnet: 1 32244 0.57 MB/sec execute 236 sec latency 11924.729 ms oleg305-client.virtnet: 1 32244 0.56 MB/sec execute 237 sec latency 12924.899 ms oleg305-client.virtnet: 1 32244 0.56 MB/sec execute 238 sec latency 13925.142 ms oleg305-client.oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec virtnet: 1 32244 0.56 MB/sec execute 239 sec latency 14925.413 ms oleg305-client.virtnet: 1 32244 0.56 MB/sec execute 240 sec latency 15925.624 ms oleg305-client.virtnet: 1 32244 0.55 MB/sec execute 241 sec latency 16925.828 ms oleg305-client.virtnet: 1 32465 0.55 MB/sec execute 242 sec latency 16991.554 ms oleg305-client.virtnet: 1 32704 0.55 MB/sec execute 243 sec latency 57.339 ms oleg305-client.virtnet: 1 32914 0.55 MB/sec execute 244 sec latency 44.554 ms oleg305-client.virtnet: 1 33217 0.56 MB/sec execute 245 sec latency 43.698 ms oleg305-client.virtnet: 1 33563 0.57 MB/sec execute 246 sec latency 39.193 ms oleg305-client.virtnet: 1 33834 0.56 MB/sec execute 247 sec latency 36.243 ms oleg305-client.virtnet: 1 34159 0.56 MB/sec execute 248 sec latency 36.438 ms oleg305-client.virtnet: 1 34462 0.57 MB/sec execute 249 sec latency 38.240 ms oleg305-client.virtnet: 1 34974 0.59 MB/sec execute 250 sec latency 39.358 ms oleg305-client.virtnet: 1 35571 0.60 MB/sec execute 251 sec latency 33.455 ms oleg305-client.virtnet: 1 35848 0.61 MB/sec execute 252 sec latency 15.735 ms oleg305-client.virtnet: 1 36128 0.60 MB/sec execute 253 sec latency 16.891 ms oleg305-client.virtnet: 1 36339 0.60 MB/sec execute 254 sec latency 64.097 ms oleg305-client.virtnet: 1 36635 0.61 MB/sec execute 255 sec latency 43.120 ms oleg305-client.virtnet: 1 37001 0.62 MB/sec execute 256 sec latency 36.443 ms oleg305-client.virtnet: 1 37276 0.62 MB/sec execute 257 sec latency 36.576 ms oleg305-client.virtnet: 1 37579 0.62 MB/sec execute 258 sec latency 36.294 ms oleg305-client.virtnet: 1 37925 0.62 MB/sec execute 259 sec latency 37.892 ms oleg305-client.virtnet: 1 38426 0.64 MB/sec execute 260 sec latency 38.576 ms oleg305-client.virtnet: 1 39016 0.65 MB/sec execute 261 sec latency 33.937 ms oleg305-client.virtnet: 1 39317 0.66 MB/sec execute 262 sec latency 16.675 ms oleg305-client.virtnet: 1 39550 0.65 MB/sec execute 263 sec latency 16.202 ms oleg305-client.virtnet: 1 39836 0.65 MB/sec execute 264 sec latency 51.522 ms oleg305-client.virtnet: 1 40050 0.65 MB/sec execute 265 sec latency 38.438 ms oleg305-client.virtnet: 1 40461 0.67 MB/sec execute 266 sec latency 18.644 ms oleg305-client.virtnet: 1 40724 0.66 MB/sec execute 267 sec latency 39.196 ms oleg305-client.virtnet: 1 41036 0.66 MB/sec execute 268 sec latency 32.399 ms oleg305-client.virtnet: 1 41404 0.67 MB/sec execute 269 sec latency 32.468 ms oleg305-client.virtnet: 1 41773 0.68 MB/sec execute 270 sec latency 37.039 ms oleg305-client.virtnet: 1 42330 0.69 MB/sec execute 271 sec latency 37.037 ms oleg305-client.virtnet: 1 42757 0.70 MB/sec execute 272 sec latency 18.512 ms oleg305-client.virtnet: 1 43018 0.70 MB/sec execute 273 sec latency 16.107 ms oleg305-client.virtnet: 1 43304 0.70 MB/sec execute 274 sec latency 55.105 ms oleg305-client.virtnet: 1 43522 0.70 MB/sec execute 275 sec latency 39.458 ms oleg305-client.virtnet: 1 43835 0.70 MB/sec execute 276 sec latency 38.119 ms oleg305-client.virtnet: 1 44188 0.71 MB/sec execute 277 sec latency 48.348 ms oleg305-client.virtnet: 1 44464 0.71 MB/sec execute 278 sec latency 41.906 ms oleg305-client.virtnet: 1 44858 0.71 MB/sec execute 279 sec latency 35.274 ms oleg305-client.virtnet: 1 45179 0.71 MB/sec execute 280 sec latency 39.375 ms oleg305-client.virtnet: 1 45871 0.73 MB/sec execute 281 sec latency 38.594 ms oleg305-client.virtnet: 1 46289 0.74 MB/sec execute 282 sec latency 20.427 ms oleg305-client.virtnet: 1 46558 0.74 MB/sec execute 283 sec latency 16.285 ms oleg305-client.virtnet: 1 46848 0.74 MB/sec execute 284 sec latency 50.615 ms oleg305-client.virtnet: 1 47072 0.74 MB/sec execute 285 sec latency 38.970 ms oleg305-client.virtnet: 1 47386 0.74 MB/sec execute 286 sec latency 39.166 ms oleg305-client.virtnet: 1 47740 0.75 MB/sec execute 287 sec latency 49.650 ms oleg305-client.virtnet: 1 48014 0.75 MB/sec execute 288 sec latency 36.472 ms oleg305-client.virtnet: 1 48350 0.75 MB/sec execute 289 sec latency 34.960 ms oleg305-client.virtnet: 1 48654 0.75 MB/sec execute 290 sec latency 40.732 ms oleg305-client.virtnet: 1 49158 0.77 MB/sec execute 291 sec latency 39.977 ms oleg305-client.virtnet: 1 49809 0.78 MB/sec execute 292 sec latency 33.637 ms oleg305-client.virtnet: 1 50079 0.78 MB/sec execute 293 sec latency 16.117 ms oleg305-client.virtnet: 1 50367 0.78 MB/sec execute 294 sec latency 59.184 ms oleg305-client.virtnet: 1 50584 0.78 MB/sec execute 295 sec latency 38.084 ms oleg305-client.virtnet: 1 50874 0.78 MB/sec execute 296 sec latency 34.887 ms oleg305-client.virtnet: 1 51246 0.79 MB/sec execute 297 sec latency 39.861 ms oleg305-client.virtnet: 1 51501 0.79 MB/sec execute 298 sec latency 44.154 ms oleg305-client.virtnet: 1 51846 0.79 MB/sec execute 299 sec latency 36.359 ms oleg305-client.virtnet: 1 cleanup 300 sec oleg305-client.virtnet: 0 cleanup 300 sec oleg305-client.virtnet: oleg305-client.virtnet: Operation Count AvgLat MaxLat oleg305-client.virtnet: ---------------------------------------- oleg305-client.virtnet: NTCreateX 7786 21.972 18330.255 oleg305-client.virtnet: Close 5731 4.806 17590.444 oleg305-client.virtnet: Rename 326 65.084 16869.893 oleg305-client.virtnet: Unlink 1569 2.947 4.917 oleg305-client.virtnet: Qpathinfo 7069 7.201 17068.643 oleg305-client.virtnet: Qfileinfo 1225 0.203 2.427 oleg305-client.virtnet: Qfsinfo 1268 0.178 7.159 oleg305-client.virtnet: Sfileinfo 644 7.711 15.107 oleg305-client.virtnet: Find 2706 0.549 13.089 oleg305-client.virtnet: WriteX 3828 1.162 6.089 oleg305-client.virtnet: ReadX 11946 0.035 7.046 oleg305-client.virtnet: LockX 24 1.577 2.006 oleg305-client.virtnet: UnlockX 24 1.713 2.118 oleg305-client.virtnet: Flush 549 25.013 66.842 oleg305-client.virtnet: oleg305-client.virtnet: Throughput 0.786645 MB/sec 1 clients 1 procs max_latency=18330.267 ms oleg305-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg305-client.virtnet at Sun Apr 13 06:00:07 EDT 2025 with return code 0 oleg305-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg305-client.virtnet oleg305-client.virtnet: /mnt/lustre/d70b.replay-single/oleg305-client.virtnet /mnt/lustre/d70b.replay-single/oleg305-client.virtnet oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg305-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg305-client.virtnet: removed directory: 'clients/client0' oleg305-client.virtnet: removed directory: 'clients' oleg305-client.virtnet: removed 'client.txt' oleg305-client.virtnet: /mnt/lustre/d70b.replay-single/oleg305-client.virtnet oleg305-client.virtnet: dbench successfully finished