== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 17:18:58 (1741900738) Starting client oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre Started clients oleg423-client.virtnet: 192.168.204.123@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg423-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=5943 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3665920 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3719168 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 9216 7385088 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:19:04 (1741900744) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:19:17 (1741900757) targets are mounted 17:19:17 (1741900757) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 32768 3660800 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 46080 7396352 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server oleg423-client.virtnet: looking for dbench program oleg423-client.virtnet: /usr/bin/dbench oleg423-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg423-client.virtnet oleg423-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg423-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg423-client.virtnet' oleg423-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg423-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg423-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg423-client.virtnet at Thu Mar 13 17:18:59 EDT 2025 oleg423-client.virtnet: waiting for dbench pid 5967 oleg423-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg423-client.virtnet: oleg423-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg423-client.virtnet: failed to create barrier semaphore oleg423-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg423-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg423-client.virtnet: releasing clients oleg423-client.virtnet: 1 612 20.97 MB/sec warmup 1 sec latency 17.764 ms oleg423-client.virtnet: 1 1063 11.45 MB/sec warmup 2 sec latency 50.169 ms oleg423-client.virtnet: 1 1717 8.75 MB/sec warmup 3 sec latency 29.816 ms oleg423-client.virtnet: 1 2388 6.99 MB/sec warmup 4 sec latency 17.940 ms oleg423-client.virtnet: 1 3024 6.71 MB/sec warmup 5 sec latency 221.243 ms oleg423-client.virtnet: 1 3024 5.59 MB/sec warmup 6 sec latency 1221.411 ms oleg423-client.virtnet: 1 3024 4.79 MB/sec warmup 7 sec latency 2221.559 ms oleg423-client.virtnet: 1 3024 4.19 MB/sec warmup 8 sec latency 3221.724 ms oleg423-client.virtnet: 1 3024 3.73 MB/sec warmup 9 sec latency 4221.936 ms oleg423-client.virtnet: 1 3024 3.35 MB/sec warmup 10 sec latency 5222.073 ms oleg423-client.virtnet: 1 3024 3.05 MB/sec warmup 11 sec latency 6222.243 ms oleg423-client.virtnet: 1 3024 2.79 MB/sec warmup 12 sec latency 7222.404 ms oleg423-client.virtnet: 1 3024 2.58 MB/sec warmup 13 sec latency 8222.601 ms oleg423-client.virtnet: 1 3024 2.40 MB/sec warmup 14 sec latency 9222.823 ms oleg423-client.virtnet: 1 3024 2.24 MB/sec warmup 15 sec latency 10223.201 ms oleg423-client.virtnet: 1 3024 2.10 MB/sec warmup 16 sec latency 11223.331 ms oleg423-client.virtnet: 1 3024 1.97 MB/sec warmup 17 sec latency 12223.440 ms oleg423-client.virtnet: 1 3024 1.86 MB/sec warmup 18 sec latency 13223.625 ms oleg423-client.virtnet: 1 3024 1.76 MB/sec warmup 19 sec latency 14223.807 ms oleg423-client.virtnet: 1 3024 1.68 MB/sec warmup 20 sec latency 15223.987 ms oleg423-client.virtnet: 1 3024 1.60 MB/sec warmup 21 sec latency 16224.164 ms oleg423-client.virtnet: 1 3024 1.52 MB/sec warmup 22 sec latency 17224.286 ms oleg423-client.virtnet: 1 3024 1.46 MB/sec warmup 23 sec latency 18224.444 ms oleg423-client.virtnet: 1 3024 1.40 MB/sec warmup 24 sec latency 19224.664 ms oleg423-client.virtnet: 1 3024 1.34 MB/sec warmup 25 sec latency 20224.770 ms oleg423-client.virtnet: 1 3343 1.36 MB/sec warmup 26 sec latency 20954.043 ms oleg423-client.virtnet: 1 3982 1.48 MB/sec warmup 27 sec latency 10.915 ms oleg423-client.virtnet: 1 4452 1.46 MB/sec warmup 28 sec latency 34.024 ms oleg423-client.virtnet: 1 5067 1.56 MB/sec warmup 29 sec latency 28.423 ms oleg423-client.virtnet: 1 5575 1.52 MB/sec warmup 30 sec latency 25.202 ms oleg423-client.virtnet: 1 6167 1.55 MB/sec warmup 31 sec latency 28.528 ms oleg423-client.virtnet: 1 7134 1.80 MB/sec warmup 32 sec latency 30.917:19:32 (1741900772) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:19:44 (1741900784) targets are mounted 17:19:44 (1741900784) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3710976 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3719168 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7430144 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:19:57 (1741900797) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:20:09 (1741900809) targets are mounted 17:20:09 (1741900809) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 85 ms oleg423-client.virtnet: 1 7240 1.75 MB/sec warmup 33 sec latency 825.025 ms oleg423-client.virtnet: 1 7240 1.70 MB/sec warmup 34 sec latency 1825.215 ms oleg423-client.virtnet: 1 7240 1.65 MB/sec warmup 35 sec latency 2825.410 ms oleg423-client.virtnet: 1 7240 1.61 MB/sec warmup 36 sec latency 3825.582 ms oleg423-client.virtnet: 1 7240 1.56 MB/sec warmup 37 sec latency 4825.774 ms oleg423-client.virtnet: 1 7240 1.52 MB/sec warmup 38 sec latency 5825.953 ms oleg423-client.virtnet: 1 7240 1.48 MB/sec warmup 39 sec latency 6826.104 ms oleg423-client.virtnet: 1 7240 1.45 MB/sec warmup 40 sec latency 7826.263 ms oleg423-client.virtnet: 1 7240 1.41 MB/sec warmup 41 sec latency 8826.436 ms oleg423-client.virtnet: 1 7240 1.38 MB/sec warmup 42 sec latency 9826.651 ms oleg423-client.virtnet: 1 7240 1.35 MB/sec warmup 43 sec latency 10826.765 ms oleg423-client.virtnet: 1 7240 1.32 MB/sec warmup 44 sec latency 11826.948 ms oleg423-client.virtnet: 1 7240 1.29 MB/sec warmup 45 sec latency 12827.086 ms oleg423-client.virtnet: 1 7240 1.26 MB/sec warmup 46 sec latency 13827.262 ms oleg423-client.virtnet: 1 7240 1.23 MB/sec warmup 47 sec latency 14827.454 ms oleg423-client.virtnet: 1 7240 1.21 MB/sec warmup 48 sec latency 15827.672 ms oleg423-client.virtnet: 1 7240 1.18 MB/sec warmup 49 sec latency 16827.875 ms oleg423-client.virtnet: 1 7240 1.16 MB/sec warmup 50 sec latency 17828.042 ms oleg423-client.virtnet: 1 7345 1.15 MB/sec warmup 51 sec latency 18623.406 ms oleg423-client.virtnet: 1 7781 1.14 MB/sec warmup 52 sec latency 39.002 ms oleg423-client.virtnet: 1 8236 1.16 MB/sec warmup 53 sec latency 23.394 ms oleg423-client.virtnet: 1 8835 1.19 MB/sec warmup 54 sec latency 29.613 ms oleg423-client.virtnet: 1 9471 1.20 MB/sec warmup 55 sec latency 26.803 ms oleg423-client.virtnet: 1 10308 1.29 MB/sec warmup 56 sec latency 24.690 ms oleg423-client.virtnet: 1 11010 1.37 MB/sec warmup 57 sec latency 10.447 ms oleg423-client.virtnet: 1 11089 1.35 MB/sec warmup 58 sec latency 814.499 ms oleg423-client.virtnet: 1 11089 1.33 MB/sec warmup 59 sec latency 1814.699 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 1 sec latency 3814.989 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 2 sec latency 4815.149 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 3 sec latency 5815.309 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 4 sec latency 6815.490 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 5 sec latency 7815.700 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 6 sec latency 8815.918 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 7 sec latency 9816.131 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 8 sec latency 10816.287 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 9 sec latency 11816.490 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 10 sec latency 12816.651 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 11 sec latency 13816.845 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 12 sec latency 14817.046 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 13 sec latency 15817.220 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 14 sec latency 16817.392 ms oleg423-client.virtnet: 1 11089 0.00 MB/sec execute 15 sec latency 17817.576 ms oleg423-client.virtnet: 1 11142 0.00 MB/sec execute 16 sec latency 18683.983 ms oleg423-client.virtnet: 1 11575 0.05 MB/sec execute 17 sec latency 30.924 ms oleg423-client.virtnet:UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3712000 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3747840 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7459840 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:20:22 (1741900822) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:20:34 (1741900834) targets are mounted 17:20:34 (1741900834) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3709952 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7455744 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:20:47 (1741900847) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:20:59 (1741900859) targets are mounted 17:20:59 (1741900859) facet_failover done 1 12209 0.29 MB/sec execute 18 sec latency 27.599 ms oleg423-client.virtnet: 1 12718 0.30 MB/sec execute 19 sec latency 27.994 ms oleg423-client.virtnet: 1 13531 0.56 MB/sec execute 20 sec latency 24.772 ms oleg423-client.virtnet: 1 14407 0.90 MB/sec execute 21 sec latency 21.328 ms oleg423-client.virtnet: 1 14886 0.88 MB/sec execute 22 sec latency 33.090 ms oleg423-client.virtnet: 1 14947 0.85 MB/sec execute 23 sec latency 818.337 ms oleg423-client.virtnet: 1 14947 0.81 MB/sec execute 24 sec latency 1818.510 ms oleg423-client.virtnet: 1 14947 0.78 MB/sec execute 25 sec latency 2818.689 ms oleg423-client.virtnet: 1 14947 0.75 MB/sec execute 26 sec latency 3818.868 ms oleg423-client.virtnet: 1 14947 0.72 MB/sec execute 27 sec latency 4819.036 ms oleg423-client.virtnet: 1 14947 0.70 MB/sec execute 28 sec latency 5819.176 ms oleg423-client.virtnet: 1 14947 0.67 MB/sec execute 29 sec latency 6819.384 ms oleg423-client.virtnet: 1 14947 0.65 MB/sec execute 30 sec latency 7819.516 ms oleg423-client.virtnet: 1 14947 0.63 MB/sec execute 31 sec latency 8819.699 ms oleg423-client.virtnet: 1 14947 0.61 MB/sec execute 32 sec latency 9819.850 ms oleg423-client.virtnet: 1 14947 0.59 MB/sec execute 33 sec latency 10819.981 ms oleg423-client.virtnet: 1 14947 0.57 MB/sec execute 34 sec latency 11820.199 ms oleg423-client.virtnet: 1 14947 0.56 MB/sec execute 35 sec latency 12820.325 ms oleg423-client.virtnet: 1 14947 0.54 MB/sec execute 36 sec latency 13820.486 ms oleg423-client.virtnet: 1 14947 0.53 MB/sec execute 37 sec latency 14820.673 ms oleg423-client.virtnet: 1 14947 0.51 MB/sec execute 38 sec latency 15820.907 ms oleg423-client.virtnet: 1 14947 0.50 MB/sec execute 39 sec latency 16821.123 ms oleg423-client.virtnet: 1 14947 0.49 MB/sec execute 40 sec latency 17821.241 ms oleg423-client.virtnet: 1 14996 0.48 MB/sec execute 41 sec latency 18719.632 ms oleg423-client.virtnet: 1 15586 0.58 MB/sec execute 42 sec latency 22.389 ms oleg423-client.virtnet: 1 16108 0.57 MB/sec execute 43 sec latency 32.545 ms oleg423-client.virtnet: 1 16772 0.62 MB/sec execute 44 sec latency 26.206 ms oleg423-client.virtnet: 1 17818 0.82 MB/sec execute 45 sec latency 21.099 ms oleg423-client.virtnet: 1 18327 0.83 MB/sec execute 46 sec latency 9.112 ms oleg423-client.virtnet: 1 18738 0.83 MB/sec execute 47 sec latency 31.755 ms oleg423-client.virtnet: 1 18893 0.84 MB/sec execute 48 sec latency 773.370 ms oleg423-client.virtnet: 1 18893 0.82 MB/sec execute 49 sec latency 1773.558 ms oleg423-client.virtnet: 1 18893 0.81 MB/sec execute 50 sec latency 2773.680 ms oleg423-client.virtnet: 1 18893 0.79 MB/sec execute 51 sec latency 3773.819 ms oleg423-client.virtnet: 1 18893 0.77 MB/sec execute 52 sec latency 4774.023 ms oleg423-client.virtnet: 1 18893 0.76 MB/sec execute 53 sec latency 5774.209 ms oleg423-client.virtnet: 1 18893 0.75 MB/sec execute 54 sec latency 6774.384 ms oleg423-client.virtnet: 1 18893 0.73 MB/sec execute 55 sec latency 7774.509 ms oleg423-client.virtnet: 1 18893 0.72 MB/sec execute 56 sec latency 8774.678 ms oleg423-client.virtnet: 1 18893 0.71 MB/sec execute 57 sec latency 9774.842 ms oleg423-client.virtnet: 1 18893 0.69 MB/sec execute 58 sec latency 10774.974 ms oleg423-client.virtnet: 1 18893 0.68 MB/sec execute 59 sec latency 11775.116 ms oleg423-client.virtnet: 1 18893 0.67 MB/sec execute 60 sec latency 12775.221 ms oleg423-client.virtnet: 1 18893 0.66 MB/sec execute 61 sec latency 13775.383 ms oleg423-client.virtnet: 1 18893 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3699712 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3741696 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7441408 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:21:12 (1741900872) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:21:24 (1741900884) targets are mounted 17:21:24 (1741900884) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3701760 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3721216 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7422976 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:21:37 (1741900897) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server 0.65 MB/sec execute 62 sec latency 14775.536 ms oleg423-client.virtnet: 1 18893 0.64 MB/sec execute 63 sec latency 15775.675 ms oleg423-client.virtnet: 1 18893 0.63 MB/sec execute 64 sec latency 16775.854 ms oleg423-client.virtnet: 1 18893 0.62 MB/sec execute 65 sec latency 17775.996 ms oleg423-client.virtnet: 1 18928 0.61 MB/sec execute 66 sec latency 18694.892 ms oleg423-client.virtnet: 1 19531 0.65 MB/sec execute 67 sec latency 28.269 ms oleg423-client.virtnet: 1 20161 0.67 MB/sec execute 68 sec latency 27.429 ms oleg423-client.virtnet: 1 20955 0.75 MB/sec execute 69 sec latency 29.224 ms oleg423-client.virtnet: 1 21647 0.82 MB/sec execute 70 sec latency 20.219 ms oleg423-client.virtnet: 1 22130 0.82 MB/sec execute 71 sec latency 23.792 ms oleg423-client.virtnet: 1 22715 0.87 MB/sec execute 72 sec latency 23.129 ms oleg423-client.virtnet: 1 22854 0.86 MB/sec execute 73 sec latency 722.094 ms oleg423-client.virtnet: 1 22854 0.85 MB/sec execute 74 sec latency 1722.259 ms oleg423-client.virtnet: 1 22854 0.83 MB/sec execute 75 sec latency 2722.376 ms oleg423-client.virtnet: 1 22854 0.82 MB/sec execute 76 sec latency 3722.578 ms oleg423-client.virtnet: 1 22854 0.81 MB/sec execute 77 sec latency 4722.781 ms oleg423-client.virtnet: 1 22854 0.80 MB/sec execute 78 sec latency 5722.951 ms oleg423-client.virtnet: 1 22854 0.79 MB/sec execute 79 sec latency 6723.098 ms oleg423-client.virtnet: 1 22854 0.78 MB/sec execute 80 sec latency 7723.221 ms oleg423-client.virtnet: 1 22854 0.77 MB/sec execute 81 sec latency 8723.388 ms oleg423-client.virtnet: 1 22854 0.76 MB/sec execute 82 sec latency 9723.601 ms oleg423-client.virtnet: 1 22854 0.75 MB/sec execute 83 sec latency 10723.720 ms oleg423-client.virtnet: 1 22854 0.74 MB/sec execute 84 sec latency 11723.881 ms oleg423-client.virtnet: 1 22854 0.74 MB/sec execute 85 sec latency 12723.996 ms oleg423-client.virtnet: 1 22854 0.73 MB/sec execute 86 sec latency 13724.176 ms oleg423-client.virtnet: 1 22854 0.72 MB/sec execute 87 sec latency 14724.349 ms oleg423-client.virtnet: 1 22854 0.71 MB/sec execute 88 sec latency 15724.583 ms oleg423-client.virtnet: 1 22854 0.70 MB/sec execute 89 sec latency 16724.782 ms oleg423-client.virtnet: 1 22854 0.70 MB/sec execute 90 sec latency 17724.960 ms oleg423-client.virtnet: 1 22857 0.69 MB/sec execute 91 sec latency 18694.481 ms oleg423-client.virtnet: 1 23218 0.68 MB/sec execute 92 sec latency 27.332 ms oleg423-client.virtnet: 1 23726 0.69 MB/sec execute 93 sec latency 34.472 ms oleg423-client.virtnet: 1 24494 0.75 MB/sec execute 94 sec latency 40.225 ms oleg423-client.virtnet: 1 25176 0.80 MB/sec execute 95 sec latency 24.761 ms oleg423-client.virtnet: 1 25655 0.80 MB/sec execute 96 sec latency 31.739 ms oleg423-client.virtnet: 1 26144 0.81 MB/sec execute 97 sec latency 43.881 ms oleg423-client.virtnet: 1 26391 0.83 MB/sec execute 98 sec latency 696.815 ms oleg423-client.virtnet: 1 26391 0.82 MB/sec execute 99 sec latency 1696.977 ms oleg423-client.virtnet: 1 26391 0.82 MB/sec execute 100 sec latency 2697.167 ms oleg423-client.virtnet: 1 26391 0.81 MB/sec execute 101 sec latency 3697.339 ms oleg423-client.virtnet: 1 26391 0.80 MB/sec execute 102 sec latency 4697.512 ms oleg423-client.virtnet: 1 26391 0.79 MB/sec execute 103 sec latency 5697.666 ms oleg423-client.virtnet: 1 26391 0.78 MB/sec execute 104 sec latency 6697.834 ms oleg423-client.virtnet: 1 26391 0.78 MB/sec execute 105 sec latency 7698.025 ms oleg423-client.virtnet: 1 26391 0.77 MB/sec Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:21:49 (1741900909) targets are mounted 17:21:49 (1741900909) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3703808 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3727360 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7431168 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:22:01 (1741900921) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:22:14 (1741900934) targets are mounted 17:22:14 (1741900934) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3707904 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3729408 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7437312 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:22:28 (1741900948) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server execute 106 sec latency 8698.175 ms oleg423-client.virtnet: 1 26391 0.76 MB/sec execute 107 sec latency 9698.356 ms oleg423-client.virtnet: 1 26391 0.76 MB/sec execute 108 sec latency 10698.470 ms oleg423-client.virtnet: 1 26391 0.75 MB/sec execute 109 sec latency 11698.620 ms oleg423-client.virtnet: 1 26391 0.74 MB/sec execute 110 sec latency 12698.724 ms oleg423-client.virtnet: 1 26391 0.74 MB/sec execute 111 sec latency 13698.894 ms oleg423-client.virtnet: 1 26391 0.73 MB/sec execute 112 sec latency 14699.057 ms oleg423-client.virtnet: 1 26391 0.72 MB/sec execute 113 sec latency 15699.211 ms oleg423-client.virtnet: 1 26391 0.72 MB/sec execute 114 sec latency 16699.354 ms oleg423-client.virtnet: 1 26391 0.71 MB/sec execute 115 sec latency 17699.501 ms oleg423-client.virtnet: 1 26434 0.70 MB/sec execute 116 sec latency 18633.065 ms oleg423-client.virtnet: 1 27046 0.70 MB/sec execute 117 sec latency 21.825 ms oleg423-client.virtnet: 1 27868 0.76 MB/sec execute 118 sec latency 23.970 ms oleg423-client.virtnet: 1 28766 0.80 MB/sec execute 119 sec latency 21.526 ms oleg423-client.virtnet: 1 29297 0.80 MB/sec execute 120 sec latency 31.893 ms oleg423-client.virtnet: 1 29997 0.83 MB/sec execute 121 sec latency 23.565 ms oleg423-client.virtnet: 1 30711 0.84 MB/sec execute 122 sec latency 21.069 ms oleg423-client.virtnet: 1 30719 0.83 MB/sec execute 123 sec latency 989.729 ms oleg423-client.virtnet: 1 30719 0.83 MB/sec execute 124 sec latency 1989.888 ms oleg423-client.virtnet: 1 30719 0.82 MB/sec execute 125 sec latency 2990.055 ms oleg423-client.virtnet: 1 30719 0.81 MB/sec execute 126 sec latency 3990.617 ms oleg423-client.virtnet: 1 30719 0.81 MB/sec execute 127 sec latency 4990.802 ms oleg423-client.virtnet: 1 30719 0.80 MB/sec execute 128 sec latency 5991.042 ms oleg423-client.virtnet: 1 30719 0.79 MB/sec execute 129 sec latency 6991.287 ms oleg423-client.virtnet: 1 30719 0.79 MB/sec execute 130 sec latency 7991.446 ms oleg423-client.virtnet: 1 30719 0.78 MB/sec execute 131 sec latency 8991.671 ms oleg423-client.virtnet: 1 30719 0.78 MB/sec execute 132 sec latency 9992.095 ms oleg423-client.virtnet: 1 30719 0.77 MB/sec execute 133 sec latency 10992.251 ms oleg423-client.virtnet: 1 30719 0.76 MB/sec execute 134 sec latency 11992.404 ms oleg423-client.virtnet: 1 30719 0.76 MB/sec execute 135 sec latency 12992.556 ms oleg423-client.virtnet: 1 30719 0.75 MB/sec execute 136 sec latency 13992.746 ms oleg423-client.virtnet: 1 30719 0.75 MB/sec execute 137 sec latency 14992.941 ms oleg423-client.virtnet: 1 30719 0.74 MB/sec execute 138 sec latency 15993.109 ms oleg423-client.virtnet: 1 30719 0.74 MB/sec execute 139 sec latency 16993.269 ms oleg423-client.virtnet: 1 30719 0.73 MB/sec execute 140 sec latency 17993.454 ms oleg423-client.virtnet: 1 30719 0.73 MB/sec execute 141 sec latency 18994.821 ms oleg423-client.virtnet: 1 31294 0.75 MB/sec execute 142 sec latency 19037.066 ms oleg423-client.virtnet: 1 31767 0.77 MB/sec execute 143 sec latency 61.695 ms oleg423-client.virtnet: 1 32170 0.79 MB/sec execute 144 sec latency 22.168 ms oleg423-client.virtnet: 1 32417 0.79 MB/sec execute 145 sec latency 23.221 ms oleg423-client.virtnet: 1 32688 0.79 MB/sec execute 146 sec latency 27.315 ms oleg423-client.virtnet: 1 32908 0.79 MB/sec execute 147 sec latency 52.761 ms oleg423-client.virtnet: 1 33119 0.79 MB/sec execute 148 sec latency 58.727 ms oleg423-client.virtnet: 1 33170 0.78 MB/sec execute 149 sec latency 696.168 ms oleg423-client.virtnet: 1 33170 0.78 MB/sec execute 15Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:22:41 (1741900961) targets are mounted 17:22:41 (1741900961) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3722240 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3741696 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7463936 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:22:54 (1741900974) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:23:08 (1741900988) targets are mounted 17:23:08 (1741900988) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 0 sec latency 1696.350 ms oleg423-client.virtnet: 1 33170 0.77 MB/sec execute 151 sec latency 2696.514 ms oleg423-client.virtnet: 1 33170 0.77 MB/sec execute 152 sec latency 3696.671 ms oleg423-client.virtnet: 1 33170 0.76 MB/sec execute 153 sec latency 4696.993 ms oleg423-client.virtnet: 1 33170 0.76 MB/sec execute 154 sec latency 5697.182 ms oleg423-client.virtnet: 1 33170 0.75 MB/sec execute 155 sec latency 6697.351 ms oleg423-client.virtnet: 1 33170 0.75 MB/sec execute 156 sec latency 7697.474 ms oleg423-client.virtnet: 1 33170 0.74 MB/sec execute 157 sec latency 8698.553 ms oleg423-client.virtnet: 1 33170 0.74 MB/sec execute 158 sec latency 9698.772 ms oleg423-client.virtnet: 1 33170 0.74 MB/sec execute 159 sec latency 10699.733 ms oleg423-client.virtnet: 1 33170 0.73 MB/sec execute 160 sec latency 11701.892 ms oleg423-client.virtnet: 1 33170 0.73 MB/sec execute 161 sec latency 12702.258 ms oleg423-client.virtnet: 1 33170 0.72 MB/sec execute 162 sec latency 13702.455 ms oleg423-client.virtnet: 1 33170 0.72 MB/sec execute 163 sec latency 14702.645 ms oleg423-client.virtnet: 1 33170 0.71 MB/sec execute 164 sec latency 15702.822 ms oleg423-client.virtnet: 1 33170 0.71 MB/sec execute 165 sec latency 16702.989 ms oleg423-client.virtnet: 1 33170 0.70 MB/sec execute 166 sec latency 17703.416 ms oleg423-client.virtnet: 1 33362 0.72 MB/sec execute 167 sec latency 17847.548 ms oleg423-client.virtnet: 1 33503 0.71 MB/sec execute 168 sec latency 93.087 ms oleg423-client.virtnet: 1 33659 0.71 MB/sec execute 169 sec latency 88.312 ms oleg423-client.virtnet: 1 33786 0.71 MB/sec execute 170 sec latency 60.458 ms oleg423-client.virtnet: 1 33965 0.70 MB/sec execute 171 sec latency 92.960 ms oleg423-client.virtnet: 1 34123 0.70 MB/sec execute 172 sec latency 41.839 ms oleg423-client.virtnet: 1 34349 0.70 MB/sec execute 173 sec latency 68.160 ms oleg423-client.virtnet: 1 34568 0.71 MB/sec execute 174 sec latency 60.218 ms oleg423-client.virtnet: 1 34892 0.73 MB/sec execute 175 sec latency 424.276 ms oleg423-client.virtnet: 1 34892 0.72 MB/sec execute 176 sec latency 1425.012 ms oleg423-client.virtnet: 1 34892 0.72 MB/sec execute 177 sec latency 2425.206 ms oleg423-client.virtnet: 1 34892 0.72 MB/sec execute 178 sec latency 3425.425 ms oleg423-client.virtnet: 1 34892 0.71 MB/sec execute 179 sec latency 4425.613 ms oleg423-client.virtnet: 1 34892 0.71 MB/sec execute 180 sec latency 5425.807 ms oleg423-client.virtnet: 1 34892 0.70 MB/sec execute 181 sec latency 6426.327 ms oleg423-client.virtnet: 1 34892 0.70 MB/sec execute 182 sec latency 7426.667 ms oleg423-client.virtnet: 1 34892 0.70 MB/sec execute 183 sec latency 8427.060 ms oleg423-client.virtnet: 1 34892 0.69 MB/sec execute 184 sec latency 9427.879 ms oleg423-client.virtnet: 1 34892 0.69 MB/sec execute 185 sec latency 10428.877 ms oleg423-client.virtnet: 1 34892 0.68 MB/sec execute 186 sec latency 11429.035 ms oleg423-client.virtnet: 1 34892 0.68 MB/sec execute 187 sec latency 12429.632 ms oleg423-client.virtnet: 1 34892 0.68 MB/sec execute 188 sec latency 13429.814 ms oleg423-client.virtnet: 1 34892 0.67 MB/sec execute 189 sec latency 14430.468 ms oleg423-client.virtnet: 1 34892 0.67 MB/sec execute 190 sec latency 15430.812 ms oleg423-client.virtnet: 1 34892 0.67 MB/sec execute 191 sec latency 16430.921 ms oleg423-client.virtnet: 1 35151 0.67 MB/sec execute 192 sec latency 16590.374 ms oleg423-client.virtnet: 1 35339 0.67 MB/sec execute 193 sec latency 90.731 ms oleg423-client.virtnet: 1 35622 0.68 MB/sec execute 194 seUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210048 3968 2204032 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3731456 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7444480 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:23:19 (1741900999) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:23:33 (1741901013) targets are mounted 17:23:33 (1741901013) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3707904 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3740672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7448576 1% /mnt/lustre test_70b fail mds1 12 times Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:23:43 (1741901023) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:23:56 (1741901036) targets are mounted 17:23:56 (1741901036) facet_failover done c latency 40.265 ms oleg423-client.virtnet: 1 35790 0.68 MB/sec execute 195 sec latency 35.403 ms oleg423-client.virtnet: 1 35953 0.68 MB/sec execute 196 sec latency 27.149 ms oleg423-client.virtnet: 1 36109 0.68 MB/sec execute 197 sec latency 36.838 ms oleg423-client.virtnet: 1 36286 0.68 MB/sec execute 198 sec latency 66.003 ms oleg423-client.virtnet: 1 36419 0.68 MB/sec execute 199 sec latency 102.440 ms oleg423-client.virtnet: 1 36504 0.67 MB/sec execute 200 sec latency 211.887 ms oleg423-client.virtnet: 1 36504 0.67 MB/sec execute 201 sec latency 1213.073 ms oleg423-client.virtnet: 1 36504 0.67 MB/sec execute 202 sec latency 2213.247 ms oleg423-client.virtnet: 1 36504 0.66 MB/sec execute 203 sec latency 3213.920 ms oleg423-client.virtnet: 1 36504 0.66 MB/sec execute 204 sec latency 4214.092 ms oleg423-client.virtnet: 1 36504 0.66 MB/sec execute 205 sec latency 5214.741 ms oleg423-client.virtnet: 1 36504 0.65 MB/sec execute 206 sec latency 6215.474 ms oleg423-client.virtnet: 1 36504 0.65 MB/sec execute 207 sec latency 7215.842 ms oleg423-client.virtnet: 1 36504 0.65 MB/sec execute 208 sec latency 8216.082 ms oleg423-client.virtnet: 1 36504 0.65 MB/sec execute 209 sec latency 9217.894 ms oleg423-client.virtnet: 1 36504 0.64 MB/sec execute 210 sec latency 10218.069 ms oleg423-client.virtnet: 1 36504 0.64 MB/sec execute 211 sec latency 11218.223 ms oleg423-client.virtnet: 1 36504 0.64 MB/sec execute 212 sec latency 12218.393 ms oleg423-client.virtnet: 1 36504 0.63 MB/sec execute 213 sec latency 13218.549 ms oleg423-client.virtnet: 1 36504 0.63 MB/sec execute 214 sec latency 14219.103 ms oleg423-client.virtnet: 1 36504 0.63 MB/sec execute 215 sec latency 15219.308 ms oleg423-client.virtnet: 1 36504 0.62 MB/sec execute 216 sec latency 16219.437 ms oleg423-client.virtnet: 1 36739 0.63 MB/sec execute 217 sec latency 16282.880 ms oleg423-client.virtnet: 1 37067 0.64 MB/sec execute 218 sec latency 71.706 ms oleg423-client.virtnet: 1 37272 0.64 MB/sec execute 219 sec latency 61.786 ms oleg423-client.virtnet: 1 37524 0.63 MB/sec execute 220 sec latency 50.506 ms oleg423-client.virtnet: 1 37843 0.64 MB/sec execute 221 sec latency 19.202 ms oleg423-client.virtnet: 1 38011 0.64 MB/sec execute 222 sec latency 71.766 ms oleg423-client.virtnet: 1 38441 0.66 MB/sec execute 223 sec latency 72.912 ms oleg423-client.virtnet: 1 38736 0.66 MB/sec execute 224 sec latency 446.463 ms oleg423-client.virtnet: 1 38736 0.65 MB/sec execute 225 sec latency 1448.501 ms oleg423-client.virtnet: 1 38736 0.65 MB/sec execute 226 sec latency 2449.324 ms oleg423-client.virtnet: 1 38736 0.65 MB/sec execute 227 sec latency 3451.105 ms oleg423-client.virtnet: 1 38736 0.65 MB/sec execute 228 sec latency 4451.311 ms oleg423-client.virtnet: 1 38736 0.64 MB/sec execute 229 sec latency 5451.476 ms oleg423-client.virtnet: 1 38736 0.64 MB/sec execute 230 sec latency 6453.136 ms oleg423-client.virtnet: 1 38736 0.64 MB/sec execute 231 sec latency 7453.295 ms oleg423-client.virtnet: 1 38736 0.63 MB/sec execute 232 sec latency 8453.563 ms oleg423-client.virtnet: 1 38736 0.63 MB/sec execute 233 sec latency 9454.106 ms oleg423-client.virtnet: 1 38736 0.63 MB/sec execute 234 sec latency 10454.318 ms oleg423-client.virtnet: 1 38736 0.63 MB/sec execute 235 sec latency 11454.438 ms oleg423-client.virtnet: 1 38736 0.62 MB/sec execute 236 sec latency 12454.687 ms oleg423-client.virtnet: 1 38736 0.62 MB/sec execute 237 sec latency 13454.874 ms oleg423-client.virtnet: 1 38736 0.62 MB/sec execute 238 sec latency 14455.oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 066 ms oleg423-client.virtnet: 1 38736 0.62 MB/sec execute 239 sec latency 15455.218 ms oleg423-client.virtnet: 1 38736 0.61 MB/sec execute 240 sec latency 16455.385 ms oleg423-client.virtnet: 1 38781 0.62 MB/sec execute 241 sec latency 17393.285 ms oleg423-client.virtnet: 1 39443 0.63 MB/sec execute 242 sec latency 43.816 ms oleg423-client.virtnet: 1 39939 0.63 MB/sec execute 243 sec latency 30.151 ms oleg423-client.virtnet: 1 40620 0.65 MB/sec execute 244 sec latency 28.019 ms oleg423-client.virtnet: 1 41351 0.65 MB/sec execute 245 sec latency 19.096 ms oleg423-client.virtnet: 1 42251 0.68 MB/sec execute 246 sec latency 29.521 ms oleg423-client.virtnet: 1 42902 0.70 MB/sec execute 247 sec latency 17.713 ms oleg423-client.virtnet: 1 43330 0.70 MB/sec execute 248 sec latency 36.687 ms oleg423-client.virtnet: 1 43803 0.70 MB/sec execute 249 sec latency 25.645 ms oleg423-client.virtnet: 1 44438 0.71 MB/sec execute 250 sec latency 21.746 ms oleg423-client.virtnet: 1 45041 0.71 MB/sec execute 251 sec latency 48.170 ms oleg423-client.virtnet: 1 45955 0.74 MB/sec execute 252 sec latency 24.982 ms oleg423-client.virtnet: 1 46532 0.76 MB/sec execute 253 sec latency 11.704 ms oleg423-client.virtnet: 1 46960 0.75 MB/sec execute 254 sec latency 39.926 ms oleg423-client.virtnet: 1 47550 0.77 MB/sec execute 255 sec latency 25.577 ms oleg423-client.virtnet: 1 48039 0.77 MB/sec execute 256 sec latency 34.675 ms oleg423-client.virtnet: 1 48650 0.77 MB/sec execute 257 sec latency 29.781 ms oleg423-client.virtnet: 1 49689 0.81 MB/sec execute 258 sec latency 23.378 ms oleg423-client.virtnet: 1 50137 0.81 MB/sec execute 259 sec latency 16.018 ms oleg423-client.virtnet: 1 50447 0.81 MB/sec execute 260 sec latency 41.069 ms oleg423-client.virtnet: 1 50892 0.81 MB/sec execute 261 sec latency 32.059 ms oleg423-client.virtnet: 1 51425 0.82 MB/sec execute 262 sec latency 27.616 ms oleg423-client.virtnet: 1 51990 0.83 MB/sec execute 263 sec latency 24.996 ms oleg423-client.virtnet: 1 52604 0.84 MB/sec execute 264 sec latency 43.570 ms oleg423-client.virtnet: 1 53265 0.86 MB/sec execute 265 sec latency 47.265 ms oleg423-client.virtnet: 1 53605 0.86 MB/sec execute 266 sec latency 20.980 ms oleg423-client.virtnet: 1 53918 0.86 MB/sec execute 267 sec latency 65.526 ms oleg423-client.virtnet: 1 54160 0.86 MB/sec execute 268 sec latency 46.292 ms oleg423-client.virtnet: 1 54510 0.86 MB/sec execute 269 sec latency 51.151 ms oleg423-client.virtnet: 1 54874 0.87 MB/sec execute 270 sec latency 38.500 ms oleg423-client.virtnet: 1 55304 0.87 MB/sec execute 271 sec latency 24.494 ms oleg423-client.virtnet: 1 55820 0.87 MB/sec execute 272 sec latency 40.120 ms oleg423-client.virtnet: 1 56594 0.89 MB/sec execute 273 sec latency 49.874 ms oleg423-client.virtnet: 1 57102 0.91 MB/sec execute 274 sec latency 15.847 ms oleg423-client.virtnet: 1 57530 0.91 MB/sec execute 275 sec latency 39.792 ms oleg423-client.virtnet: 1 57985 0.91 MB/sec execute 276 sec latency 26.192 ms oleg423-client.virtnet: 1 58485 0.92 MB/sec execute 277 sec latency 35.813 ms oleg423-client.virtnet: 1 59009 0.92 MB/sec execute 278 sec latency 38.100 ms oleg423-client.virtnet: 1 59788 0.94 MB/sec execute 279 sec latency 30.995 ms oleg423-client.virtnet: 1 60662 0.96 MB/sec execute 280 sec latency 25.683 ms oleg423-client.virtnet: 1 61129 0.95 MB/sec execute 281 sec latency 38.358 ms oleg423-client.virtnet: 1 61463 0.96 MB/sec execute 282 sec latency 41.470 ms oleg423-client.virtnet: 1 62033 0.96 MB/sec execute 283 sec latency 28.507 ms oleg423-client.virtnet: 1 62664 0.97 MB/sec execute 284 sec latency 18.599 ms oleg423-client.virtnet: 1 63515 0.99 MB/sec execute 285 sec latency 25.752 ms oleg423-client.virtnet: 1 64221 1.00 MB/sec execute 286 sec latency 21.729 ms oleg423-client.virtnet: 1 64607 1.00 MB/sec execute 287 sec latency 49.935 ms oleg423-client.virtnet: 1 64920 1.00 MB/sec execute 288 sec latency 44.370 ms oleg423-client.virtnet: 1 65444 1.01 MB/sec execute 289 sec latency 54.444 ms oleg423-client.virtnet: 1 65862 1.01 MB/sec execute 290 sec latency 31.719 ms oleg423-client.virtnet: 1 66361 1.01 MB/sec execute 291 sec latency 29.385 ms oleg423-client.virtnet: 1 67100 1.03 MB/sec execute 292 sec latency 54.129 ms oleg423-client.virtnet: 1 67673 1.04 MB/sec execute 293 sec latency 17.297 ms oleg423-client.virtnet: 1 68049 1.04 MB/sec execute 294 sec latency 19.316 ms oleg423-client.virtnet: 1 68307 1.04 MB/sec execute 295 sec latency 47.945 ms oleg423-client.virtnet: 1 68678 1.04 MB/sec execute 296 sec latency 42.052 ms oleg423-client.virtnet: 1 69002 1.05 MB/sec execute 297 sec latency 75.367 ms oleg423-client.virtnet: 1 69297 1.04 MB/sec execute 298 sec latency 46.586 ms oleg423-client.virtnet: 1 69798 1.05 MB/sec execute 299 sec latency 37.706 ms oleg423-client.virtnet: 1 cleanup 300 sec oleg423-client.virtnet: 0 cleanup 300 sec oleg423-client.virtnet: oleg423-client.virtnet: Operation Count AvgLat MaxLat oleg423-client.virtnet: ---------------------------------------- oleg423-client.virtnet: NTCreateX 10268 13.203 18719.621 oleg423-client.virtnet: Close 7566 8.188 18694.869 oleg423-client.virtnet: Rename 437 9.445 83.598 oleg423-client.virtnet: Unlink 2066 2.195 15.838 oleg423-client.virtnet: Qpathinfo 9349 5.646 19037.054 oleg423-client.virtnet: Qfileinfo 1640 0.152 4.937 oleg423-client.virtnet: Qfsinfo 1694 0.123 7.001 oleg423-client.virtnet: Sfileinfo 850 24.496 16282.869 oleg423-client.virtnet: Find 3592 0.458 19.957 oleg423-client.virtnet: WriteX 5092 0.835 18.553 oleg423-client.virtnet: ReadX 16184 0.025 8.255 oleg423-client.virtnet: LockX 34 1.207 4.652 oleg423-client.virtnet: UnlockX 34 1.134 3.846 oleg423-client.virtnet: Flush 727 21.655 136.989 oleg423-client.virtnet: oleg423-client.virtnet: Throughput 1.04657 MB/sec 1 clients 1 procs max_latency=19037.066 ms oleg423-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg423-client.virtnet at Thu Mar 13 17:24:59 EDT 2025 with return code 0 oleg423-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg423-client.virtnet oleg423-client.virtnet: /mnt/lustre/d70b.replay-single/oleg423-client.virtnet /mnt/lustre/d70b.replay-single/oleg423-client.virtnet oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg423-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg423-client.virtnet: removed directory: 'clients/client0' oleg423-client.virtnet: removed directory: 'clients' oleg423-client.virtnet: removed 'client.txt' oleg423-client.virtnet: /mnt/lustre/d70b.replay-single/oleg423-client.virtnet oleg423-client.virtnet: dbench successfully finished