== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 01:17:34 (1708841854) Starting client oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre Started clients oleg319-client.virtnet: 192.168.203.119@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg319-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=14005 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3685376 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3716096 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 8192 7401472 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:17:44 (1708841864) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:17:58 (1708841878) targets are mounted 01:17:58 (1708841878) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg319-client.virtnet: looking for dbench program oleg319-client.virtnet: /usr/bin/dbench oleg319-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg319-client.virtnet oleg319-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg319-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg319-client.virtnet' oleg319-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg319-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg319-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg319-client.virtnet at Sun Feb 25 01:17:35 EST 2024 oleg319-client.virtnet: waiting for dbench pid 14029 oleg319-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg319-client.virtnet: oleg319-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg319-client.virtnet: failed to create barrier semaphore oleg319-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg319-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg319-client.virtnet: releasing clients oleg319-client.virtnet: 1 305 11.22 MB/sec warmup 1 sec latency 37.006 ms oleg319-client.virtnet: 1 664 10.58 MB/sec warmup 2 sec latency 29.472 ms oleg319-client.virtnet: 1 873 7.22 MB/sec warmup 3 sec latency 88.687 ms oleg319-client.virtnet: 1 1167 5.75 MB/sec warmup 4 sec latency 49.396 ms oleg319-client.virtnet: 1 1551 5.22 MB/sec warmup 5 sec latency 67.078 ms oleg319-client.virtnet: 1 1785 4.38 MB/sec warmup 6 sec latency 319.581 ms oleg319-client.virtnet: 1 1785 3.76 MB/sec warmup 7 sec latency 1319.908 ms oleg319-client.virtnet: 1 1785 3.29 MB/sec warmup 8 sec latency 2320.115 ms oleg319-client.virtnet: 1 1785 2.92 MB/sec warmup 9 sec latency 3320.349 ms oleg319-client.virtnet: 1 1785 2.63 MB/sec warmup 10 sec latency 4320.545 ms oleg319-client.virtnet: 1 1785 2.39 MB/sec warmup 11 sec latency 5320.803 ms oleg319-client.virtnet: 1 1785 2.19 MB/sec warmup 12 sec latency 6321.043 ms oleg319-client.virtnet: 1 1785 2.02 MB/sec warmup 13 sec latency 7321.294 ms oleg319-client.virtnet: 1 1785 1.88 MB/sec warmup 14 sec latency 8321.583 ms oleg319-client.virtnet: 1 1785 1.75 MB/sec warmup 15 sec latency 9321.891 ms oleg319-client.virtnet: 1 1785 1.64 MB/sec warmup 16 sec latency 10322.150 ms oleg319-client.virtnet: 1 1785 1.55 MB/sec warmup 17 sec latency 11322.407 ms oleg319-client.virtnet: 1 1785 1.46 MB/sec warmup 18 sec latency 12322.687 ms oleg319-client.virtnet: 1 1785 1.38 MB/sec warmup 19 sec latency 13322.917 ms oleg319-client.virtnet: 1 1785 1.31 MB/sec warmup 20 sec latency 14323.113 ms oleg319-client.virtnet: 1 1785 1.25 MB/sec warmup 21 sec latency 15323.338 ms oleg319-client.virtnet: 1 1785 1.20 MB/sec warmup 22 sec latency 16323.525 ms oleg319-client.virtnet: 1 1785 1.14 MB/sec warmup 23 sec latency 17323.783 ms oleg319-client.virtnet: 1 1785 1.10 MB/sec warmup 24 sec latency 18324.008 ms oleg319-client.virtnet: 1 1785 1.05 MB/sec warmup 25 sec latency 19324.246 ms oleg319-client.virtnet: 1 1785 1.01 MB/sec warmup 26 sec latency 20324.541 ms oleg319-client.virtnet: 1 1785 0.97 MB/sec warmup 27 sec latency 21324.844 ms oleg319-client.virtnet: 1 1785 0.94 MB/sec warmup 28 sec latency 22325.109 ms oleg319-client.virtnet: 1 1785 0.91 MB/sec warmup 29 sec latency 23325.342 ms oleg319-client.virtnet: 1 2108 0.89 MB/sec warmup 30 sec latency 23454.453 ms oleg319-client.virtnet: 1 2470 0.92 MB/sec warmup 31 sec latency 41.325 ms oleg319-client.virtnet: 1 3190 1.07 MB/sec warmup 32 sec laUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3680256 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3733504 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 47104 7413760 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:18:13 (1708841893) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:18:26 (1708841906) targets are mounted 01:18:26 (1708841906) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3716096 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7453696 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:18:38 (1708841918) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:18:52 (1708841932) targets are mounted 01:18:52 (1708841932) facet_failover done tency 37.986 ms oleg319-client.virtnet: 1 3673 1.17 MB/sec warmup 33 sec latency 45.762 ms oleg319-client.virtnet: 1 3948 1.18 MB/sec warmup 34 sec latency 16.928 ms oleg319-client.virtnet: 1 4236 1.15 MB/sec warmup 35 sec latency 49.353 ms oleg319-client.virtnet: 1 4446 1.13 MB/sec warmup 36 sec latency 52.790 ms oleg319-client.virtnet: 1 4609 1.14 MB/sec warmup 37 sec latency 411.412 ms oleg319-client.virtnet: 1 4609 1.11 MB/sec warmup 38 sec latency 1411.614 ms oleg319-client.virtnet: 1 4609 1.08 MB/sec warmup 39 sec latency 2411.787 ms oleg319-client.virtnet: 1 4609 1.05 MB/sec warmup 40 sec latency 3412.031 ms oleg319-client.virtnet: 1 4609 1.02 MB/sec warmup 41 sec latency 4412.348 ms oleg319-client.virtnet: 1 4609 1.00 MB/sec warmup 42 sec latency 5412.557 ms oleg319-client.virtnet: 1 4609 0.98 MB/sec warmup 43 sec latency 6412.822 ms oleg319-client.virtnet: 1 4609 0.95 MB/sec warmup 44 sec latency 7413.107 ms oleg319-client.virtnet: 1 4609 0.93 MB/sec warmup 45 sec latency 8413.369 ms oleg319-client.virtnet: 1 4609 0.91 MB/sec warmup 46 sec latency 9413.633 ms oleg319-client.virtnet: 1 4609 0.89 MB/sec warmup 47 sec latency 10413.891 ms oleg319-client.virtnet: 1 4609 0.88 MB/sec warmup 48 sec latency 11414.214 ms oleg319-client.virtnet: 1 4609 0.86 MB/sec warmup 49 sec latency 12414.451 ms oleg319-client.virtnet: 1 4609 0.84 MB/sec warmup 50 sec latency 13414.716 ms oleg319-client.virtnet: 1 4609 0.82 MB/sec warmup 51 sec latency 14415.007 ms oleg319-client.virtnet: 1 4609 0.81 MB/sec warmup 52 sec latency 15415.280 ms oleg319-client.virtnet: 1 4609 0.79 MB/sec warmup 53 sec latency 16415.517 ms oleg319-client.virtnet: 1 4609 0.78 MB/sec warmup 54 sec latency 17415.726 ms oleg319-client.virtnet: 1 4946 0.82 MB/sec warmup 55 sec latency 17652.573 ms oleg319-client.virtnet: 1 5218 0.81 MB/sec warmup 56 sec latency 54.671 ms oleg319-client.virtnet: 1 5549 0.80 MB/sec warmup 57 sec latency 38.583 ms oleg319-client.virtnet: 1 5936 0.81 MB/sec warmup 58 sec latency 38.536 ms oleg319-client.virtnet: 1 6332 0.87 MB/sec warmup 59 sec latency 56.214 ms oleg319-client.virtnet: 1 7324 4.38 MB/sec execute 1 sec latency 38.930 ms oleg319-client.virtnet: 1 7516 2.34 MB/sec execute 2 sec latency 189.744 ms oleg319-client.virtnet: 1 7516 1.56 MB/sec execute 3 sec latency 1189.961 ms oleg319-client.virtnet: 1 7516 1.17 MB/sec execute 4 sec latency 2190.225 ms oleg319-client.virtnet: 1 7516 0.94 MB/sec execute 5 sec latency 3190.489 ms oleg319-client.virtnet: 1 7516 0.78 MB/sec execute 6 sec latency 4190.804 ms oleg319-client.virtnet: 1 7516 0.67 MB/sec execute 7 sec latency 5191.057 ms oleg319-client.virtnet: 1 7516 0.59 MB/sec execute 8 sec latency 6191.303 ms oleg319-client.virtnet: 1 7516 0.52 MB/sec execute 9 sec latency 7191.582 ms oleg319-client.virtnet: 1 7516 0.47 MB/sec execute 10 sec latency 8191.847 ms oleg319-client.virtnet: 1 7516 0.43 MB/sec execute 11 sec latency 9192.103 ms oleg319-client.virtnet: 1 7516 0.39 MB/sec execute 12 sec latency 10192.350 ms oleg319-client.virtnet: 1 7516 0.36 MB/sec execute 13 sec latency 11192.611 ms oleg319-client.virtnet: 1 7516 0.33 MB/sec execute 14 sec latency 12192.808 ms oleg319-client.virtnet: 1 7516 0.31 MB/sec execute 15 sec latency 13192.966 ms oleg319-client.virtnet: 1 7516 0.29 MB/sec execute 16 sec latency 14193.140 ms oleg319-client.virtnet: 1 7516 0.28 MB/sec execute 17 sec latency 15193.409 ms oleg319-client.virtnoleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3712000 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3749888 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7461888 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:19:02 (1708841942) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:19:16 (1708841956) targets are mounted 01:19:16 (1708841956) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3689472 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7427072 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:19:28 (1708841968) shut down et: 1 7516 0.26 MB/sec execute 18 sec latency 16193.687 ms oleg319-client.virtnet: 1 7516 0.25 MB/sec execute 19 sec latency 17193.871 ms oleg319-client.virtnet: 1 7692 0.24 MB/sec execute 20 sec latency 17487.142 ms oleg319-client.virtnet: 1 7914 0.25 MB/sec execute 21 sec latency 62.288 ms oleg319-client.virtnet: 1 8209 0.30 MB/sec execute 22 sec latency 41.479 ms oleg319-client.virtnet: 1 8601 0.43 MB/sec execute 23 sec latency 45.630 ms oleg319-client.virtnet: 1 8897 0.42 MB/sec execute 24 sec latency 46.593 ms oleg319-client.virtnet: 1 9290 0.42 MB/sec execute 25 sec latency 26.637 ms oleg319-client.virtnet: 1 10081 0.66 MB/sec execute 26 sec latency 25.995 ms oleg319-client.virtnet: 1 10371 0.71 MB/sec execute 27 sec latency 699.138 ms oleg319-client.virtnet: 1 10371 0.68 MB/sec execute 28 sec latency 1699.376 ms oleg319-client.virtnet: 1 10371 0.66 MB/sec execute 29 sec latency 2699.670 ms oleg319-client.virtnet: 1 10371 0.64 MB/sec execute 30 sec latency 3699.929 ms oleg319-client.virtnet: 1 10371 0.61 MB/sec execute 31 sec latency 4700.238 ms oleg319-client.virtnet: 1 10371 0.60 MB/sec execute 32 sec latency 5700.549 ms oleg319-client.virtnet: 1 10371 0.58 MB/sec execute 33 sec latency 6700.929 ms oleg319-client.virtnet: 1 10371 0.56 MB/sec execute 34 sec latency 7701.201 ms oleg319-client.virtnet: 1 10371 0.54 MB/sec execute 35 sec latency 8701.508 ms oleg319-client.virtnet: 1 10371 0.53 MB/sec execute 36 sec latency 9701.830 ms oleg319-client.virtnet: 1 10371 0.52 MB/sec execute 37 sec latency 10702.053 ms oleg319-client.virtnet: 1 10371 0.50 MB/sec execute 38 sec latency 11702.224 ms oleg319-client.virtnet: 1 10371 0.49 MB/sec execute 39 sec latency 12702.465 ms oleg319-client.virtnet: 1 10371 0.48 MB/sec execute 40 sec latency 13702.726 ms oleg319-client.virtnet: 1 10371 0.46 MB/sec execute 41 sec latency 14702.921 ms oleg319-client.virtnet: 1 10371 0.45 MB/sec execute 42 sec latency 15703.097 ms oleg319-client.virtnet: 1 10371 0.44 MB/sec execute 43 sec latency 16703.374 ms oleg319-client.virtnet: 1 10371 0.43 MB/sec execute 44 sec latency 17703.603 ms oleg319-client.virtnet: 1 10710 0.50 MB/sec execute 45 sec latency 17955.528 ms oleg319-client.virtnet: 1 10975 0.51 MB/sec execute 46 sec latency 17.311 ms oleg319-client.virtnet: 1 11219 0.51 MB/sec execute 47 sec latency 15.982 ms oleg319-client.virtnet: 1 11451 0.51 MB/sec execute 48 sec latency 55.357 ms oleg319-client.virtnet: 1 11720 0.53 MB/sec execute 49 sec latency 41.271 ms oleg319-client.virtnet: 1 12137 0.58 MB/sec execute 50 sec latency 15.958 ms oleg319-client.virtnet: 1 12446 0.57 MB/sec execute 51 sec latency 34.828 ms oleg319-client.virtnet: 1 12689 0.57 MB/sec execute 52 sec latency 283.804 ms oleg319-client.virtnet: 1 12689 0.56 MB/sec execute 53 sec latency 1284.076 ms oleg319-client.virtnet: 1 12689 0.55 MB/sec execute 54 sec latency 2284.295 ms oleg319-client.virtnet: 1 12689 0.54 MB/sec execute 55 sec latency 3284.530 ms oleg319-client.virtnet: 1 12689 0.53 MB/sec execute 56 sec latency 4284.742 ms oleg319-client.virtnet: 1 12689 0.52 MB/sec execute 57 sec latency 5284.966 ms oleg319-client.virtnet: 1 12689 0.51 MB/sec execute 58 sec latency 6285.162 ms oleg319-client.virtnet: 1 12689 0.50 MB/sec execute 59 sec latency 7285.334 ms oleg319-client.virtnet: 1 12689 0.49 MB/sec execute 60 sec latency 8285.583 ms oleg319-client.virtnet: 1 12689 0.48 MB/sec execute 61 sec latency 9285.839 ms oleg319-client.virtnet: 1 12689Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:19:42 (1708841982) targets are mounted 01:19:42 (1708841982) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3706880 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7443456 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:19:53 (1708841993) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:20:06 (1708842006) targets are mounted 01:20:06 (1708842006) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3704832 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3740672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7445504 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:20:17 (1708842017) shut down 0.47 MB/sec execute 62 sec latency 10286.122 ms oleg319-client.virtnet: 1 12689 0.47 MB/sec execute 63 sec latency 11286.361 ms oleg319-client.virtnet: 1 12689 0.46 MB/sec execute 64 sec latency 12286.596 ms oleg319-client.virtnet: 1 12689 0.45 MB/sec execute 65 sec latency 13286.813 ms oleg319-client.virtnet: 1 12689 0.45 MB/sec execute 66 sec latency 14287.079 ms oleg319-client.virtnet: 1 12689 0.44 MB/sec execute 67 sec latency 15287.311 ms oleg319-client.virtnet: 1 12689 0.43 MB/sec execute 68 sec latency 16287.547 ms oleg319-client.virtnet: 1 12689 0.43 MB/sec execute 69 sec latency 17287.749 ms oleg319-client.virtnet: 1 13030 0.44 MB/sec execute 70 sec latency 17488.045 ms oleg319-client.virtnet: 1 13521 0.49 MB/sec execute 71 sec latency 54.201 ms oleg319-client.virtnet: 1 13992 0.53 MB/sec execute 72 sec latency 55.287 ms oleg319-client.virtnet: 1 14406 0.58 MB/sec execute 73 sec latency 17.487 ms oleg319-client.virtnet: 1 14638 0.58 MB/sec execute 74 sec latency 16.481 ms oleg319-client.virtnet: 1 14925 0.58 MB/sec execute 75 sec latency 53.051 ms oleg319-client.virtnet: 1 15122 0.58 MB/sec execute 76 sec latency 68.190 ms oleg319-client.virtnet: 1 15336 0.58 MB/sec execute 77 sec latency 338.000 ms oleg319-client.virtnet: 1 15336 0.58 MB/sec execute 78 sec latency 1338.266 ms oleg319-client.virtnet: 1 15336 0.57 MB/sec execute 79 sec latency 2338.492 ms oleg319-client.virtnet: 1 15336 0.56 MB/sec execute 80 sec latency 3338.721 ms oleg319-client.virtnet: 1 15336 0.56 MB/sec execute 81 sec latency 4338.889 ms oleg319-client.virtnet: 1 15336 0.55 MB/sec execute 82 sec latency 5339.082 ms oleg319-client.virtnet: 1 15336 0.54 MB/sec execute 83 sec latency 6339.258 ms oleg319-client.virtnet: 1 15336 0.54 MB/sec execute 84 sec latency 7339.443 ms oleg319-client.virtnet: 1 15336 0.53 MB/sec execute 85 sec latency 8339.705 ms oleg319-client.virtnet: 1 15336 0.52 MB/sec execute 86 sec latency 9339.916 ms oleg319-client.virtnet: 1 15336 0.52 MB/sec execute 87 sec latency 10340.117 ms oleg319-client.virtnet: 1 15336 0.51 MB/sec execute 88 sec latency 11340.360 ms oleg319-client.virtnet: 1 15336 0.51 MB/sec execute 89 sec latency 12340.608 ms oleg319-client.virtnet: 1 15336 0.50 MB/sec execute 90 sec latency 13340.753 ms oleg319-client.virtnet: 1 15336 0.49 MB/sec execute 91 sec latency 14340.938 ms oleg319-client.virtnet: 1 15336 0.49 MB/sec execute 92 sec latency 15341.125 ms oleg319-client.virtnet: 1 15336 0.48 MB/sec execute 93 sec latency 16341.329 ms oleg319-client.virtnet: 1 15336 0.48 MB/sec execute 94 sec latency 17341.558 ms oleg319-client.virtnet: 1 15683 0.51 MB/sec execute 95 sec latency 17582.581 ms oleg319-client.virtnet: 1 16004 0.50 MB/sec execute 96 sec latency 44.213 ms oleg319-client.virtnet: 1 16407 0.50 MB/sec execute 97 sec latency 36.158 ms oleg319-client.virtnet: 1 16799 0.52 MB/sec execute 98 sec latency 43.038 ms oleg319-client.virtnet: 1 17571 0.58 MB/sec execute 99 sec latency 40.735 ms oleg319-client.virtnet: 1 18027 0.62 MB/sec execute 100 sec latency 13.056 ms oleg319-client.virtnet: 1 18349 0.61 MB/sec execute 101 sec latency 16.996 ms oleg319-client.virtnet: 1 18349 0.61 MB/sec execute 102 sec latency 1017.145 ms oleg319-client.virtnet: 1 18349 0.60 MB/sec execute 103 sec latency 2017.332 ms oleg319-client.virtnet: 1 18349 0.60 MB/sec execute 104 sec latency 3017.554 ms oleg319-client.virtnet: 1 18349 0.59 MB/sec execute 105 sec latency 4017.809 ms oleg319-client.virtnet: 1 18349 0.59 MFailover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:20:30 (1708842030) targets are mounted 01:20:30 (1708842030) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3721216 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7467008 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:20:42 (1708842042) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:20:55 (1708842055) targets are mounted 01:20:55 (1708842055) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3695616 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7439360 1% /mnt/lustre test_70b fail mds1 9 times B/sec execute 106 sec latency 5018.043 ms oleg319-client.virtnet: 1 18349 0.58 MB/sec execute 107 sec latency 6018.266 ms oleg319-client.virtnet: 1 18349 0.57 MB/sec execute 108 sec latency 7018.468 ms oleg319-client.virtnet: 1 18349 0.57 MB/sec execute 109 sec latency 8018.666 ms oleg319-client.virtnet: 1 18349 0.56 MB/sec execute 110 sec latency 9018.913 ms oleg319-client.virtnet: 1 18349 0.56 MB/sec execute 111 sec latency 10019.097 ms oleg319-client.virtnet: 1 18349 0.55 MB/sec execute 112 sec latency 11019.304 ms oleg319-client.virtnet: 1 18349 0.55 MB/sec execute 113 sec latency 12019.489 ms oleg319-client.virtnet: 1 18349 0.54 MB/sec execute 114 sec latency 13019.770 ms oleg319-client.virtnet: 1 18349 0.54 MB/sec execute 115 sec latency 14019.985 ms oleg319-client.virtnet: 1 18349 0.53 MB/sec execute 116 sec latency 15020.113 ms oleg319-client.virtnet: 1 18349 0.53 MB/sec execute 117 sec latency 16020.241 ms oleg319-client.virtnet: 1 18349 0.53 MB/sec execute 118 sec latency 17020.409 ms oleg319-client.virtnet: 1 18349 0.52 MB/sec execute 119 sec latency 18020.606 ms oleg319-client.virtnet: 1 18495 0.52 MB/sec execute 120 sec latency 18464.327 ms oleg319-client.virtnet: 1 18819 0.53 MB/sec execute 121 sec latency 34.765 ms oleg319-client.virtnet: 1 19310 0.55 MB/sec execute 122 sec latency 31.394 ms oleg319-client.virtnet: 1 19665 0.55 MB/sec execute 123 sec latency 46.833 ms oleg319-client.virtnet: 1 20163 0.56 MB/sec execute 124 sec latency 31.106 ms oleg319-client.virtnet: 1 20758 0.60 MB/sec execute 125 sec latency 31.979 ms oleg319-client.virtnet: 1 21399 0.63 MB/sec execute 126 sec latency 29.517 ms oleg319-client.virtnet: 1 21404 0.63 MB/sec execute 127 sec latency 989.527 ms oleg319-client.virtnet: 1 21404 0.62 MB/sec execute 128 sec latency 1989.660 ms oleg319-client.virtnet: 1 21404 0.62 MB/sec execute 129 sec latency 2989.786 ms oleg319-client.virtnet: 1 21404 0.61 MB/sec execute 130 sec latency 3989.930 ms oleg319-client.virtnet: 1 21404 0.61 MB/sec execute 131 sec latency 4990.152 ms oleg319-client.virtnet: 1 21404 0.60 MB/sec execute 132 sec latency 5990.344 ms oleg319-client.virtnet: 1 21404 0.60 MB/sec execute 133 sec latency 6990.564 ms oleg319-client.virtnet: 1 21404 0.60 MB/sec execute 134 sec latency 7990.737 ms oleg319-client.virtnet: 1 21404 0.59 MB/sec execute 135 sec latency 8990.930 ms oleg319-client.virtnet: 1 21404 0.59 MB/sec execute 136 sec latency 9991.141 ms oleg319-client.virtnet: 1 21404 0.58 MB/sec execute 137 sec latency 10991.321 ms oleg319-client.virtnet: 1 21404 0.58 MB/sec execute 138 sec latency 11991.490 ms oleg319-client.virtnet: 1 21404 0.57 MB/sec execute 139 sec latency 12991.615 ms oleg319-client.virtnet: 1 21404 0.57 MB/sec execute 140 sec latency 13991.769 ms oleg319-client.virtnet: 1 21404 0.57 MB/sec execute 141 sec latency 14991.925 ms oleg319-client.virtnet: 1 21404 0.56 MB/sec execute 142 sec latency 15992.129 ms oleg319-client.virtnet: 1 21404 0.56 MB/sec execute 143 sec latency 16992.322 ms oleg319-client.virtnet: 1 21404 0.55 MB/sec execute 144 sec latency 17992.464 ms oleg319-client.virtnet: 1 21671 0.56 MB/sec execute 145 sec latency 18255.602 ms oleg319-client.virtnet: 1 21944 0.56 MB/sec execute 146 sec latency 15.749 ms oleg319-client.virtnet: 1 22198 0.56 MB/sec execute 147 sec latency 53.119 ms oleg319-client.virtnet: 1 22508 0.56 MB/sec execute 148 sec latency 41.076 ms oleg319-client.virtnet: 1 22984 0.58 MB/sec execute 149 sec latency 34.456 ms oleg319-client.virtnet: 1 23485 0.58 MB/sec Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:21:07 (1708842067) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:21:21 (1708842081) targets are mounted 01:21:21 (1708842081) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3714048 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7450624 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:21:32 (1708842092) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:21:46 (1708842106) targets are mounted 01:21:46 (1708842106) facet_failover done execute 150 sec latency 22.125 ms oleg319-client.virtnet: 1 23826 0.59 MB/sec execute 151 sec latency 40.422 ms oleg319-client.virtnet: 1 23882 0.59 MB/sec execute 152 sec latency 964.299 ms oleg319-client.virtnet: 1 23882 0.58 MB/sec execute 153 sec latency 1964.446 ms oleg319-client.virtnet: 1 23882 0.58 MB/sec execute 154 sec latency 2964.631 ms oleg319-client.virtnet: 1 23882 0.58 MB/sec execute 155 sec latency 3964.908 ms oleg319-client.virtnet: 1 23882 0.57 MB/sec execute 156 sec latency 4965.108 ms oleg319-client.virtnet: 1 23882 0.57 MB/sec execute 157 sec latency 5965.304 ms oleg319-client.virtnet: 1 23882 0.56 MB/sec execute 158 sec latency 6965.477 ms oleg319-client.virtnet: 1 23882 0.56 MB/sec execute 159 sec latency 7965.648 ms oleg319-client.virtnet: 1 23882 0.56 MB/sec execute 160 sec latency 8965.918 ms oleg319-client.virtnet: 1 23882 0.55 MB/sec execute 161 sec latency 9966.171 ms oleg319-client.virtnet: 1 23882 0.55 MB/sec execute 162 sec latency 10966.361 ms oleg319-client.virtnet: 1 23882 0.55 MB/sec execute 163 sec latency 11966.558 ms oleg319-client.virtnet: 1 23882 0.54 MB/sec execute 164 sec latency 12966.719 ms oleg319-client.virtnet: 1 23882 0.54 MB/sec execute 165 sec latency 13966.945 ms oleg319-client.virtnet: 1 23882 0.54 MB/sec execute 166 sec latency 14967.104 ms oleg319-client.virtnet: 1 23882 0.53 MB/sec execute 167 sec latency 15967.295 ms oleg319-client.virtnet: 1 23882 0.53 MB/sec execute 168 sec latency 16967.440 ms oleg319-client.virtnet: 1 23882 0.53 MB/sec execute 169 sec latency 17967.596 ms oleg319-client.virtnet: 1 24305 0.55 MB/sec execute 170 sec latency 18265.262 ms oleg319-client.virtnet: 1 24907 0.58 MB/sec execute 171 sec latency 38.557 ms oleg319-client.virtnet: 1 25243 0.58 MB/sec execute 172 sec latency 13.079 ms oleg319-client.virtnet: 1 25551 0.58 MB/sec execute 173 sec latency 41.601 ms oleg319-client.virtnet: 1 25801 0.58 MB/sec execute 174 sec latency 49.146 ms oleg319-client.virtnet: 1 26199 0.59 MB/sec execute 175 sec latency 39.371 ms oleg319-client.virtnet: 1 26566 0.60 MB/sec execute 176 sec latency 50.144 ms oleg319-client.virtnet: 1 26656 0.60 MB/sec execute 177 sec latency 744.712 ms oleg319-client.virtnet: 1 26656 0.59 MB/sec execute 178 sec latency 1744.911 ms oleg319-client.virtnet: 1 26656 0.59 MB/sec execute 179 sec latency 2745.106 ms oleg319-client.virtnet: 1 26656 0.59 MB/sec execute 180 sec latency 3745.280 ms oleg319-client.virtnet: 1 26656 0.58 MB/sec execute 181 sec latency 4745.463 ms oleg319-client.virtnet: 1 26656 0.58 MB/sec execute 182 sec latency 5745.688 ms oleg319-client.virtnet: 1 26656 0.58 MB/sec execute 183 sec latency 6745.929 ms oleg319-client.virtnet: 1 26656 0.57 MB/sec execute 184 sec latency 7746.120 ms oleg319-client.virtnet: 1 26656 0.57 MB/sec execute 185 sec latency 8746.309 ms oleg319-client.virtnet: 1 26656 0.57 MB/sec execute 186 sec latency 9746.457 ms oleg319-client.virtnet: 1 26656 0.57 MB/sec execute 187 sec latency 10746.641 ms oleg319-client.virtnet: 1 26656 0.56 MB/sec execute 188 sec latency 11746.811 ms oleg319-client.virtnet: 1 26656 0.56 MB/sec execute 189 sec latency 12746.997 ms oleg319-client.virtnet: 1 26656 0.56 MB/sec execute 190 sec latency 13747.211 ms oleg319-client.virtnet: 1 26656 0.55 MB/sec execute 191 sec latency 14747.408 ms oleg319-client.virtnet: 1 26656 0.55 MB/sec execute 192 sec latency 15747.610 ms oleg319-client.virtnet: 1 26656 0.55 MB/sec execute 193 sec latency 16747.783 ms oleg319-client.virtnet: 1 26656 0.54 MB/sec execuoleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3729408 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7442432 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:21:58 (1708842118) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:22:11 (1708842131) targets are mounted 01:22:11 (1708842131) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3707904 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3740672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7448576 1% /mnt/lustre test_70b fail mds1 12 times Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 01:22:22 (1708842142) shut down Failover mds1 to oleg319-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 te 194 sec latency 17747.972 ms oleg319-client.virtnet: 1 26891 0.54 MB/sec execute 195 sec latency 18120.630 ms oleg319-client.virtnet: 1 27292 0.55 MB/sec execute 196 sec latency 43.388 ms oleg319-client.virtnet: 1 27803 0.57 MB/sec execute 197 sec latency 44.399 ms oleg319-client.virtnet: 1 28407 0.59 MB/sec execute 198 sec latency 44.770 ms oleg319-client.virtnet: 1 28720 0.60 MB/sec execute 199 sec latency 15.408 ms oleg319-client.virtnet: 1 29090 0.60 MB/sec execute 200 sec latency 12.681 ms oleg319-client.virtnet: 1 29311 0.60 MB/sec execute 201 sec latency 53.048 ms oleg319-client.virtnet: 1 29466 0.60 MB/sec execute 202 sec latency 476.690 ms oleg319-client.virtnet: 1 29466 0.60 MB/sec execute 203 sec latency 1476.950 ms oleg319-client.virtnet: 1 29466 0.60 MB/sec execute 204 sec latency 2477.147 ms oleg319-client.virtnet: 1 29466 0.59 MB/sec execute 205 sec latency 3477.311 ms oleg319-client.virtnet: 1 29466 0.59 MB/sec execute 206 sec latency 4477.526 ms oleg319-client.virtnet: 1 29466 0.59 MB/sec execute 207 sec latency 5477.698 ms oleg319-client.virtnet: 1 29466 0.58 MB/sec execute 208 sec latency 6477.911 ms oleg319-client.virtnet: 1 29466 0.58 MB/sec execute 209 sec latency 7478.122 ms oleg319-client.virtnet: 1 29466 0.58 MB/sec execute 210 sec latency 8478.283 ms oleg319-client.virtnet: 1 29466 0.58 MB/sec execute 211 sec latency 9478.521 ms oleg319-client.virtnet: 1 29466 0.57 MB/sec execute 212 sec latency 10478.716 ms oleg319-client.virtnet: 1 29466 0.57 MB/sec execute 213 sec latency 11478.908 ms oleg319-client.virtnet: 1 29466 0.57 MB/sec execute 214 sec latency 12479.112 ms oleg319-client.virtnet: 1 29466 0.56 MB/sec execute 215 sec latency 13479.288 ms oleg319-client.virtnet: 1 29466 0.56 MB/sec execute 216 sec latency 14479.515 ms oleg319-client.virtnet: 1 29466 0.56 MB/sec execute 217 sec latency 15479.700 ms oleg319-client.virtnet: 1 29466 0.56 MB/sec execute 218 sec latency 16479.908 ms oleg319-client.virtnet: 1 29625 0.56 MB/sec execute 219 sec latency 17043.936 ms oleg319-client.virtnet: 1 30037 0.57 MB/sec execute 220 sec latency 51.757 ms oleg319-client.virtnet: 1 30388 0.57 MB/sec execute 221 sec latency 46.631 ms oleg319-client.virtnet: 1 30808 0.57 MB/sec execute 222 sec latency 40.466 ms oleg319-client.virtnet: 1 31331 0.59 MB/sec execute 223 sec latency 58.083 ms oleg319-client.virtnet: 1 31908 0.61 MB/sec execute 224 sec latency 43.586 ms oleg319-client.virtnet: 1 32241 0.61 MB/sec execute 225 sec latency 14.019 ms oleg319-client.virtnet: 1 32402 0.61 MB/sec execute 226 sec latency 367.238 ms oleg319-client.virtnet: 1 32402 0.61 MB/sec execute 227 sec latency 1367.473 ms oleg319-client.virtnet: 1 32402 0.61 MB/sec execute 228 sec latency 2367.662 ms oleg319-client.virtnet: 1 32402 0.61 MB/sec execute 229 sec latency 3367.860 ms oleg319-client.virtnet: 1 32402 0.60 MB/sec execute 230 sec latency 4368.094 ms oleg319-client.virtnet: 1 32402 0.60 MB/sec execute 231 sec latency 5368.286 ms oleg319-client.virtnet: 1 32402 0.60 MB/sec execute 232 sec latency 6368.459 ms oleg319-client.virtnet: 1 32402 0.59 MB/sec execute 233 sec latency 7368.593 ms oleg319-client.virtnet: 1 32402 0.59 MB/sec execute 234 sec latency 8368.715 ms oleg319-client.virtnet: 1 32402 0.59 MB/sec execute 235 sec latency 9368.970 ms oleg319-client.virtnet: 1 32402 0.59 MB/sec execute 236 sec latency 10369.175 ms oleg319-client.virtnet: 1 32402 0.58 MB/sec execute 237 sec latency 11369.353 ms oleg319-client.virtnet: 1 32402 0.58 MB/sec execute 238 sec latencoleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 01:22:35 (1708842155) targets are mounted 01:22:35 (1708842155) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec y 12369.615 ms oleg319-client.virtnet: 1 32402 0.58 MB/sec execute 239 sec latency 13369.869 ms oleg319-client.virtnet: 1 32402 0.58 MB/sec execute 240 sec latency 14370.063 ms oleg319-client.virtnet: 1 32402 0.57 MB/sec execute 241 sec latency 15370.181 ms oleg319-client.virtnet: 1 32402 0.57 MB/sec execute 242 sec latency 16370.353 ms oleg319-client.virtnet: 1 32402 0.57 MB/sec execute 243 sec latency 17370.607 ms oleg319-client.virtnet: 1 32542 0.57 MB/sec execute 244 sec latency 17893.860 ms oleg319-client.virtnet: 1 32845 0.57 MB/sec execute 245 sec latency 67.335 ms oleg319-client.virtnet: 1 33148 0.57 MB/sec execute 246 sec latency 35.189 ms oleg319-client.virtnet: 1 33548 0.58 MB/sec execute 247 sec latency 35.500 ms oleg319-client.virtnet: 1 33894 0.58 MB/sec execute 248 sec latency 44.206 ms oleg319-client.virtnet: 1 34354 0.58 MB/sec execute 249 sec latency 35.420 ms oleg319-client.virtnet: 1 34937 0.60 MB/sec execute 250 sec latency 32.957 ms oleg319-client.virtnet: 1 35743 0.63 MB/sec execute 251 sec latency 26.441 ms oleg319-client.virtnet: 1 36169 0.63 MB/sec execute 252 sec latency 14.746 ms oleg319-client.virtnet: 1 36495 0.63 MB/sec execute 253 sec latency 39.070 ms oleg319-client.virtnet: 1 37148 0.64 MB/sec execute 254 sec latency 28.786 ms oleg319-client.virtnet: 1 37755 0.64 MB/sec execute 255 sec latency 32.036 ms oleg319-client.virtnet: 1 38530 0.67 MB/sec execute 256 sec latency 24.975 ms oleg319-client.virtnet: 1 39324 0.69 MB/sec execute 257 sec latency 26.663 ms oleg319-client.virtnet: 1 39739 0.69 MB/sec execute 258 sec latency 45.303 ms oleg319-client.virtnet: 1 40115 0.69 MB/sec execute 259 sec latency 30.736 ms oleg319-client.virtnet: 1 40677 0.70 MB/sec execute 260 sec latency 32.171 ms oleg319-client.virtnet: 1 41324 0.70 MB/sec execute 261 sec latency 36.061 ms oleg319-client.virtnet: 1 42043 0.72 MB/sec execute 262 sec latency 32.422 ms oleg319-client.virtnet: 1 42840 0.74 MB/sec execute 263 sec latency 21.174 ms oleg319-client.virtnet: 1 43200 0.74 MB/sec execute 264 sec latency 11.194 ms oleg319-client.virtnet: 1 43575 0.74 MB/sec execute 265 sec latency 44.347 ms oleg319-client.virtnet: 1 44061 0.76 MB/sec execute 266 sec latency 37.440 ms oleg319-client.virtnet: 1 44480 0.75 MB/sec execute 267 sec latency 41.976 ms oleg319-client.virtnet: 1 44996 0.76 MB/sec execute 268 sec latency 35.836 ms oleg319-client.virtnet: 1 45579 0.78 MB/sec execute 269 sec latency 36.592 ms oleg319-client.virtnet: 1 46263 0.79 MB/sec execute 270 sec latency 37.003 ms oleg319-client.virtnet: 1 46574 0.79 MB/sec execute 271 sec latency 13.379 ms oleg319-client.virtnet: 1 46892 0.79 MB/sec execute 272 sec latency 53.871 ms oleg319-client.virtnet: 1 47220 0.79 MB/sec execute 273 sec latency 49.976 ms oleg319-client.virtnet: 1 47656 0.80 MB/sec execute 274 sec latency 40.944 ms oleg319-client.virtnet: 1 48021 0.80 MB/sec execute 275 sec latency 36.170 ms oleg319-client.virtnet: 1 48507 0.80 MB/sec execute 276 sec latency 36.979 ms oleg319-client.virtnet: 1 49041 0.82 MB/sec execute 277 sec latency 47.893 ms oleg319-client.virtnet: 1 49647 0.84 MB/sec execute 278 sec latency 40.015 ms oleg319-client.virtnet: 1 50015 0.84 MB/sec execute 279 sec latency 12.791 ms oleg319-client.virtnet: 1 50365 0.84 MB/sec execute 280 sec latency 14.353 ms oleg319-client.virtnet: 1 50591 0.84 MB/sec execute 281 sec latency 55.449 ms oleg319-client.virtnet: 1 50941 0.84 MB/sec execute 282 sec latency 38.542 ms oleg319-client.virtnet: 1 51363 0.85 MB/sec execute 283 sec latency 48.381 ms oleg319-client.virtnet: 1 51789 0.84 MB/sec execute 284 sec latency 36.544 ms oleg319-client.virtnet: 1 52182 0.85 MB/sec execute 285 sec latency 41.861 ms oleg319-client.virtnet: 1 52884 0.87 MB/sec execute 286 sec latency 43.015 ms oleg319-client.virtnet: 1 53402 0.88 MB/sec execute 287 sec latency 36.424 ms oleg319-client.virtnet: 1 53700 0.88 MB/sec execute 288 sec latency 13.703 ms oleg319-client.virtnet: 1 54027 0.88 MB/sec execute 289 sec latency 72.759 ms oleg319-client.virtnet: 1 54318 0.88 MB/sec execute 290 sec latency 39.309 ms oleg319-client.virtnet: 1 54739 0.89 MB/sec execute 291 sec latency 49.674 ms oleg319-client.virtnet: 1 55084 0.89 MB/sec execute 292 sec latency 36.213 ms oleg319-client.virtnet: 1 55528 0.89 MB/sec execute 293 sec latency 35.350 ms oleg319-client.virtnet: 1 55845 0.89 MB/sec execute 294 sec latency 64.127 ms oleg319-client.virtnet: 1 56539 0.91 MB/sec execute 295 sec latency 50.526 ms oleg319-client.virtnet: 1 57021 0.92 MB/sec execute 296 sec latency 20.624 ms oleg319-client.virtnet: 1 57292 0.92 MB/sec execute 297 sec latency 13.564 ms oleg319-client.virtnet: 1 57590 0.92 MB/sec execute 298 sec latency 60.526 ms oleg319-client.virtnet: 1 57914 0.92 MB/sec execute 299 sec latency 42.832 ms oleg319-client.virtnet: 1 cleanup 300 sec oleg319-client.virtnet: 0 cleanup 300 sec oleg319-client.virtnet: oleg319-client.virtnet: Operation Count AvgLat MaxLat oleg319-client.virtnet: ---------------------------------------- oleg319-client.virtnet: NTCreateX 8954 12.613 18120.619 oleg319-client.virtnet: Close 6552 9.629 17893.818 oleg319-client.virtnet: Rename 378 9.937 32.675 oleg319-client.virtnet: Unlink 1831 22.780 18464.301 oleg319-client.virtnet: Qpathinfo 8073 6.383 18255.579 oleg319-client.virtnet: Qfileinfo 1391 0.158 2.438 oleg319-client.virtnet: Qfsinfo 1491 0.146 4.704 oleg319-client.virtnet: Sfileinfo 734 4.950 9.923 oleg319-client.virtnet: Find 3099 0.566 22.478 oleg319-client.virtnet: WriteX 4384 0.962 5.370 oleg319-client.virtnet: ReadX 13882 0.021 2.161 oleg319-client.virtnet: LockX 28 1.458 1.948 oleg319-client.virtnet: UnlockX 28 1.489 2.052 oleg319-client.virtnet: Flush 627 25.788 72.745 oleg319-client.virtnet: oleg319-client.virtnet: Throughput 0.917778 MB/sec 1 clients 1 procs max_latency=18464.327 ms oleg319-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg319-client.virtnet at Sun Feb 25 01:23:36 EST 2024 with return code 0 oleg319-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg319-client.virtnet oleg319-client.virtnet: /mnt/lustre/d70b.replay-single/oleg319-client.virtnet /mnt/lustre/d70b.replay-single/oleg319-client.virtnet oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg319-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg319-client.virtnet: removed directory: 'clients/client0' oleg319-client.virtnet: removed directory: 'clients' oleg319-client.virtnet: removed 'client.txt' oleg319-client.virtnet: /mnt/lustre/d70b.replay-single/oleg319-client.virtnet oleg319-client.virtnet: dbench successfully finished