== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 17:47:17 (1743457637) Starting client oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre Started clients oleg242-client.virtnet: 192.168.202.142@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg242-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13465 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2468 1285220 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1285524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26160 3580860 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1884 3592924 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28044 7173784 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg242-server Stopping /mnt/lustre-mds1 (opts:) on oleg242-server 17:47:24 (1743457644) shut down facet: mds1 facet_host: oleg242-server facet_failover_host: oleg242-server Failover mds1 to oleg242-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 17:47:39 (1743457659) targets are mounted 17:47:39 (1743457659) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1285260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2132 1285556 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26476 3569664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1728 3592852 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28204 7162516 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg242-server Stopping /mnt/lustre-mds2 (opts:) on oleg242-server 17:47:51 (1743457671) shut down facet: mds2 facet_host: oleg242-server facet_failover_host: oleg242-server oleg242-client.virtnet: looking for dbench program oleg242-client.virtnet: /usr/bin/dbench oleg242-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg242-client.virtnet oleg242-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg242-client.virtnet' oleg242-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg242-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg242-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg242-client.virtnet at Mon Mar 31 17:47:18 EDT 2025 oleg242-client.virtnet: waiting for dbench pid 13490 oleg242-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg242-client.virtnet: oleg242-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg242-client.virtnet: failed to create barrier semaphore oleg242-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg242-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg242-client.virtnet: releasing clients oleg242-client.virtnet: 1 531 18.45 MB/sec warmup 1 sec latency 12.380 ms oleg242-client.virtnet: 1 1190 11.52 MB/sec warmup 2 sec latency 9.707 ms oleg242-client.virtnet: 1 1848 8.79 MB/sec warmup 3 sec latency 10.002 ms oleg242-client.virtnet: 1 2489 7.18 MB/sec warmup 4 sec latency 9.553 ms oleg242-client.virtnet: 1 3673 7.74 MB/sec warmup 5 sec latency 10.065 ms oleg242-client.virtnet: 1 4083 6.68 MB/sec warmup 6 sec latency 131.366 ms oleg242-client.virtnet: 1 4083 5.73 MB/sec warmup 7 sec latency 1131.568 ms oleg242-client.virtnet: 1 4083 5.01 MB/sec warmup 8 sec latency 2131.756 ms oleg242-client.virtnet: 1 4083 4.45 MB/sec warmup 9 sec latency 3131.968 ms oleg242-client.virtnet: 1 4083 4.01 MB/sec warmup 10 sec latency 4132.159 ms oleg242-client.virtnet: 1 4083 3.64 MB/sec warmup 11 sec latency 5132.360 ms oleg242-client.virtnet: 1 4083 3.34 MB/sec warmup 12 sec latency 6133.070 ms oleg242-client.virtnet: 1 4083 3.08 MB/sec warmup 13 sec latency 7133.239 ms oleg242-client.virtnet: 1 4083 2.86 MB/sec warmup 14 sec latency 8133.401 ms oleg242-client.virtnet: 1 4083 2.67 MB/sec warmup 15 sec latency 9133.601 ms oleg242-client.virtnet: 1 4083 2.51 MB/sec warmup 16 sec latency 10133.772 ms oleg242-client.virtnet: 1 4083 2.36 MB/sec warmup 17 sec latency 11133.931 ms oleg242-client.virtnet: 1 4083 2.23 MB/sec warmup 18 sec latency 12134.179 ms oleg242-client.virtnet: 1 4083 2.11 MB/sec warmup 19 sec latency 13134.386 ms oleg242-client.virtnet: 1 4083 2.00 MB/sec warmup 20 sec latency 14134.575 ms oleg242-client.virtnet: 1 4083 1.91 MB/sec warmup 21 sec latency 15134.759 ms oleg242-client.virtnet: 1 4083 1.82 MB/sec warmup 22 sec latency 16134.903 ms oleg242-client.virtnet: 1 4083 1.74 MB/sec warmup 23 sec latency 17135.061 ms oleg242-client.virtnet: 1 4083 1.67 MB/sec warmup 24 sec latency 18135.262 ms oleg242-client.virtnet: 1 4434 1.63 MB/sec warmup 25 sec latency 18529.436 ms oleg242-client.virtnet: 1 5005 1.74 MB/sec warmup 26 sec latency 14.944 ms oleg242-client.virtnet: 1 5483 1.69 MB/sec warmup 27 sec latency 14.374 ms oleg242-client.virtnet: 1 6049 1.71 MB/sec warmup 28 sec latency 10.323 ms oleg242-client.virtnet: 1 7057 1.98 MB/sec warmup 29 sec latency 12.429 ms oleg242-client.virtnet: 1 7483 1.97 MB/sec warmup 30 sec latency 12.241 ms oleg242-client.virtnet: 1 7919 1.93 MB/sec warmup 31 sec latency 16.054 ms oleg242-client.virtnet: 1 8359 1.92 MB/sec warmup 32 sec latency 17.763 ms oleg242-client.virtnet: 1 8537 1.95 MB/sec warmup 33 sec latency 749Failover mds2 to oleg242-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0001 17:48:05 (1743457685) targets are mounted 17:48:05 (1743457685) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3567852 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13212 3592788 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160640 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg242-server Stopping /mnt/lustre-mds1 (opts:) on oleg242-server 17:48:17 (1743457697) shut down facet: mds1 facet_host: oleg242-server facet_failover_host: oleg242-server Failover mds1 to oleg242-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 17:48:31 (1743457711) targets are mounted 17:48:31 (1743457711) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec .574 ms oleg242-client.virtnet: 1 8537 1.89 MB/sec warmup 34 sec latency 1749.708 ms oleg242-client.virtnet: 1 8537 1.84 MB/sec warmup 35 sec latency 2749.877 ms oleg242-client.virtnet: 1 8537 1.78 MB/sec warmup 36 sec latency 3750.079 ms oleg242-client.virtnet: 1 8537 1.74 MB/sec warmup 37 sec latency 4750.309 ms oleg242-client.virtnet: 1 8537 1.69 MB/sec warmup 38 sec latency 5750.485 ms oleg242-client.virtnet: 1 8537 1.65 MB/sec warmup 39 sec latency 6750.674 ms oleg242-client.virtnet: 1 8537 1.61 MB/sec warmup 40 sec latency 7750.871 ms oleg242-client.virtnet: 1 8537 1.57 MB/sec warmup 41 sec latency 8751.067 ms oleg242-client.virtnet: 1 8537 1.53 MB/sec warmup 42 sec latency 9751.337 ms oleg242-client.virtnet: 1 8537 1.49 MB/sec warmup 43 sec latency 10751.468 ms oleg242-client.virtnet: 1 8537 1.46 MB/sec warmup 44 sec latency 11751.586 ms oleg242-client.virtnet: 1 8537 1.43 MB/sec warmup 45 sec latency 12751.748 ms oleg242-client.virtnet: 1 8537 1.40 MB/sec warmup 46 sec latency 13751.973 ms oleg242-client.virtnet: 1 8537 1.37 MB/sec warmup 47 sec latency 14752.148 ms oleg242-client.virtnet: 1 8537 1.34 MB/sec warmup 48 sec latency 15752.276 ms oleg242-client.virtnet: 1 8537 1.31 MB/sec warmup 49 sec latency 16752.416 ms oleg242-client.virtnet: 1 8537 1.28 MB/sec warmup 50 sec latency 17752.587 ms oleg242-client.virtnet: 1 8540 1.26 MB/sec warmup 51 sec latency 18737.700 ms oleg242-client.virtnet: 1 8922 1.24 MB/sec warmup 52 sec latency 15.280 ms oleg242-client.virtnet: 1 9525 1.25 MB/sec warmup 53 sec latency 11.766 ms oleg242-client.virtnet: 1 10468 1.36 MB/sec warmup 54 sec latency 13.293 ms oleg242-client.virtnet: 1 11034 1.42 MB/sec warmup 55 sec latency 11.996 ms oleg242-client.virtnet: 1 11531 1.41 MB/sec warmup 56 sec latency 10.824 ms oleg242-client.virtnet: 1 12078 1.46 MB/sec warmup 57 sec latency 11.162 ms oleg242-client.virtnet: 1 12520 1.44 MB/sec warmup 58 sec latency 12.570 ms oleg242-client.virtnet: 1 13103 1.45 MB/sec warmup 59 sec latency 9.828 ms oleg242-client.virtnet: 1 14641 4.61 MB/sec execute 1 sec latency 11.195 ms oleg242-client.virtnet: 1 15181 2.78 MB/sec execute 2 sec latency 10.529 ms oleg242-client.virtnet: 1 15852 3.29 MB/sec execute 3 sec latency 9.589 ms oleg242-client.virtnet: 1 16528 2.87 MB/sec execute 4 sec latency 10.714 ms oleg242-client.virtnet: 1 17467 3.80 MB/sec execute 5 sec latency 9.717 ms oleg242-client.virtnet: 1 18182 3.95 MB/sec execute 6 sec latency 7.780 ms oleg242-client.virtnet: 1 18836 3.67 MB/sec execute 7 sec latency 9.600 ms oleg242-client.virtnet: 1 19475 3.63 MB/sec execute 8 sec latency 11.330 ms oleg242-client.virtnet: 1 20162 3.42 MB/sec execute 9 sec latency 8.763 ms oleg242-client.virtnet: 1 21293 4.12 MB/sec execute 10 sec latency 8.973 ms oleg242-client.virtnet: 1 21826 3.90 MB/sec execute 11 sec latency 9.436 ms oleg242-client.virtnet: 1 22464 3.74 MB/sec execute 12 sec latency 10.521 ms oleg242-client.virtnet: 1 23107 3.71 MB/sec execute 13 sec latency 10.657 ms oleg242-client.virtnet: 1 23778 3.58 MB/sec execute 14 sec latency 8.976 ms oleg242-client.virtnet: 1 24943 4.04 MB/sec execute 15 sec latency 9.434 ms oleg242-client.virtnet: 1 25495 3.88 MB/sec execute 16 sec latency 8.530 ms oleg242-client.virtnet: 1 26102 3.77 MB/sec execute 17 sec latency 13.315 ms oleg242-client.virtnet: 1 26738 3.75 MB/sec execute 18 sec latency 10.321 ms oleg242-client.virtnet: 1 27378 3.67 MB/sec execute 19 sec latUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36632 3568972 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12984 3592856 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7161828 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg242-server Stopping /mnt/lustre-mds2 (opts:) on oleg242-server 17:48:43 (1743457723) shut down facet: mds2 facet_host: oleg242-server facet_failover_host: oleg242-server Failover mds2 to oleg242-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0001 17:48:56 (1743457736) targets are mounted 17:48:56 (1743457736) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3568088 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13320 3592928 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49724 7161016 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg242-server Stopping /mnt/lustre-mds1 (opts:) on oleg242-server 17:49:08 (1743457748) shut down facet: mds1 facet_host: oleg242-server facet_failover_host: oleg242-server Failover mds1 to oleg242-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all ency 14.207 ms oleg242-client.virtnet: 1 28514 3.99 MB/sec execute 20 sec latency 10.249 ms oleg242-client.virtnet: 1 29103 3.87 MB/sec execute 21 sec latency 8.072 ms oleg242-client.virtnet: 1 29667 3.79 MB/sec execute 22 sec latency 10.443 ms oleg242-client.virtnet: 1 30328 3.77 MB/sec execute 23 sec latency 9.293 ms oleg242-client.virtnet: 1 31135 3.84 MB/sec execute 24 sec latency 9.487 ms oleg242-client.virtnet: 1 31404 3.75 MB/sec execute 25 sec latency 754.350 ms oleg242-client.virtnet: 1 31404 3.60 MB/sec execute 26 sec latency 1754.540 ms oleg242-client.virtnet: 1 31404 3.47 MB/sec execute 27 sec latency 2754.735 ms oleg242-client.virtnet: 1 31404 3.35 MB/sec execute 28 sec latency 3754.881 ms oleg242-client.virtnet: 1 31404 3.23 MB/sec execute 29 sec latency 4755.063 ms oleg242-client.virtnet: 1 31404 3.12 MB/sec execute 30 sec latency 5755.215 ms oleg242-client.virtnet: 1 31404 3.02 MB/sec execute 31 sec latency 6755.378 ms oleg242-client.virtnet: 1 31404 2.93 MB/sec execute 32 sec latency 7755.583 ms oleg242-client.virtnet: 1 31404 2.84 MB/sec execute 33 sec latency 8755.739 ms oleg242-client.virtnet: 1 31404 2.76 MB/sec execute 34 sec latency 9755.923 ms oleg242-client.virtnet: 1 31404 2.68 MB/sec execute 35 sec latency 10756.121 ms oleg242-client.virtnet: 1 31404 2.60 MB/sec execute 36 sec latency 11756.291 ms oleg242-client.virtnet: 1 31404 2.53 MB/sec execute 37 sec latency 12756.399 ms oleg242-client.virtnet: 1 31404 2.47 MB/sec execute 38 sec latency 13756.570 ms oleg242-client.virtnet: 1 31404 2.40 MB/sec execute 39 sec latency 14756.748 ms oleg242-client.virtnet: 1 31404 2.34 MB/sec execute 40 sec latency 15756.943 ms oleg242-client.virtnet: 1 31404 2.29 MB/sec execute 41 sec latency 16757.146 ms oleg242-client.virtnet: 1 31404 2.23 MB/sec execute 42 sec latency 17757.317 ms oleg242-client.virtnet: 1 31960 2.29 MB/sec execute 43 sec latency 18303.502 ms oleg242-client.virtnet: 1 32458 2.28 MB/sec execute 44 sec latency 10.401 ms oleg242-client.virtnet: 1 33049 2.27 MB/sec execute 45 sec latency 12.223 ms oleg242-client.virtnet: 1 33691 2.29 MB/sec execute 46 sec latency 10.387 ms oleg242-client.virtnet: 1 34352 2.28 MB/sec execute 47 sec latency 9.883 ms oleg242-client.virtnet: 1 35453 2.45 MB/sec execute 48 sec latency 10.488 ms oleg242-client.virtnet: 1 35997 2.44 MB/sec execute 49 sec latency 8.630 ms oleg242-client.virtnet: 1 36642 2.43 MB/sec execute 50 sec latency 11.818 ms oleg242-client.virtnet: 1 37303 2.45 MB/sec execute 51 sec latency 10.196 ms oleg242-client.virtnet: 1 37997 2.45 MB/sec execute 52 sec latency 9.088 ms oleg242-client.virtnet: 1 39180 2.59 MB/sec execute 53 sec latency 8.763 ms oleg242-client.virtnet: 1 39773 2.57 MB/sec execute 54 sec latency 10.438 ms oleg242-client.virtnet: 1 40459 2.61 MB/sec execute 55 sec latency 10.783 ms oleg242-client.virtnet: 1 41049 2.57 MB/sec execute 56 sec latency 10.692 ms oleg242-client.virtnet: 1 41928 2.63 MB/sec execute 57 sec latency 9.681 ms oleg242-client.virtnet: 1 42849 2.71 MB/sec execute 58 sec latency 9.614 ms oleg242-client.virtnet: 1 43389 2.68 MB/sec execute 59 sec latency 10.536 ms oleg242-client.virtnet: 1 44048 2.71 MB/sec execute 60 sec latency 9.362 ms oleg242-client.virtnet: 1 44619 2.68 MB/sec execute 61 sec latency 10.383 ms oleg242-client.virtnet: 1 45471 2.72 MB/sec execute 62 sec latency 11.878 ms oleg242-client.virtnet: 1 46380 2.80 MB/sec execute 63 sec latency 9.153 ms oleg242-client.virtnet: 1 46926 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 17:49:22 (1743457762) targets are mounted 17:49:22 (1743457762) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36528 3569952 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13084 3592908 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49612 7162860 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg242-server Stopping /mnt/lustre-mds2 (opts:) on oleg242-server 17:49:34 (1743457774) shut down facet: mds2 facet_host: oleg242-server facet_failover_host: oleg242-server Failover mds2 to oleg242-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0001 17:49:48 (1743457788) targets are mounted 17:49:48 (1743457788) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36544 3569700 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13224 3590792 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49768 7160492 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg242-server Stopping /mnt/lustre-mds1 (opts:) on oleg242-server 17:50:00 (1743457800) shut down facet: mds1 facet_host: oleg242-server facet_failover_host: oleg242-server 2.77 MB/sec execute 64 sec latency 12.732 ms oleg242-client.virtnet: 1 47564 2.80 MB/sec execute 65 sec latency 13.083 ms oleg242-client.virtnet: 1 48151 2.76 MB/sec execute 66 sec latency 9.993 ms oleg242-client.virtnet: 1 48755 2.76 MB/sec execute 67 sec latency 83.470 ms oleg242-client.virtnet: 1 49830 2.86 MB/sec execute 68 sec latency 8.942 ms oleg242-client.virtnet: 1 50362 2.84 MB/sec execute 69 sec latency 10.897 ms oleg242-client.virtnet: 1 50941 2.83 MB/sec execute 70 sec latency 10.671 ms oleg242-client.virtnet: 1 51585 2.83 MB/sec execute 71 sec latency 11.967 ms oleg242-client.virtnet: 1 52406 2.87 MB/sec execute 72 sec latency 12.081 ms oleg242-client.virtnet: 1 53416 2.93 MB/sec execute 73 sec latency 9.682 ms oleg242-client.virtnet: 1 53965 2.91 MB/sec execute 74 sec latency 12.328 ms oleg242-client.virtnet: 1 54621 2.93 MB/sec execute 75 sec latency 9.995 ms oleg242-client.virtnet: 1 54845 2.90 MB/sec execute 76 sec latency 586.651 ms oleg242-client.virtnet: 1 54845 2.86 MB/sec execute 77 sec latency 1586.801 ms oleg242-client.virtnet: 1 54845 2.82 MB/sec execute 78 sec latency 2586.956 ms oleg242-client.virtnet: 1 54845 2.79 MB/sec execute 79 sec latency 3587.117 ms oleg242-client.virtnet: 1 54845 2.75 MB/sec execute 80 sec latency 4587.259 ms oleg242-client.virtnet: 1 54845 2.72 MB/sec execute 81 sec latency 5587.454 ms oleg242-client.virtnet: 1 54845 2.69 MB/sec execute 82 sec latency 6587.704 ms oleg242-client.virtnet: 1 54845 2.65 MB/sec execute 83 sec latency 7587.917 ms oleg242-client.virtnet: 1 54845 2.62 MB/sec execute 84 sec latency 8588.131 ms oleg242-client.virtnet: 1 54845 2.59 MB/sec execute 85 sec latency 9588.252 ms oleg242-client.virtnet: 1 54845 2.56 MB/sec execute 86 sec latency 10588.403 ms oleg242-client.virtnet: 1 54845 2.53 MB/sec execute 87 sec latency 11588.571 ms oleg242-client.virtnet: 1 54845 2.50 MB/sec execute 88 sec latency 12588.684 ms oleg242-client.virtnet: 1 54845 2.47 MB/sec execute 89 sec latency 13588.860 ms oleg242-client.virtnet: 1 54845 2.45 MB/sec execute 90 sec latency 14589.031 ms oleg242-client.virtnet: 1 54845 2.42 MB/sec execute 91 sec latency 15589.147 ms oleg242-client.virtnet: 1 54845 2.39 MB/sec execute 92 sec latency 16589.320 ms oleg242-client.virtnet: 1 54845 2.37 MB/sec execute 93 sec latency 17589.485 ms oleg242-client.virtnet: 1 54904 2.34 MB/sec execute 94 sec latency 18483.219 ms oleg242-client.virtnet: 1 55575 2.34 MB/sec execute 95 sec latency 9.847 ms oleg242-client.virtnet: 1 56601 2.39 MB/sec execute 96 sec latency 12.134 ms oleg242-client.virtnet: 1 57246 2.41 MB/sec execute 97 sec latency 18.697 ms oleg242-client.virtnet: 1 57853 2.41 MB/sec execute 98 sec latency 12.506 ms oleg242-client.virtnet: 1 58507 2.42 MB/sec execute 99 sec latency 10.068 ms oleg242-client.virtnet: 1 59199 2.41 MB/sec execute 100 sec latency 8.202 ms oleg242-client.virtnet: 1 60284 2.49 MB/sec execute 101 sec latency 10.572 ms oleg242-client.virtnet: 1 60814 2.48 MB/sec execute 102 sec latency 8.634 ms oleg242-client.virtnet: 1 61438 2.48 MB/sec execute 103 sec latency 11.015 ms oleg242-client.virtnet: 1 62082 2.49 MB/sec execute 104 sec latency 9.622 ms oleg242-client.virtnet: 1 62785 2.48 MB/sec execute 105 sec latency 9.565 ms oleg242-client.virtnet: 1 63953 2.56 MB/sec execute 106 sec latency 9.204 ms oleg242-client.virtnet: 1 64543 2.55 MB/sec execute 107 sec latency 7.868 ms oleg242-client.virtnet: 1 65122 2.54 MB/sec execute 108 sec latency 10.465 ms oleg242-Failover mds1 to oleg242-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 17:50:14 (1743457814) targets are mounted 17:50:14 (1743457814) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1285584 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36404 3569708 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13320 3592788 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49724 7162496 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg242-server Stopping /mnt/lustre-mds2 (opts:) on oleg242-server 17:50:25 (1743457825) shut down facet: mds2 facet_host: oleg242-server facet_failover_host: oleg242-server Failover mds2 to oleg242-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0001 17:50:39 (1743457839) targets are mounted 17:50:39 (1743457839) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36640 3569728 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12932 3593228 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49572 7162956 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg242-server Stopping /mnt/lustre-mds1 (opts:) on oleg242-server client.virtnet: 1 65778 2.55 MB/sec execute 109 sec latency 12.613 ms oleg242-client.virtnet: 1 66478 2.55 MB/sec execute 110 sec latency 9.643 ms oleg242-client.virtnet: 1 67569 2.61 MB/sec execute 111 sec latency 9.472 ms oleg242-client.virtnet: 1 68163 2.61 MB/sec execute 112 sec latency 10.202 ms oleg242-client.virtnet: 1 68821 2.62 MB/sec execute 113 sec latency 9.185 ms oleg242-client.virtnet: 1 69415 2.60 MB/sec execute 114 sec latency 11.183 ms oleg242-client.virtnet: 1 70193 2.63 MB/sec execute 115 sec latency 9.863 ms oleg242-client.virtnet: 1 71206 2.67 MB/sec execute 116 sec latency 8.854 ms oleg242-client.virtnet: 1 71770 2.66 MB/sec execute 117 sec latency 12.822 ms oleg242-client.virtnet: 1 72451 2.68 MB/sec execute 118 sec latency 9.192 ms oleg242-client.virtnet: 1 73078 2.66 MB/sec execute 119 sec latency 9.717 ms oleg242-client.virtnet: 1 73977 2.69 MB/sec execute 120 sec latency 11.446 ms oleg242-client.virtnet: 1 74852 2.72 MB/sec execute 121 sec latency 11.136 ms oleg242-client.virtnet: 1 75428 2.71 MB/sec execute 122 sec latency 11.963 ms oleg242-client.virtnet: 1 76111 2.72 MB/sec execute 123 sec latency 9.591 ms oleg242-client.virtnet: 1 76747 2.71 MB/sec execute 124 sec latency 9.208 ms oleg242-client.virtnet: 1 77152 2.70 MB/sec execute 125 sec latency 461.614 ms oleg242-client.virtnet: 1 78310 2.77 MB/sec execute 126 sec latency 8.934 ms oleg242-client.virtnet: 1 78644 2.75 MB/sec execute 127 sec latency 319.283 ms oleg242-client.virtnet: 1 78644 2.73 MB/sec execute 128 sec latency 1319.479 ms oleg242-client.virtnet: 1 78644 2.70 MB/sec execute 129 sec latency 2319.606 ms oleg242-client.virtnet: 1 78644 2.68 MB/sec execute 130 sec latency 3319.760 ms oleg242-client.virtnet: 1 78644 2.66 MB/sec execute 131 sec latency 4319.957 ms oleg242-client.virtnet: 1 78644 2.64 MB/sec execute 132 sec latency 5320.131 ms oleg242-client.virtnet: 1 78644 2.62 MB/sec execute 133 sec latency 6320.315 ms oleg242-client.virtnet: 1 78644 2.60 MB/sec execute 134 sec latency 7320.512 ms oleg242-client.virtnet: 1 78644 2.58 MB/sec execute 135 sec latency 8320.675 ms oleg242-client.virtnet: 1 78644 2.57 MB/sec execute 136 sec latency 9320.832 ms oleg242-client.virtnet: 1 78644 2.55 MB/sec execute 137 sec latency 10321.002 ms oleg242-client.virtnet: 1 78644 2.53 MB/sec execute 138 sec latency 11321.168 ms oleg242-client.virtnet: 1 78644 2.51 MB/sec execute 139 sec latency 12321.286 ms oleg242-client.virtnet: 1 78644 2.49 MB/sec execute 140 sec latency 13321.440 ms oleg242-client.virtnet: 1 78644 2.47 MB/sec execute 141 sec latency 14321.562 ms oleg242-client.virtnet: 1 78644 2.46 MB/sec execute 142 sec latency 15321.698 ms oleg242-client.virtnet: 1 78644 2.44 MB/sec execute 143 sec latency 16321.816 ms oleg242-client.virtnet: 1 78644 2.42 MB/sec execute 144 sec latency 17321.940 ms oleg242-client.virtnet: 1 78644 2.41 MB/sec execute 145 sec latency 18322.046 ms oleg242-client.virtnet: 1 79045 2.40 MB/sec execute 146 sec latency 18587.201 ms oleg242-client.virtnet: 1 79650 2.41 MB/sec execute 147 sec latency 10.805 ms oleg242-client.virtnet: 1 80150 2.40 MB/sec execute 148 sec latency 18.901 ms oleg242-client.virtnet: 1 81056 2.43 MB/sec execute 149 sec latency 9.360 ms oleg242-client.virtnet: 1 81930 2.45 MB/sec execute 150 sec latency 9.783 ms oleg242-client.virtnet: 1 82509 2.44 MB/sec execute 151 sec latency 9.956 ms oleg242-client.virtnet: 1 83193 2.45 MB/sec execute 152 sec latency 10.466 ms oleg242-client.virtnet: 1 83838 2.44 MB/sec execute17:50:51 (1743457851) shut down facet: mds1 facet_host: oleg242-server facet_failover_host: oleg242-server Failover mds1 to oleg242-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 17:51:05 (1743457865) targets are mounted 17:51:05 (1743457865) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2100 1285588 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36692 3569660 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13072 3592984 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49764 7162644 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg242-server Stopping /mnt/lustre-mds2 (opts:) on oleg242-server 17:51:16 (1743457876) shut down facet: mds2 facet_host: oleg242-server facet_failover_host: oleg242-server Failover mds2 to oleg242-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0001 17:51:30 (1743457890) targets are mounted 17:51:30 (1743457890) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 153 sec latency 9.326 ms oleg242-client.virtnet: 1 84609 2.47 MB/sec execute 154 sec latency 16.979 ms oleg242-client.virtnet: 1 85389 2.50 MB/sec execute 155 sec latency 11.676 ms oleg242-client.virtnet: 1 85935 2.48 MB/sec execute 156 sec latency 10.373 ms oleg242-client.virtnet: 1 86634 2.50 MB/sec execute 157 sec latency 9.566 ms oleg242-client.virtnet: 1 87267 2.49 MB/sec execute 158 sec latency 10.659 ms oleg242-client.virtnet: 1 88175 2.51 MB/sec execute 159 sec latency 9.199 ms oleg242-client.virtnet: 1 89073 2.54 MB/sec execute 160 sec latency 9.770 ms oleg242-client.virtnet: 1 89661 2.53 MB/sec execute 161 sec latency 10.611 ms oleg242-client.virtnet: 1 90366 2.54 MB/sec execute 162 sec latency 8.865 ms oleg242-client.virtnet: 1 91102 2.54 MB/sec execute 163 sec latency 9.183 ms oleg242-client.virtnet: 1 92227 2.58 MB/sec execute 164 sec latency 9.678 ms oleg242-client.virtnet: 1 92781 2.58 MB/sec execute 165 sec latency 8.748 ms oleg242-client.virtnet: 1 93426 2.57 MB/sec execute 166 sec latency 9.867 ms oleg242-client.virtnet: 1 94046 2.58 MB/sec execute 167 sec latency 11.120 ms oleg242-client.virtnet: 1 94739 2.58 MB/sec execute 168 sec latency 9.237 ms oleg242-client.virtnet: 1 95886 2.62 MB/sec execute 169 sec latency 9.700 ms oleg242-client.virtnet: 1 96445 2.61 MB/sec execute 170 sec latency 10.279 ms oleg242-client.virtnet: 1 97052 2.61 MB/sec execute 171 sec latency 11.774 ms oleg242-client.virtnet: 1 97690 2.62 MB/sec execute 172 sec latency 10.907 ms oleg242-client.virtnet: 1 98425 2.62 MB/sec execute 173 sec latency 10.135 ms oleg242-client.virtnet: 1 99535 2.66 MB/sec execute 174 sec latency 9.597 ms oleg242-client.virtnet: 1 100084 2.65 MB/sec execute 175 sec latency 9.860 ms oleg242-client.virtnet: 1 100311 2.64 MB/sec execute 176 sec latency 561.136 ms oleg242-client.virtnet: 1 100945 2.65 MB/sec execute 177 sec latency 11.612 ms oleg242-client.virtnet: 1 101545 2.64 MB/sec execute 178 sec latency 30.454 ms oleg242-client.virtnet: 1 101545 2.62 MB/sec execute 179 sec latency 1030.649 ms oleg242-client.virtnet: 1 101545 2.61 MB/sec execute 180 sec latency 2030.834 ms oleg242-client.virtnet: 1 101545 2.59 MB/sec execute 181 sec latency 3030.998 ms oleg242-client.virtnet: 1 101545 2.58 MB/sec execute 182 sec latency 4031.199 ms oleg242-client.virtnet: 1 101545 2.57 MB/sec execute 183 sec latency 5031.384 ms oleg242-client.virtnet: 1 101545 2.55 MB/sec execute 184 sec latency 6031.582 ms oleg242-client.virtnet: 1 101545 2.54 MB/sec execute 185 sec latency 7031.782 ms oleg242-client.virtnet: 1 101545 2.52 MB/sec execute 186 sec latency 8031.949 ms oleg242-client.virtnet: 1 101545 2.51 MB/sec execute 187 sec latency 9032.135 ms oleg242-client.virtnet: 1 101545 2.50 MB/sec execute 188 sec latency 10032.301 ms oleg242-client.virtnet: 1 101545 2.48 MB/sec execute 189 sec latency 11032.415 ms oleg242-client.virtnet: 1 101545 2.47 MB/sec execute 190 sec latency 12032.533 ms oleg242-client.virtnet: 1 101545 2.46 MB/sec execute 191 sec latency 13032.722 ms oleg242-client.virtnet: 1 101545 2.44 MB/sec execute 192 sec latency 14032.844 ms oleg242-client.virtnet: 1 101545 2.43 MB/sec execute 193 sec latency 15033.014 ms oleg242-client.virtnet: 1 101545 2.42 MB/sec execute 194 sec latency 16033.173 ms oleg242-client.virtnet: 1 101545 2.41 MB/sec execute 195 sec latency 17033.350 ms oleg242-client.virtnet: 1 101545 2.39 MB/sec execute 196 sec latency 18033.555 ms oleg242-client.virtnet: 1 102228 2.41 MB/sec execute 197 sec latency 18283.649 ms oleg242-client.virtnet: 1UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36576 3566904 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13100 3593024 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49676 7159928 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg242-server Stopping /mnt/lustre-mds1 (opts:) on oleg242-server 17:51:42 (1743457902) shut down facet: mds1 facet_host: oleg242-server facet_failover_host: oleg242-server Failover mds1 to oleg242-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 17:51:56 (1743457916) targets are mounted 17:51:56 (1743457916) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1285592 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36816 3569436 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12924 3593104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49740 7162540 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg242-server Stopping /mnt/lustre-mds2 (opts:) on oleg242-server 17:52:13 (1743457933) shut down facet: mds2 facet_host: oleg242-server facet_failover_host: oleg242-server 103133 2.44 MB/sec execute 198 sec latency 11.002 ms oleg242-client.virtnet: 1 103682 2.43 MB/sec execute 199 sec latency 11.640 ms oleg242-client.virtnet: 1 104343 2.44 MB/sec execute 200 sec latency 13.869 ms oleg242-client.virtnet: 1 104935 2.43 MB/sec execute 201 sec latency 9.964 ms oleg242-client.virtnet: 1 105803 2.45 MB/sec execute 202 sec latency 9.088 ms oleg242-client.virtnet: 1 106692 2.47 MB/sec execute 203 sec latency 10.434 ms oleg242-client.virtnet: 1 107209 2.46 MB/sec execute 204 sec latency 11.475 ms oleg242-client.virtnet: 1 107892 2.47 MB/sec execute 205 sec latency 9.196 ms oleg242-client.virtnet: 1 108461 2.46 MB/sec execute 206 sec latency 13.896 ms oleg242-client.virtnet: 1 109332 2.48 MB/sec execute 207 sec latency 10.238 ms oleg242-client.virtnet: 1 110242 2.50 MB/sec execute 208 sec latency 11.447 ms oleg242-client.virtnet: 1 110691 2.49 MB/sec execute 209 sec latency 13.173 ms oleg242-client.virtnet: 1 111260 2.49 MB/sec execute 210 sec latency 10.408 ms oleg242-client.virtnet: 1 111971 2.50 MB/sec execute 211 sec latency 10.016 ms oleg242-client.virtnet: 1 112762 2.51 MB/sec execute 212 sec latency 8.826 ms oleg242-client.virtnet: 1 113795 2.53 MB/sec execute 213 sec latency 8.415 ms oleg242-client.virtnet: 1 114355 2.53 MB/sec execute 214 sec latency 11.251 ms oleg242-client.virtnet: 1 115039 2.54 MB/sec execute 215 sec latency 11.223 ms oleg242-client.virtnet: 1 115657 2.53 MB/sec execute 216 sec latency 10.601 ms oleg242-client.virtnet: 1 116529 2.55 MB/sec execute 217 sec latency 11.568 ms oleg242-client.virtnet: 1 117391 2.56 MB/sec execute 218 sec latency 9.281 ms oleg242-client.virtnet: 1 117965 2.56 MB/sec execute 219 sec latency 11.625 ms oleg242-client.virtnet: 1 118609 2.57 MB/sec execute 220 sec latency 9.490 ms oleg242-client.virtnet: 1 119257 2.56 MB/sec execute 221 sec latency 10.734 ms oleg242-client.virtnet: 1 120259 2.58 MB/sec execute 222 sec latency 10.274 ms oleg242-client.virtnet: 1 120997 2.59 MB/sec execute 223 sec latency 10.388 ms oleg242-client.virtnet: 1 121557 2.59 MB/sec execute 224 sec latency 13.147 ms oleg242-client.virtnet: 1 122221 2.59 MB/sec execute 225 sec latency 9.384 ms oleg242-client.virtnet: 1 122800 2.58 MB/sec execute 226 sec latency 10.399 ms oleg242-client.virtnet: 1 123214 2.58 MB/sec execute 227 sec latency 396.585 ms oleg242-client.virtnet: 1 124316 2.61 MB/sec execute 228 sec latency 9.825 ms oleg242-client.virtnet: 1 124859 2.61 MB/sec execute 229 sec latency 11.432 ms oleg242-client.virtnet: 1 124880 2.60 MB/sec execute 230 sec latency 970.933 ms oleg242-client.virtnet: 1 124880 2.59 MB/sec execute 231 sec latency 1971.125 ms oleg242-client.virtnet: 1 124880 2.58 MB/sec execute 232 sec latency 2971.304 ms oleg242-client.virtnet: 1 124880 2.57 MB/sec execute 233 sec latency 3971.478 ms oleg242-client.virtnet: 1 124880 2.55 MB/sec execute 234 sec latency 4971.680 ms oleg242-client.virtnet: 1 124880 2.54 MB/sec execute 235 sec latency 5971.883 ms oleg242-client.virtnet: 1 124880 2.53 MB/sec execute 236 sec latency 6972.063 ms oleg242-client.virtnet: 1 124880 2.52 MB/sec execute 237 sec latency 7972.263 ms oleg242-client.virtnet: 1 124880 2.51 MB/sec execute 238 sec latency 8972.470 ms oleg242-client.virtnet: 1 124880 2.50 MB/sec execute 239 sec latency 9972.653 ms oleg242-client.virtnet: 1 124880 2.49 MB/sec execute 240 sec latency 10972.955 ms oleg242-client.virtnet: 1 124880 2.48 MB/sec execute 241 sec latency 11973.199 ms oleg242-client.virtnet: 1 124880 2.47 MB/sec execute 242 sec latency 12973.375 ms oleg242-cFailover mds2 to oleg242-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0001 17:52:27 (1743457947) targets are mounted 17:52:27 (1743457947) facet_failover done oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lient.virtnet: 1 124880 2.46 MB/sec execute 243 sec latency 13973.562 ms oleg242-client.virtnet: 1 124880 2.45 MB/sec execute 244 sec latency 14973.752 ms oleg242-client.virtnet: 1 124880 2.44 MB/sec execute 245 sec latency 15973.953 ms oleg242-client.virtnet: 1 124880 2.43 MB/sec execute 246 sec latency 16974.135 ms oleg242-client.virtnet: 1 124880 2.42 MB/sec execute 247 sec latency 17974.308 ms oleg242-client.virtnet: 1 124880 2.41 MB/sec execute 248 sec latency 18974.549 ms oleg242-client.virtnet: 1 124880 2.40 MB/sec execute 249 sec latency 19974.739 ms oleg242-client.virtnet: 1 124880 2.39 MB/sec execute 250 sec latency 20974.965 ms oleg242-client.virtnet: 1 124880 2.38 MB/sec execute 251 sec latency 21975.204 ms oleg242-client.virtnet: 1 124880 2.37 MB/sec execute 252 sec latency 22975.441 ms oleg242-client.virtnet: 1 124880 2.36 MB/sec execute 253 sec latency 23975.626 ms oleg242-client.virtnet: 1 125162 2.36 MB/sec execute 254 sec latency 24386.831 ms oleg242-client.virtnet: 1 125768 2.36 MB/sec execute 255 sec latency 15.460 ms oleg242-client.virtnet: 1 126399 2.36 MB/sec execute 256 sec latency 8.908 ms oleg242-client.virtnet: 1 127445 2.38 MB/sec execute 257 sec latency 10.532 ms oleg242-client.virtnet: 1 128138 2.39 MB/sec execute 258 sec latency 8.859 ms oleg242-client.virtnet: 1 128718 2.38 MB/sec execute 259 sec latency 10.363 ms oleg242-client.virtnet: 1 129420 2.39 MB/sec execute 260 sec latency 9.142 ms oleg242-client.virtnet: 1 130123 2.39 MB/sec execute 261 sec latency 9.660 ms oleg242-client.virtnet: 1 131271 2.42 MB/sec execute 262 sec latency 9.614 ms oleg242-client.virtnet: 1 131831 2.42 MB/sec execute 263 sec latency 8.166 ms oleg242-client.virtnet: 1 132465 2.42 MB/sec execute 264 sec latency 11.243 ms oleg242-client.virtnet: 1 133114 2.42 MB/sec execute 265 sec latency 10.754 ms oleg242-client.virtnet: 1 133851 2.42 MB/sec execute 266 sec latency 8.519 ms oleg242-client.virtnet: 1 134942 2.45 MB/sec execute 267 sec latency 10.290 ms oleg242-client.virtnet: 1 135524 2.44 MB/sec execute 268 sec latency 10.584 ms oleg242-client.virtnet: 1 136096 2.44 MB/sec execute 269 sec latency 9.799 ms oleg242-client.virtnet: 1 136555 2.44 MB/sec execute 270 sec latency 19.236 ms oleg242-client.virtnet: 1 137211 2.44 MB/sec execute 271 sec latency 10.259 ms oleg242-client.virtnet: 1 138183 2.46 MB/sec execute 272 sec latency 12.080 ms oleg242-client.virtnet: 1 138777 2.47 MB/sec execute 273 sec latency 9.361 ms oleg242-client.virtnet: 1 139339 2.46 MB/sec execute 274 sec latency 10.141 ms oleg242-client.virtnet: 1 140019 2.47 MB/sec execute 275 sec latency 10.586 ms oleg242-client.virtnet: 1 140642 2.46 MB/sec execute 276 sec latency 11.550 ms oleg242-client.virtnet: 1 141730 2.49 MB/sec execute 277 sec latency 14.962 ms oleg242-client.virtnet: 1 142382 2.49 MB/sec execute 278 sec latency 8.381 ms oleg242-client.virtnet: 1 143043 2.49 MB/sec execute 279 sec latency 13.564 ms oleg242-client.virtnet: 1 143601 2.49 MB/sec execute 280 sec latency 13.840 ms oleg242-client.virtnet: 1 144207 2.49 MB/sec execute 281 sec latency 14.284 ms oleg242-client.virtnet: 1 145259 2.51 MB/sec execute 282 sec latency 10.096 ms oleg242-client.virtnet: 1 145934 2.52 MB/sec execute 283 sec latency 9.831 ms oleg242-client.virtnet: 1 146570 2.51 MB/sec execute 284 sec latency 11.829 ms oleg242-client.virtnet: 1 147240 2.52 MB/sec execute 285 sec latency 10.836 ms oleg242-client.virtnet: 1 147911 2.52 MB/sec execute 286 sec latency 9.056 ms oleg242-client.virtnet: 1 149035 2.54 MB/sec execute 287 sec latency 9.226 ms oleg242-client.virtnet: 1 149593 2.54 MB/sec execute 288 sec latency 9.604 ms oleg242-client.virtnet: 1 150183 2.54 MB/sec execute 289 sec latency 12.139 ms oleg242-client.virtnet: 1 150810 2.54 MB/sec execute 290 sec latency 11.300 ms oleg242-client.virtnet: 1 151469 2.54 MB/sec execute 291 sec latency 10.631 ms oleg242-client.virtnet: 1 152577 2.56 MB/sec execute 292 sec latency 13.488 ms oleg242-client.virtnet: 1 153087 2.56 MB/sec execute 293 sec latency 8.401 ms oleg242-client.virtnet: 1 153734 2.56 MB/sec execute 294 sec latency 10.013 ms oleg242-client.virtnet: 1 154341 2.56 MB/sec execute 295 sec latency 10.620 ms oleg242-client.virtnet: 1 155036 2.56 MB/sec execute 296 sec latency 8.579 ms oleg242-client.virtnet: 1 156223 2.59 MB/sec execute 297 sec latency 8.544 ms oleg242-client.virtnet: 1 156782 2.58 MB/sec execute 298 sec latency 10.101 ms oleg242-client.virtnet: 1 157346 2.58 MB/sec execute 299 sec latency 10.332 ms oleg242-client.virtnet: 1 cleanup 300 sec oleg242-client.virtnet: 0 cleanup 300 sec oleg242-client.virtnet: oleg242-client.virtnet: Operation Count AvgLat MaxLat oleg242-client.virtnet: ---------------------------------------- oleg242-client.virtnet: NTCreateX 24979 6.145 24386.817 oleg242-client.virtnet: Close 18315 0.832 6.052 oleg242-client.virtnet: Rename 1055 24.423 18303.483 oleg242-client.virtnet: Unlink 5056 1.595 4.360 oleg242-client.virtnet: Qpathinfo 22591 2.075 18483.208 oleg242-client.virtnet: Qfileinfo 3925 0.130 2.161 oleg242-client.virtnet: Qfsinfo 4141 0.202 9.226 oleg242-client.virtnet: Sfileinfo 2034 3.999 13.857 oleg242-client.virtnet: Find 8705 2.429 18587.186 oleg242-client.virtnet: WriteX 12282 0.745 8.778 oleg242-client.virtnet: ReadX 38955 0.018 3.681 oleg242-client.virtnet: LockX 80 0.847 1.452 oleg242-client.virtnet: UnlockX 80 0.867 1.537 oleg242-client.virtnet: Flush 1747 5.149 561.116 oleg242-client.virtnet: oleg242-client.virtnet: Throughput 2.5807 MB/sec 1 clients 1 procs max_latency=24386.831 ms oleg242-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg242-client.virtnet at Mon Mar 31 17:53:18 EDT 2025 with return code 0 oleg242-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg242-client.virtnet oleg242-client.virtnet: /mnt/lustre/d70b.replay-single/oleg242-client.virtnet /mnt/lustre/d70b.replay-single/oleg242-client.virtnet oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg242-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg242-client.virtnet: removed directory: 'clients/client0' oleg242-client.virtnet: removed directory: 'clients' oleg242-client.virtnet: removed 'client.txt' oleg242-client.virtnet: /mnt/lustre/d70b.replay-single/oleg242-client.virtnet oleg242-client.virtnet: dbench successfully finished