== replay-dual test 26: dbench and tar with mds failover ========================================================== 01:07:00 (1708841220) Starting client oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 6359 Started dbench loop with 6360 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Feb 25 01:07:02 EST 2024 waiting for dbench pid 6401 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 164 6.80 MB/sec warmup 1 sec latency 38.079 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1816 3605204 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26372 3580452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28188 7185656 1% /mnt/lustre 1 331 6.00 MB/sec warmup 2 sec latency 30.501 ms 1 454 5.30 MB/sec warmup 3 sec latency 426.961 ms 1 678 5.29 MB/sec warmup 4 sec latency 27.936 ms 1 863 4.31 MB/sec warmup 5 sec latency 25.235 ms test_26 fail mds1 1 times Failing mds1 on oleg318-server 1 1050 3.81 MB/sec warmup 6 sec latency 24.603 ms Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 1 1190 3.29 MB/sec warmup 7 sec latency 532.449 ms 01:07:09 (1708841229) shut down 1 1190 2.88 MB/sec warmup 8 sec latency 1532.717 ms 1 1190 2.56 MB/sec warmup 9 sec latency 2532.920 ms 1 1190 2.30 MB/sec warmup 10 sec latency 3533.113 ms 1 1190 2.09 MB/sec warmup 11 sec latency 4533.261 ms 1 1190 1.92 MB/sec warmup 12 sec latency 5533.397 ms 1 1190 1.77 MB/sec warmup 13 sec latency 6533.580 ms 1 1190 1.65 MB/sec warmup 14 sec latency 7533.786 ms 1 1190 1.54 MB/sec warmup 15 sec latency 8534.000 ms 1 1190 1.44 MB/sec warmup 16 sec latency 9534.178 ms 1 1190 1.36 MB/sec warmup 17 sec latency 10534.374 ms Failover mds1 to oleg318-server mount facets: mds1 1 1190 1.28 MB/sec warmup 18 sec latency 11534.550 ms 1 1190 1.21 MB/sec warmup 19 sec latency 12534.697 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1190 0.00 MB/sec execute 1 sec latency 14534.990 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:07:24 (1708841244) targets are mounted 01:07:24 (1708841244) facet_failover done 1 1190 0.00 MB/sec execute 2 sec latency 15535.196 ms 1 1190 0.00 MB/sec execute 3 sec latency 16535.376 ms 1 1190 0.00 MB/sec execute 4 sec latency 17535.586 ms 1 1190 0.00 MB/sec execute 5 sec latency 18535.768 ms 1 1190 0.00 MB/sec execute 6 sec latency 19535.928 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1526 0.44 MB/sec execute 7 sec latency 19613.807 ms 1 1787 0.41 MB/sec execute 8 sec latency 19.110 ms 1 2058 0.40 MB/sec execute 9 sec latency 17.205 ms 1 2396 0.49 MB/sec execute 10 sec latency 16.410 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2924 1284764 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2612 1285076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 7620 3590696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 33388 3563424 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41008 7154120 1% /mnt/lustre 1 2770 0.81 MB/sec execute 11 sec latency 19.945 ms 1 3197 0.94 MB/sec execute 12 sec latency 252.259 ms 1 3583 1.18 MB/sec execute 13 sec latency 459.801 ms 1 3828 1.19 MB/sec execute 14 sec latency 25.914 ms test_26 fail mds2 2 times Failing mds2 on oleg318-server 1 4029 1.13 MB/sec execute 15 sec latency 18.688 ms Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 1 4141 1.07 MB/sec execute 16 sec latency 605.920 ms 01:07:38 (1708841258) shut down 1 4141 1.00 MB/sec execute 17 sec latency 1606.019 ms 1 4141 0.95 MB/sec execute 18 sec latency 2606.119 ms 1 4141 0.90 MB/sec execute 19 sec latency 3606.292 ms 1 4141 0.85 MB/sec execute 20 sec latency 4606.415 ms 1 4141 0.81 MB/sec execute 21 sec latency 5606.526 ms 1 4141 0.77 MB/sec execute 22 sec latency 6606.642 ms 1 4141 0.74 MB/sec execute 23 sec latency 7606.770 ms 1 4141 0.71 MB/sec execute 24 sec latency 8606.880 ms 1 4141 0.68 MB/sec execute 25 sec latency 9607.063 ms 1 4141 0.66 MB/sec execute 26 sec latency 10607.186 ms Failover mds2 to oleg318-server mount facets: mds2 1 4141 0.63 MB/sec execute 27 sec latency 11607.307 ms 1 4141 0.61 MB/sec execute 28 sec latency 12607.407 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4141 0.59 MB/sec execute 29 sec latency 13607.524 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 1 4141 0.57 MB/sec execute 30 sec latency 14607.629 ms pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 01:07:52 (1708841272) targets are mounted 01:07:52 (1708841272) facet_failover done 1 4141 0.55 MB/sec execute 31 sec latency 15607.759 ms 1 4141 0.53 MB/sec execute 32 sec latency 16607.874 ms 1 4141 0.52 MB/sec execute 33 sec latency 17607.983 ms 1 4141 0.50 MB/sec execute 34 sec latency 18608.105 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4260 0.49 MB/sec execute 35 sec latency 19246.547 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4488 0.50 MB/sec execute 36 sec latency 28.141 ms 1 4756 0.52 MB/sec execute 37 sec latency 21.388 ms 1 5116 0.58 MB/sec execute 38 sec latency 19.658 ms 1 5373 0.57 MB/sec execute 39 sec latency 19.903 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4376 1283312 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12980 3582964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38160 3560412 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51140 7143376 1% /mnt/lustre 1 5712 0.57 MB/sec execute 40 sec latency 12.600 ms 1 5986 0.59 MB/sec execute 41 sec latency 28.151 ms 1 6429 0.67 MB/sec execute 42 sec latency 19.071 ms 1 6920 0.73 MB/sec execute 43 sec latency 15.715 ms test_26 fail mds1 3 times Failing mds1 on oleg318-server 1 7239 0.79 MB/sec execute 44 sec latency 50.787 ms Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 1 7422 0.80 MB/sec execute 45 sec latency 22.768 ms 01:08:07 (1708841287) shut down 1 7596 0.79 MB/sec execute 46 sec latency 20.180 ms 1 7627 0.77 MB/sec execute 47 sec latency 837.261 ms 1 7627 0.75 MB/sec execute 48 sec latency 1837.461 ms 1 7627 0.74 MB/sec execute 49 sec latency 2837.644 ms 1 7627 0.72 MB/sec execute 50 sec latency 3837.861 ms 1 7627 0.71 MB/sec execute 51 sec latency 4838.070 ms 1 7627 0.70 MB/sec execute 52 sec latency 5838.213 ms 1 7627 0.68 MB/sec execute 53 sec latency 6838.374 ms 1 7627 0.67 MB/sec execute 54 sec latency 7838.517 ms 1 7627 0.66 MB/sec execute 55 sec latency 8838.705 ms Failover mds1 to oleg318-server mount facets: mds1 1 7627 0.65 MB/sec execute 56 sec latency 9838.859 ms 1 7627 0.63 MB/sec execute 57 sec latency 10838.983 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7627 0.62 MB/sec execute 58 sec latency 11839.170 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 1 7627 0.61 MB/sec execute 59 sec latency 12839.303 ms pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:08:21 (1708841301) targets are mounted 01:08:21 (1708841301) facet_failover done 1 7627 0.60 MB/sec execute 60 sec latency 13839.562 ms 1 7627 0.59 MB/sec execute 61 sec latency 14839.754 ms 1 7627 0.58 MB/sec execute 62 sec latency 15839.935 ms 1 7627 0.57 MB/sec execute 63 sec latency 16840.178 ms 1 7627 0.57 MB/sec execute 64 sec latency 17840.381 ms 1 7639 0.56 MB/sec execute 65 sec latency 18797.573 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7986 0.56 MB/sec execute 66 sec latency 18.789 ms 1 8237 0.57 MB/sec execute 67 sec latency 21.017 ms 1 8585 0.61 MB/sec execute 68 sec latency 22.250 ms 1 8829 0.60 MB/sec execute 69 sec latency 17.304 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4888 1282800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2896 1284792 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21252 3563532 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37716 3553276 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 58968 7116808 1% /mnt/lustre 1 9186 0.60 MB/sec execute 70 sec latency 12.928 ms 1 9405 0.61 MB/sec execute 71 sec latency 406.767 ms 1 9679 0.61 MB/sec execute 72 sec latency 19.831 ms 1 10117 0.67 MB/sec execute 73 sec latency 23.097 ms test_26 fail mds2 4 times Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 1 10679 0.72 MB/sec execute 74 sec latency 22.914 ms 01:08:37 (1708841317) shut down 1 10728 0.72 MB/sec execute 75 sec latency 860.747 ms 1 10728 0.71 MB/sec execute 76 sec latency 1861.055 ms 1 10728 0.70 MB/sec execute 77 sec latency 2861.248 ms 1 10728 0.69 MB/sec execute 78 sec latency 3861.388 ms 1 10728 0.68 MB/sec execute 79 sec latency 4861.545 ms 1 10728 0.67 MB/sec execute 80 sec latency 5861.845 ms 1 10728 0.66 MB/sec execute 81 sec latency 6862.098 ms 1 10728 0.66 MB/sec execute 82 sec latency 7862.308 ms 1 10728 0.65 MB/sec execute 83 sec latency 8862.547 ms 1 10728 0.64 MB/sec execute 84 sec latency 9862.857 ms Failover mds2 to oleg318-server mount facets: mds2 1 10728 0.63 MB/sec execute 85 sec latency 10862.978 ms 1 10728 0.63 MB/sec execute 86 sec latency 11863.112 ms 1 10728 0.62 MB/sec execute 87 sec latency 12863.233 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 10728 0.61 MB/sec execute 88 sec latency 13863.380 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 01:08:51 (1708841331) targets are mounted 01:08:51 (1708841331) facet_failover done 1 10728 0.60 MB/sec execute 89 sec latency 14863.536 ms 1 10728 0.60 MB/sec execute 90 sec latency 15863.692 ms 1 10728 0.59 MB/sec execute 91 sec latency 16863.914 ms 1 10728 0.59 MB/sec execute 92 sec latency 17864.084 ms 1 10728 0.58 MB/sec execute 93 sec latency 18864.250 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 10879 0.58 MB/sec execute 94 sec latency 19216.821 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11039 0.58 MB/sec execute 95 sec latency 22.650 ms 1 11193 0.58 MB/sec execute 96 sec latency 27.805 ms 1 11438 0.57 MB/sec execute 97 sec latency 28.842 ms 1 11686 0.58 MB/sec execute 98 sec latency 21.123 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5404 1282284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3504 1284184 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 31148 3570364 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 47076 3554716 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 78224 7125080 2% /mnt/lustre 1 11941 0.58 MB/sec execute 99 sec latency 20.591 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1895 30.301 19613.793 Close 1392 1.973 15.886 Rename 81 13.749 23.074 Unlink 379 55.511 19246.506 Qpathinfo 1740 2.768 50.755 Qfileinfo 301 0.428 3.442 Qfsinfo 311 0.890 8.187 Sfileinfo 159 7.360 26.050 Find 662 30.916 18797.552 WriteX 934 2.155 9.739 ReadX 2999 0.092 10.611 LockX 6 1.890 2.460 UnlockX 6 1.938 2.631 Flush 130 15.079 459.790 Throughput 0.580337 MB/sec 1 clients 1 procs max_latency=19613.807 ms stopping dbench on /mnt/lustre at Sun Feb 25 01:09:03 EST 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt test_26 fail mds1 5 times '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Feb 25 01:09:04 EST 2024 waiting for dbench pid 10975 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 1 173 7.06 MB/sec warmup 1 sec latency 257.981 ms 1 173 3.53 MB/sec warmup 2 sec latency 1258.154 ms 1 173 2.35 MB/sec warmup 3 sec latency 2258.322 ms 1 173 1.77 MB/sec warmup 4 sec latency 3258.621 ms 1 173 1.41 MB/sec warmup 5 sec latency 4258.914 ms 1 173 1.18 MB/sec warmup 6 sec latency 5259.196 ms 1 173 1.01 MB/sec warmup 7 sec latency 6259.378 ms 01:09:11 (1708841351) shut down 1 173 0.88 MB/sec warmup 8 sec latency 7259.598 ms 1 173 0.78 MB/sec warmup 9 sec latency 8259.823 ms 1 173 0.71 MB/sec warmup 10 sec latency 9260.007 ms 1 173 0.64 MB/sec warmup 11 sec latency 10260.211 ms 1 173 0.59 MB/sec warmup 12 sec latency 11260.480 ms 1 173 0.54 MB/sec warmup 13 sec latency 12260.786 ms 1 173 0.50 MB/sec warmup 14 sec latency 13261.003 ms 1 173 0.47 MB/sec warmup 15 sec latency 14261.269 ms 1 173 0.44 MB/sec warmup 16 sec latency 15261.610 ms 1 173 0.42 MB/sec warmup 17 sec latency 16261.985 ms Failover mds1 to oleg318-server mount facets: mds1 1 173 0.39 MB/sec warmup 18 sec latency 17262.170 ms 1 173 0.37 MB/sec warmup 19 sec latency 18262.319 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 173 0.00 MB/sec execute 1 sec latency 20262.649 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:09:26 (1708841366) targets are mounted 01:09:26 (1708841366) facet_failover done 1 173 0.00 MB/sec execute 2 sec latency 21262.835 ms 1 173 0.00 MB/sec execute 3 sec latency 22262.994 ms 1 173 0.00 MB/sec execute 4 sec latency 23263.179 ms 1 173 0.00 MB/sec execute 5 sec latency 24263.312 ms 1 173 0.00 MB/sec execute 6 sec latency 25263.447 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 268 0.43 MB/sec execute 7 sec latency 25921.464 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 486 1.24 MB/sec execute 8 sec latency 23.319 ms 1 744 1.58 MB/sec execute 9 sec latency 41.728 ms 1 988 1.48 MB/sec execute 10 sec latency 24.170 ms 1 1221 1.46 MB/sec execute 11 sec latency 18.876 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4860 1282828 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3432 1284256 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 45576 3549636 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13672 3582216 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 59248 7131852 1% /mnt/lustre 1 1508 1.58 MB/sec execute 12 sec latency 27.629 ms 1 1731 1.48 MB/sec execute 13 sec latency 25.076 ms 1 1982 1.39 MB/sec execute 14 sec latency 36.642 ms 1 2305 1.38 MB/sec execute 15 sec latency 15.848 ms test_26 fail mds2 6 times 1 2508 1.35 MB/sec execute 16 sec latency 21.607 ms Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 1 2922 1.55 MB/sec execute 17 sec latency 90.366 ms 01:09:42 (1708841382) shut down 1 2922 1.47 MB/sec execute 18 sec latency 1090.565 ms 1 2922 1.39 MB/sec execute 19 sec latency 2090.784 ms 1 2922 1.32 MB/sec execute 20 sec latency 3090.953 ms 1 2922 1.26 MB/sec execute 21 sec latency 4091.121 ms 1 2922 1.20 MB/sec execute 22 sec latency 5091.312 ms 1 2922 1.15 MB/sec execute 23 sec latency 6091.437 ms 1 2922 1.10 MB/sec execute 24 sec latency 7091.563 ms 1 2922 1.06 MB/sec execute 25 sec latency 8091.697 ms 1 2922 1.02 MB/sec execute 26 sec latency 9091.855 ms 1 2922 0.98 MB/sec execute 27 sec latency 10092.003 ms Failover mds2 to oleg318-server mount facets: mds2 1 2922 0.94 MB/sec execute 28 sec latency 11092.137 ms 1 2922 0.91 MB/sec execute 29 sec latency 12092.403 ms 1 2922 0.88 MB/sec execute 30 sec latency 13092.591 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2922 0.85 MB/sec execute 31 sec latency 14092.797 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 01:09:56 (1708841396) targets are mounted 01:09:56 (1708841396) facet_failover done 1 2922 0.83 MB/sec execute 32 sec latency 15092.957 ms 1 2922 0.80 MB/sec execute 33 sec latency 16093.080 ms 1 2922 0.78 MB/sec execute 34 sec latency 17093.255 ms 1 2922 0.75 MB/sec execute 35 sec latency 18093.413 ms 1 2932 0.73 MB/sec execute 36 sec latency 19025.586 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3501 0.85 MB/sec execute 37 sec latency 15.926 ms 1 3781 0.86 MB/sec execute 38 sec latency 16.883 ms 1 3958 0.85 MB/sec execute 39 sec latency 23.740 ms 1 4176 0.83 MB/sec execute 40 sec latency 20.771 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3080 1284608 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2616 1285072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 48092 3552276 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15460 3580840 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 63552 7133116 1% /mnt/lustre 1 4428 0.82 MB/sec execute 41 sec latency 68.765 ms 1 4549 0.81 MB/sec execute 42 sec latency 392.636 ms 1 4780 0.82 MB/sec execute 43 sec latency 485.381 ms 1 5058 0.87 MB/sec execute 44 sec latency 37.094 ms test_26 fail mds1 7 times Failing mds1 on oleg318-server 1 5340 0.85 MB/sec execute 45 sec latency 15.743 ms Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 01:10:10 (1708841410) shut down 1 5402 0.83 MB/sec execute 46 sec latency 709.574 ms 1 5402 0.82 MB/sec execute 47 sec latency 1709.759 ms 1 5402 0.80 MB/sec execute 48 sec latency 2709.933 ms 1 5402 0.78 MB/sec execute 49 sec latency 3710.156 ms 1 5402 0.77 MB/sec execute 50 sec latency 4710.365 ms 1 5402 0.75 MB/sec execute 51 sec latency 5710.598 ms 1 5402 0.74 MB/sec execute 52 sec latency 6710.819 ms 1 5402 0.72 MB/sec execute 53 sec latency 7711.024 ms 1 5402 0.71 MB/sec execute 54 sec latency 8711.183 ms 1 5402 0.70 MB/sec execute 55 sec latency 9711.410 ms 1 5402 0.69 MB/sec execute 56 sec latency 10711.589 ms Failover mds1 to oleg318-server mount facets: mds1 1 5402 0.67 MB/sec execute 57 sec latency 11711.796 ms 1 5402 0.66 MB/sec execute 58 sec latency 12712.008 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5402 0.65 MB/sec execute 59 sec latency 13712.190 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 1 5402 0.64 MB/sec execute 60 sec latency 14712.501 ms Started lustre-MDT0000 01:10:24 (1708841424) targets are mounted 01:10:24 (1708841424) facet_failover done 1 5402 0.63 MB/sec execute 61 sec latency 15712.687 ms 1 5402 0.62 MB/sec execute 62 sec latency 16712.931 ms 1 5402 0.61 MB/sec execute 63 sec latency 17713.140 ms 1 5402 0.60 MB/sec execute 64 sec latency 18713.320 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5551 0.59 MB/sec execute 65 sec latency 19297.247 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar 1 5893 0.60 MB/sec execute 66 sec latency 17.395 ms 1 6115 0.61 MB/sec execute 67 sec latency 34.511 ms 1 6544 0.67 MB/sec execute 68 sec latency 19.507 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sun Feb 25 01:10:33 EST 2024 with return code 0 10975 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 10975 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished