== replay-dual test 26: dbench and tar with mds failover ========================================================== 17:01:05 (1771970465) Starting client oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre Started clients oleg414-client.virtnet: 192.168.204.114@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started tar loop with pid 44874 Started dbench loop with 44875 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Feb 24 17:01:08 EST 2026 waiting for dbench pid 44910 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3732480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3695616 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7428096 1% /mnt/lustre 1 150 6.28 MB/sec warmup 1 sec latency 43.091 ms 1 282 5.22 MB/sec warmup 2 sec latency 38.325 ms 1 437 5.13 MB/sec warmup 3 sec latency 32.259 ms test_26 fail mds1 1 times 1 511 4.43 MB/sec warmup 4 sec latency 74.421 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 608 4.19 MB/sec warmup 5 sec latency 252.234 ms reboot facets: mds1 1 608 3.49 MB/sec warmup 6 sec latency 1252.381 ms 1 608 3.00 MB/sec warmup 7 sec latency 2252.512 ms 1 608 2.62 MB/sec warmup 8 sec latency 3252.785 ms 1 608 2.33 MB/sec warmup 9 sec latency 4252.971 ms 1 608 2.10 MB/sec warmup 10 sec latency 5253.254 ms 1 608 1.91 MB/sec warmup 11 sec latency 6253.471 ms 1 608 1.75 MB/sec warmup 12 sec latency 7253.677 ms 1 608 1.61 MB/sec warmup 13 sec latency 8253.907 ms 1 608 1.50 MB/sec warmup 14 sec latency 9254.091 ms 1 608 1.40 MB/sec warmup 15 sec latency 10254.217 ms Failover mds1 to oleg414-server 1 608 1.31 MB/sec warmup 16 sec latency 11254.360 ms mount facets: mds1 1 608 1.23 MB/sec warmup 17 sec latency 12254.535 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 608 1.16 MB/sec warmup 18 sec latency 13254.792 ms 1 608 1.10 MB/sec warmup 19 sec latency 14255.002 ms 1 608 0.00 MB/sec execute 1 sec latency 16255.339 ms 1 608 0.00 MB/sec execute 2 sec latency 17255.538 ms 1 608 0.00 MB/sec execute 3 sec latency 18255.787 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 1 608 0.00 MB/sec execute 4 sec latency 19255.933 ms pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 1 608 0.00 MB/sec execute 5 sec latency 20256.126 ms 1 608 0.00 MB/sec execute 6 sec latency 21256.361 ms 1 608 0.00 MB/sec execute 7 sec latency 22256.541 ms 1 608 0.00 MB/sec execute 8 sec latency 23256.764 ms 1 608 0.00 MB/sec execute 9 sec latency 24256.962 ms 1 608 0.00 MB/sec execute 10 sec latency 25257.170 ms 1 608 0.00 MB/sec execute 11 sec latency 26257.349 ms 1 608 0.00 MB/sec execute 12 sec latency 27257.519 ms 1 608 0.00 MB/sec execute 13 sec latency 28257.700 ms 1 608 0.00 MB/sec execute 14 sec latency 29257.919 ms 1 608 0.00 MB/sec execute 15 sec latency 30258.057 ms 1 699 0.01 MB/sec execute 16 sec latency 30652.994 ms 1 864 0.04 MB/sec execute 17 sec latency 49.222 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1002 0.06 MB/sec execute 18 sec latency 66.079 ms 1 1179 0.11 MB/sec execute 19 sec latency 69.630 ms 1 1391 0.25 MB/sec execute 20 sec latency 41.960 ms 1 1504 0.24 MB/sec execute 21 sec latency 118.663 ms 1 1680 0.24 MB/sec execute 22 sec latency 58.779 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3680256 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 49152 3702784 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 7383040 1% /mnt/lustre 1 1886 0.23 MB/sec execute 23 sec latency 36.424 ms 1 2145 0.24 MB/sec execute 24 sec latency 19.938 ms 1 2395 0.28 MB/sec execute 25 sec latency 66.050 ms test_26 fail mds1 2 times Failing mds1 on oleg414-server 1 2594 0.31 MB/sec execute 26 sec latency 40.160 ms Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 2879 0.42 MB/sec execute 27 sec latency 440.634 ms reboot facets: mds1 1 2879 0.40 MB/sec execute 28 sec latency 1440.787 ms 1 2879 0.39 MB/sec execute 29 sec latency 2440.941 ms 1 2879 0.37 MB/sec execute 30 sec latency 3441.084 ms 1 2879 0.36 MB/sec execute 31 sec latency 4441.249 ms 1 2879 0.35 MB/sec execute 32 sec latency 5441.413 ms 1 2879 0.34 MB/sec execute 33 sec latency 6441.551 ms 1 2879 0.33 MB/sec execute 34 sec latency 7441.694 ms 1 2879 0.32 MB/sec execute 35 sec latency 8441.848 ms 1 2879 0.31 MB/sec execute 36 sec latency 9441.992 ms 1 2879 0.30 MB/sec execute 37 sec latency 10442.149 ms Failover mds1 to oleg414-server 1 2879 0.30 MB/sec execute 38 sec latency 11442.289 ms mount facets: mds1 1 2879 0.29 MB/sec execute 39 sec latency 12442.455 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2879 0.28 MB/sec execute 40 sec latency 13442.657 ms 1 2879 0.27 MB/sec execute 41 sec latency 14442.865 ms 1 2879 0.27 MB/sec execute 42 sec latency 15443.036 ms 1 2879 0.26 MB/sec execute 43 sec latency 16443.216 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 1 2879 0.26 MB/sec execute 44 sec latency 17443.411 ms 1 2879 0.25 MB/sec execute 45 sec latency 18443.561 ms 1 2879 0.24 MB/sec execute 46 sec latency 19443.741 ms 1 2879 0.24 MB/sec execute 47 sec latency 20443.894 ms 1 2879 0.23 MB/sec execute 48 sec latency 21444.080 ms 1 2879 0.23 MB/sec execute 49 sec latency 22444.261 ms 1 2879 0.22 MB/sec execute 50 sec latency 23444.436 ms 1 2879 0.22 MB/sec execute 51 sec latency 24444.657 ms 1 2879 0.22 MB/sec execute 52 sec latency 25444.804 ms 1 2879 0.21 MB/sec execute 53 sec latency 26444.962 ms 1 2879 0.21 MB/sec execute 54 sec latency 27445.124 ms 1 2879 0.20 MB/sec execute 55 sec latency 28445.313 ms 1 2879 0.20 MB/sec execute 56 sec latency 29445.599 ms 1 2879 0.20 MB/sec execute 57 sec latency 30445.749 ms 1 2879 0.19 MB/sec execute 58 sec latency 31445.915 ms 1 3013 0.21 MB/sec execute 59 sec latency 31499.353 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3351 0.24 MB/sec execute 60 sec latency 65.541 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3629 0.29 MB/sec execute 61 sec latency 20.462 ms 1 3828 0.30 MB/sec execute 62 sec latency 25.277 ms 1 3983 0.30 MB/sec execute 63 sec latency 25.080 ms 1 4180 0.30 MB/sec execute 64 sec latency 25.673 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5760 2202752 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 29696 3716096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 50176 3699712 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 79872 7415808 2% /mnt/lustre 1 4336 0.30 MB/sec execute 65 sec latency 76.456 ms 1 4496 0.30 MB/sec execute 66 sec latency 42.933 ms 1 4716 0.32 MB/sec execute 67 sec latency 58.229 ms test_26 fail mds1 3 times 1 4992 0.36 MB/sec execute 68 sec latency 23.012 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 5172 0.35 MB/sec execute 69 sec latency 64.677 ms 1 5210 0.35 MB/sec execute 70 sec latency 794.394 ms reboot facets: mds1 1 5210 0.34 MB/sec execute 71 sec latency 1794.544 ms 1 5210 0.34 MB/sec execute 72 sec latency 2794.749 ms 1 5210 0.33 MB/sec execute 73 sec latency 3794.923 ms 1 5210 0.33 MB/sec execute 74 sec latency 4795.083 ms 1 5210 0.32 MB/sec execute 75 sec latency 5795.206 ms 1 5210 0.32 MB/sec execute 76 sec latency 6795.334 ms 1 5210 0.32 MB/sec execute 77 sec latency 7795.512 ms 1 5210 0.31 MB/sec execute 78 sec latency 8795.671 ms 1 5210 0.31 MB/sec execute 79 sec latency 9795.821 ms 1 5210 0.30 MB/sec execute 80 sec latency 10795.993 ms Failover mds1 to oleg414-server mount facets: mds1 1 5210 0.30 MB/sec execute 81 sec latency 11796.163 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5210 0.30 MB/sec execute 82 sec latency 12796.401 ms 1 5210 0.29 MB/sec execute 83 sec latency 13796.577 ms 1 5210 0.29 MB/sec execute 84 sec latency 14796.760 ms 1 5210 0.29 MB/sec execute 85 sec latency 15796.937 ms 1 5210 0.28 MB/sec execute 86 sec latency 16797.098 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 1 5210 0.28 MB/sec execute 87 sec latency 17797.271 ms Started lustre-MDT0000 1 5210 0.28 MB/sec execute 88 sec latency 18797.420 ms 1 5210 0.27 MB/sec execute 89 sec latency 19797.598 ms 1 5210 0.27 MB/sec execute 90 sec latency 20797.797 ms 1 5210 0.27 MB/sec execute 91 sec latency 21797.987 ms 1 5210 0.26 MB/sec execute 92 sec latency 22798.170 ms 1 5210 0.26 MB/sec execute 93 sec latency 23798.343 ms 1 5210 0.26 MB/sec execute 94 sec latency 24798.603 ms 1 5210 0.26 MB/sec execute 95 sec latency 25798.788 ms 1 5210 0.25 MB/sec execute 96 sec latency 26798.961 ms 1 5210 0.25 MB/sec execute 97 sec latency 27799.165 ms 1 5210 0.25 MB/sec execute 98 sec latency 28799.340 ms 1 5210 0.25 MB/sec execute 99 sec latency 29799.593 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 793 51.572 30652.968 Close 591 57.922 32886.835 Rename 36 17.153 32.366 Unlink 145 224.365 31499.338 Qpathinfo 722 4.002 34.100 Qfileinfo 126 0.860 5.344 Qfsinfo 131 0.497 2.758 Sfileinfo 84 8.270 14.615 Find 275 2.807 61.923 WriteX 410 3.424 10.728 ReadX 1216 0.574 69.613 LockX 2 1.796 2.137 UnlockX 2 2.065 2.093 Flush 70 34.254 118.640 Throughput 0.245902 MB/sec 1 clients 1 procs max_latency=31499.353 ms stopping dbench on /mnt/lustre at Tue Feb 24 17:03:11 EST 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Feb 24 17:03:14 EST 2026 waiting for dbench pid 48020 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 173 7.06 MB/sec warmup 1 sec latency 35.586 ms 1 303 5.58 MB/sec warmup 2 sec latency 72.295 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6656 2201856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3732480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 76800 3685376 3% /mnt/lustre[OST:1] filesystem_summary: 7542784 108544 7417856 2% /mnt/lustre 1 453 5.30 MB/sec warmup 3 sec latency 41.902 ms 1 654 5.27 MB/sec warmup 4 sec latency 47.348 ms 1 794 4.29 MB/sec warmup 5 sec latency 69.633 ms 1 977 3.65 MB/sec warmup 6 sec latency 46.160 ms test_26 fail mds1 4 times Failing mds1 on oleg414-server 1 1193 3.29 MB/sec warmup 7 sec latency 40.133 ms Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 1384 3.25 MB/sec warmup 8 sec latency 481.879 ms 1 1384 2.89 MB/sec warmup 9 sec latency 1482.053 ms 1 1384 2.60 MB/sec warmup 10 sec latency 2482.223 ms 1 1384 2.36 MB/sec warmup 11 sec latency 3482.432 ms 1 1384 2.17 MB/sec warmup 12 sec latency 4482.650 ms 1 1384 2.00 MB/sec warmup 13 sec latency 5482.816 ms 1 1384 1.86 MB/sec warmup 14 sec latency 6482.919 ms reboot facets: mds1 1 1384 1.73 MB/sec warmup 15 sec latency 7483.032 ms 1 1384 1.62 MB/sec warmup 16 sec latency 8483.165 ms 1 1384 1.53 MB/sec warmup 17 sec latency 9483.324 ms 1 1384 1.44 MB/sec warmup 18 sec latency 10483.549 ms 1 1384 1.37 MB/sec warmup 19 sec latency 11483.773 ms 1 1384 0.00 MB/sec execute 1 sec latency 13484.233 ms 1 1384 0.00 MB/sec execute 2 sec latency 14484.479 ms 1 1384 0.00 MB/sec execute 3 sec latency 15484.766 ms 1 1384 0.00 MB/sec execute 4 sec latency 16484.984 ms Failover mds1 to oleg414-server 1 1384 0.00 MB/sec execute 5 sec latency 17485.195 ms mount facets: mds1 1 1384 0.00 MB/sec execute 6 sec latency 18485.351 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1384 0.00 MB/sec execute 7 sec latency 19485.552 ms 1 1384 0.00 MB/sec execute 8 sec latency 20485.799 ms 1 1384 0.00 MB/sec execute 9 sec latency 21486.065 ms 1 1384 0.00 MB/sec execute 10 sec latency 22486.259 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 1 1384 0.00 MB/sec execute 11 sec latency 23486.408 ms Started lustre-MDT0000 1 1384 0.00 MB/sec execute 12 sec latency 24486.589 ms 1 1384 0.00 MB/sec execute 13 sec latency 25486.743 ms 1 1416 0.00 MB/sec execute 14 sec latency 26241.335 ms 1 1580 0.01 MB/sec execute 15 sec latency 132.466 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1808 0.02 MB/sec execute 16 sec latency 86.979 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2022 0.03 MB/sec execute 17 sec latency 74.902 ms 1 2263 0.05 MB/sec execute 18 sec latency 28.761 ms 1 2421 0.11 MB/sec execute 19 sec latency 99.767 ms 1 2610 0.15 MB/sec execute 20 sec latency 112.485 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3738624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 52224 3707904 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 7446528 1% /mnt/lustre 1 2985 0.36 MB/sec execute 21 sec latency 92.866 ms 1 3353 0.43 MB/sec execute 22 sec latency 68.990 ms 1 3708 0.56 MB/sec execute 23 sec latency 15.386 ms test_26 fail mds1 5 times 1 3927 0.59 MB/sec execute 24 sec latency 20.716 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 4094 0.56 MB/sec execute 25 sec latency 157.367 ms reboot facets: mds1 1 4094 0.54 MB/sec execute 26 sec latency 1157.563 ms 1 4094 0.52 MB/sec execute 27 sec latency 2157.721 ms 1 4094 0.50 MB/sec execute 28 sec latency 3157.885 ms 1 4094 0.49 MB/sec execute 29 sec latency 4158.064 ms 1 4094 0.47 MB/sec execute 30 sec latency 5158.252 ms 1 4094 0.45 MB/sec execute 31 sec latency 6158.376 ms 1 4094 0.44 MB/sec execute 32 sec latency 7158.584 ms 1 4094 0.43 MB/sec execute 33 sec latency 8158.784 ms 1 4094 0.41 MB/sec execute 34 sec latency 9158.940 ms 1 4094 0.40 MB/sec execute 35 sec latency 10159.111 ms Failover mds1 to oleg414-server 1 4094 0.39 MB/sec execute 36 sec latency 11159.273 ms mount facets: mds1 1 4094 0.38 MB/sec execute 37 sec latency 12159.510 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4094 0.37 MB/sec execute 38 sec latency 13159.699 ms 1 4094 0.36 MB/sec execute 39 sec latency 14159.870 ms 1 4094 0.35 MB/sec execute 40 sec latency 15160.064 ms 1 4094 0.34 MB/sec execute 41 sec latency 16160.258 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 1 4094 0.34 MB/sec execute 42 sec latency 17160.450 ms Started lustre-MDT0000 1 4094 0.33 MB/sec execute 43 sec latency 18160.638 ms 1 4094 0.32 MB/sec execute 44 sec latency 19160.822 ms 1 4094 0.31 MB/sec execute 45 sec latency 20160.983 ms 1 4094 0.31 MB/sec execute 46 sec latency 21161.177 ms 1 4094 0.30 MB/sec execute 47 sec latency 22161.371 ms 1 4094 0.29 MB/sec execute 48 sec latency 23161.537 ms 1 4094 0.29 MB/sec execute 49 sec latency 24161.718 ms 1 4094 0.28 MB/sec execute 50 sec latency 25161.890 ms 1 4094 0.28 MB/sec execute 51 sec latency 26162.075 ms 1 4094 0.27 MB/sec execute 52 sec latency 27162.261 ms 1 4094 0.27 MB/sec execute 53 sec latency 28162.457 ms 1 4094 0.26 MB/sec execute 54 sec latency 29162.655 ms 1 4094 0.26 MB/sec execute 55 sec latency 30162.801 ms 1 4094 0.25 MB/sec execute 56 sec latency 31162.962 ms 1 4236 0.25 MB/sec execute 57 sec latency 31349.160 ms 1 4360 0.25 MB/sec execute 58 sec latency 82.279 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4483 0.25 MB/sec execute 59 sec latency 63.661 ms 1 4682 0.27 MB/sec execute 60 sec latency 66.924 ms 1 4945 0.31 MB/sec execute 61 sec latency 22.239 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 5131 0.31 MB/sec execute 62 sec latency 79.922 ms 1 5320 0.31 MB/sec execute 63 sec latency 48.106 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Feb 24 17:04:37 EST 2026 with return code 0 48020 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48020 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished