== replay-dual test 26: dbench and tar with mds failover ========================================================== 19:38:43 (1781048323) Starting client oleg338-client.virtnet: -o user_xattr,flock oleg338-server@tcp:/lustre /mnt/lustre Started clients oleg338-client.virtnet: 192.168.203.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started tar loop with pid 44930 Started dbench loop with 44931 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 9 19:38:46 EDT 2026 waiting for dbench pid 44960 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3712000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 26624 7449600 1% /mnt/lustre 1 158 6.54 MB/sec warmup 1 sec latency 38.178 ms 1 289 5.35 MB/sec warmup 2 sec latency 88.800 ms 1 419 4.93 MB/sec warmup 3 sec latency 29.716 ms test_26 fail mds1 1 times 1 527 4.55 MB/sec warmup 4 sec latency 36.489 ms Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 1 671 4.23 MB/sec warmup 5 sec latency 85.858 ms 1 688 3.53 MB/sec warmup 6 sec latency 858.036 ms reboot facets: mds1 1 688 3.03 MB/sec warmup 7 sec latency 1858.230 ms 1 688 2.65 MB/sec warmup 8 sec latency 2858.375 ms 1 688 2.35 MB/sec warmup 9 sec latency 3858.544 ms 1 688 2.12 MB/sec warmup 10 sec latency 4858.703 ms 1 688 1.93 MB/sec warmup 11 sec latency 5858.920 ms 1 688 1.77 MB/sec warmup 12 sec latency 6859.078 ms 1 688 1.63 MB/sec warmup 13 sec latency 7859.215 ms 1 688 1.51 MB/sec warmup 14 sec latency 8859.496 ms 1 688 1.41 MB/sec warmup 15 sec latency 9859.664 ms 1 688 1.32 MB/sec warmup 16 sec latency 10859.860 ms Failover mds1 to oleg338-server mount facets: mds1 1 688 1.25 MB/sec warmup 17 sec latency 11860.013 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 688 1.18 MB/sec warmup 18 sec latency 12860.151 ms 1 688 1.11 MB/sec warmup 19 sec latency 13860.348 ms 1 688 0.00 MB/sec execute 1 sec latency 15860.565 ms 1 688 0.00 MB/sec execute 2 sec latency 16860.768 ms 1 688 0.00 MB/sec execute 3 sec latency 17861.018 ms oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 1 688 0.00 MB/sec execute 4 sec latency 18861.289 ms 1 688 0.00 MB/sec execute 5 sec latency 19861.448 ms 1 688 0.00 MB/sec execute 6 sec latency 20861.664 ms 1 688 0.00 MB/sec execute 7 sec latency 21861.910 ms 1 688 0.00 MB/sec execute 8 sec latency 22862.143 ms 1 688 0.00 MB/sec execute 9 sec latency 23862.405 ms 1 688 0.00 MB/sec execute 10 sec latency 24862.664 ms 1 688 0.00 MB/sec execute 11 sec latency 25862.928 ms 1 688 0.00 MB/sec execute 12 sec latency 26863.158 ms 1 688 0.00 MB/sec execute 13 sec latency 27863.453 ms 1 688 0.00 MB/sec execute 14 sec latency 28863.672 ms 1 688 0.00 MB/sec execute 15 sec latency 29863.891 ms 1 688 0.00 MB/sec execute 16 sec latency 30864.309 ms 1 688 0.00 MB/sec execute 17 sec latency 31864.621 ms 1 689 0.00 MB/sec execute 18 sec latency 32802.982 ms 1 803 0.02 MB/sec execute 19 sec latency 103.393 ms oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 964 0.03 MB/sec execute 20 sec latency 76.845 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1147 0.09 MB/sec execute 21 sec latency 51.841 ms 1 1285 0.09 MB/sec execute 22 sec latency 40.295 ms 1 1544 0.21 MB/sec execute 23 sec latency 66.654 ms 1 1684 0.21 MB/sec execute 24 sec latency 89.171 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3661824 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3718144 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 7379968 1% /mnt/lustre 1 1901 0.21 MB/sec execute 25 sec latency 78.570 ms 1 2150 0.21 MB/sec execute 26 sec latency 42.279 ms 1 2359 0.25 MB/sec execute 27 sec latency 93.597 ms test_26 fail mds1 2 times 1 2571 0.28 MB/sec execute 28 sec latency 54.235 ms Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 1 2922 0.42 MB/sec execute 29 sec latency 105.648 ms 1 2922 0.41 MB/sec execute 30 sec latency 1105.947 ms reboot facets: mds1 1 2922 0.40 MB/sec execute 31 sec latency 2106.179 ms 1 2922 0.38 MB/sec execute 32 sec latency 3106.464 ms 1 2922 0.37 MB/sec execute 33 sec latency 4106.681 ms 1 2922 0.36 MB/sec execute 34 sec latency 5106.953 ms 1 2922 0.35 MB/sec execute 35 sec latency 6107.181 ms 1 2922 0.34 MB/sec execute 36 sec latency 7107.420 ms 1 2922 0.33 MB/sec execute 37 sec latency 8107.637 ms 1 2922 0.32 MB/sec execute 38 sec latency 9107.839 ms 1 2922 0.32 MB/sec execute 39 sec latency 10108.144 ms 1 2922 0.31 MB/sec execute 40 sec latency 11108.443 ms Failover mds1 to oleg338-server mount facets: mds1 1 2922 0.30 MB/sec execute 41 sec latency 12108.610 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2922 0.29 MB/sec execute 42 sec latency 13108.791 ms 1 2922 0.29 MB/sec execute 43 sec latency 14109.051 ms 1 2922 0.28 MB/sec execute 44 sec latency 15109.280 ms 1 2922 0.27 MB/sec execute 45 sec latency 16109.475 ms oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all 8 1 2922 0.27 MB/sec execute 46 sec latency 17109.621 ms pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 1 2922 0.26 MB/sec execute 47 sec latency 18109.758 ms 1 2922 0.26 MB/sec execute 48 sec latency 19109.879 ms 1 2922 0.25 MB/sec execute 49 sec latency 20110.060 ms 1 2922 0.25 MB/sec execute 50 sec latency 21110.296 ms 1 2922 0.24 MB/sec execute 51 sec latency 22110.501 ms 1 2922 0.24 MB/sec execute 52 sec latency 23110.729 ms 1 2922 0.23 MB/sec execute 53 sec latency 24110.977 ms 1 2922 0.23 MB/sec execute 54 sec latency 25111.243 ms 1 2922 0.22 MB/sec execute 55 sec latency 26111.416 ms 1 2922 0.22 MB/sec execute 56 sec latency 27111.606 ms 1 2922 0.22 MB/sec execute 57 sec latency 28111.805 ms 1 2922 0.21 MB/sec execute 58 sec latency 29112.131 ms 1 2922 0.21 MB/sec execute 59 sec latency 30112.387 ms 1 2922 0.20 MB/sec execute 60 sec latency 31112.659 ms 1 2922 0.20 MB/sec execute 61 sec latency 32112.904 ms 1 2922 0.20 MB/sec execute 62 sec latency 33113.163 ms 1 3012 0.20 MB/sec execute 63 sec latency 33543.328 ms 1 3378 0.22 MB/sec execute 64 sec latency 45.639 ms oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3685 0.27 MB/sec execute 65 sec latency 20.138 ms 1 3844 0.28 MB/sec execute 66 sec latency 30.811 ms 1 4002 0.28 MB/sec execute 67 sec latency 28.859 ms 1 4127 0.28 MB/sec execute 68 sec latency 40.435 ms 1 4236 0.28 MB/sec execute 69 sec latency 218.523 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5504 2203008 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 51200 3689472 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3715072 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 80896 7404544 2% /mnt/lustre 1 4360 0.28 MB/sec execute 70 sec latency 304.172 ms 1 4491 0.28 MB/sec execute 71 sec latency 83.320 ms 1 4714 0.29 MB/sec execute 72 sec latency 44.831 ms test_26 fail mds1 3 times 1 4987 0.33 MB/sec execute 73 sec latency 26.435 ms Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 1 5135 0.33 MB/sec execute 74 sec latency 152.314 ms reboot facets: mds1 1 5135 0.32 MB/sec execute 75 sec latency 1152.487 ms 1 5135 0.32 MB/sec execute 76 sec latency 2152.643 ms 1 5135 0.31 MB/sec execute 77 sec latency 3152.807 ms 1 5135 0.31 MB/sec execute 78 sec latency 4153.057 ms 1 5135 0.30 MB/sec execute 79 sec latency 5153.242 ms 1 5135 0.30 MB/sec execute 80 sec latency 6153.429 ms 1 5135 0.30 MB/sec execute 81 sec latency 7153.633 ms 1 5135 0.29 MB/sec execute 82 sec latency 8153.842 ms 1 5135 0.29 MB/sec execute 83 sec latency 9154.019 ms 1 5135 0.29 MB/sec execute 84 sec latency 10154.239 ms 1 5135 0.28 MB/sec execute 85 sec latency 11154.386 ms Failover mds1 to oleg338-server mount facets: mds1 1 5135 0.28 MB/sec execute 86 sec latency 12154.534 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5135 0.28 MB/sec execute 87 sec latency 13154.714 ms 1 5135 0.27 MB/sec execute 88 sec latency 14154.893 ms 1 5135 0.27 MB/sec execute 89 sec latency 15155.075 ms 1 5135 0.27 MB/sec execute 90 sec latency 16155.323 ms oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all 8 1 5135 0.26 MB/sec execute 91 sec latency 17155.480 ms pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 1 5135 0.26 MB/sec execute 92 sec latency 18155.670 ms 1 5135 0.26 MB/sec execute 93 sec latency 19155.849 ms 1 5135 0.26 MB/sec execute 94 sec latency 20155.987 ms 1 5135 0.25 MB/sec execute 95 sec latency 21156.210 ms 1 5135 0.25 MB/sec execute 96 sec latency 22156.386 ms 1 5135 0.25 MB/sec execute 97 sec latency 23156.604 ms 1 5135 0.25 MB/sec execute 98 sec latency 24156.814 ms 1 5135 0.24 MB/sec execute 99 sec latency 25156.998 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 0 cleanup 106 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 761 54.540 31235.330 Close 577 59.251 32802.953 Rename 36 18.673 40.282 Unlink 144 8.175 18.061 Qpathinfo 703 4.097 61.216 Qfileinfo 120 0.783 3.838 Qfsinfo 126 0.416 3.477 Sfileinfo 81 8.690 18.678 Find 262 2.629 31.227 WriteX 392 3.124 21.024 ReadX 1180 0.469 51.825 LockX 2 3.156 3.403 UnlockX 2 2.704 3.104 Flush 62 583.459 33543.312 Throughput 0.242958 MB/sec 1 clients 1 procs max_latency=33543.328 ms stopping dbench on /mnt/lustre at Tue Jun 9 19:40:52 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 9 19:40:55 EDT 2026 waiting for dbench pid 48087 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 137 6.02 MB/sec warmup 1 sec latency 49.688 ms 1 228 4.38 MB/sec warmup 2 sec latency 103.928 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6528 2201984 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 56320 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 28672 3677184 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 84992 7390208 2% /mnt/lustre 1 374 4.43 MB/sec warmup 3 sec latency 33.660 ms 1 511 4.43 MB/sec warmup 4 sec latency 69.463 ms 1 711 4.24 MB/sec warmup 5 sec latency 60.226 ms test_26 fail mds1 4 times 1 874 3.61 MB/sec warmup 6 sec latency 60.850 ms Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 1 1074 3.27 MB/sec warmup 7 sec latency 54.678 ms reboot facets: mds1 1 1074 2.87 MB/sec warmup 8 sec latency 1001.017 ms 1 1074 2.55 MB/sec warmup 9 sec latency 2001.246 ms 1 1074 2.29 MB/sec warmup 10 sec latency 3001.508 ms 1 1074 2.08 MB/sec warmup 11 sec latency 4001.758 ms 1 1074 1.91 MB/sec warmup 12 sec latency 5001.948 ms 1 1074 1.76 MB/sec warmup 13 sec latency 6002.153 ms 1 1074 1.64 MB/sec warmup 14 sec latency 7002.377 ms 1 1074 1.53 MB/sec warmup 15 sec latency 8002.594 ms 1 1074 1.43 MB/sec warmup 16 sec latency 9002.789 ms 1 1074 1.35 MB/sec warmup 17 sec latency 10002.990 ms Failover mds1 to oleg338-server 1 1074 1.27 MB/sec warmup 18 sec latency 11003.124 ms mount facets: mds1 1 1074 1.21 MB/sec warmup 19 sec latency 12003.327 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1074 0.00 MB/sec execute 1 sec latency 14003.562 ms 1 1074 0.00 MB/sec execute 2 sec latency 15003.696 ms 1 1074 0.00 MB/sec execute 3 sec latency 16003.840 ms 1 1074 0.00 MB/sec execute 4 sec latency 17004.007 ms 1 1074 0.00 MB/sec execute 5 sec latency 18004.160 ms oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1074 0.00 MB/sec execute 6 sec latency 19004.350 ms 1 1074 0.00 MB/sec execute 7 sec latency 20004.496 ms 1 1074 0.00 MB/sec execute 8 sec latency 21004.649 ms 1 1074 0.00 MB/sec execute 9 sec latency 22004.793 ms 1 1074 0.00 MB/sec execute 10 sec latency 23005.044 ms 1 1074 0.00 MB/sec execute 11 sec latency 24005.237 ms 1 1074 0.00 MB/sec execute 12 sec latency 25005.427 ms 1 1074 0.00 MB/sec execute 13 sec latency 26005.579 ms 1 1074 0.00 MB/sec execute 14 sec latency 27005.730 ms 1 1074 0.00 MB/sec execute 15 sec latency 28005.914 ms 1 1074 0.00 MB/sec execute 16 sec latency 29006.115 ms 1 1074 0.00 MB/sec execute 17 sec latency 30006.337 ms 1 1074 0.00 MB/sec execute 18 sec latency 31006.496 ms 1 1244 0.01 MB/sec execute 19 sec latency 31201.222 ms oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1489 0.16 MB/sec execute 20 sec latency 53.352 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1651 0.16 MB/sec execute 21 sec latency 95.822 ms 1 1909 0.16 MB/sec execute 22 sec latency 71.911 ms 1 2168 0.17 MB/sec execute 23 sec latency 16.883 ms 1 2414 0.22 MB/sec execute 24 sec latency 65.526 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4352 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 41984 3710976 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7447552 1% /mnt/lustre 1 2597 0.24 MB/sec execute 25 sec latency 81.291 ms 1 2988 0.41 MB/sec execute 26 sec latency 102.156 ms 1 3377 0.46 MB/sec execute 27 sec latency 89.374 ms test_26 fail mds1 5 times 1 3735 0.57 MB/sec execute 28 sec latency 17.319 ms Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 1 3910 0.59 MB/sec execute 29 sec latency 100.229 ms reboot facets: mds1 1 3910 0.57 MB/sec execute 30 sec latency 1100.439 ms 1 3910 0.55 MB/sec execute 31 sec latency 2100.622 ms 1 3910 0.54 MB/sec execute 32 sec latency 3100.751 ms 1 3910 0.52 MB/sec execute 33 sec latency 4100.944 ms 1 3910 0.50 MB/sec execute 34 sec latency 5101.104 ms 1 3910 0.49 MB/sec execute 35 sec latency 6101.275 ms 1 3910 0.48 MB/sec execute 36 sec latency 7101.416 ms 1 3910 0.46 MB/sec execute 37 sec latency 8101.551 ms 1 3910 0.45 MB/sec execute 38 sec latency 9101.675 ms 1 3910 0.44 MB/sec execute 39 sec latency 10101.860 ms Failover mds1 to oleg338-server 1 3910 0.43 MB/sec execute 40 sec latency 11101.970 ms mount facets: mds1 1 3910 0.42 MB/sec execute 41 sec latency 12102.091 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3910 0.41 MB/sec execute 42 sec latency 13102.288 ms 1 3910 0.40 MB/sec execute 43 sec latency 14102.461 ms 1 3910 0.39 MB/sec execute 44 sec latency 15102.592 ms oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all 8 1 3910 0.38 MB/sec execute 45 sec latency 16102.728 ms pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4046 0.37 MB/sec execute 46 sec latency 16277.219 ms 1 4263 0.37 MB/sec execute 47 sec latency 88.180 ms 1 4418 0.37 MB/sec execute 48 sec latency 101.036 ms oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4546 0.37 MB/sec execute 49 sec latency 70.113 ms 1 4780 0.39 MB/sec execute 50 sec latency 45.176 ms 1 5078 0.44 MB/sec execute 51 sec latency 62.341 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 5274 0.43 MB/sec execute 52 sec latency 52.678 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Jun 9 19:42:08 EDT 2026 with return code 0 1 5474 0.43 MB/sec execute 53 sec latency 93.285 ms 48087 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48087 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished