== replay-dual test 26: dbench and tar with mds failover ========================================================== 14:32:11 (1779129131) Starting client oleg126-client.virtnet: -o user_xattr,flock 192.168.201.126@tcp:/lustre /mnt/lustre Started clients oleg126-client.virtnet: 192.168.201.126@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44602 Started dbench loop with 44604 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3740672 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 12288 7505920 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon May 18 14:32:18 EDT 2026 waiting for dbench pid 44672 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 1 of 1 processes prepared for launch 0 sec releasing clients 1 14 0.00 MB/sec warmup 1 sec latency 111.319 ms 1 52 1.05 MB/sec warmup 2 sec latency 204.199 ms 1 96 1.44 MB/sec warmup 3 sec latency 173.670 ms 1 121 1.37 MB/sec warmup 4 sec latency 249.871 ms 1 145 1.24 MB/sec warmup 5 sec latency 215.808 ms 1 177 1.19 MB/sec warmup 6 sec latency 163.984 ms test_26 fail mds1 1 times 1 199 1.12 MB/sec warmup 7 sec latency 251.890 ms 1 223 1.08 MB/sec warmup 8 sec latency 173.119 ms Failing mds1 on oleg126-server 1 247 1.04 MB/sec warmup 9 sec latency 229.024 ms Stopping /mnt/lustre-mds1 (opts:) on oleg126-server 1 282 1.04 MB/sec warmup 10 sec latency 157.793 ms 1 311 1.04 MB/sec warmup 11 sec latency 182.117 ms 1 313 0.96 MB/sec warmup 12 sec latency 983.877 ms 1 313 0.88 MB/sec warmup 13 sec latency 1984.362 ms 1 313 0.82 MB/sec warmup 14 sec latency 2987.110 ms 14:32:32 (1779129152) shut down facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server 1 313 0.76 MB/sec warmup 15 sec latency 3987.425 ms 1 313 0.72 MB/sec warmup 16 sec latency 4988.818 ms 1 313 0.67 MB/sec warmup 17 sec latency 5990.855 ms 1 313 0.64 MB/sec warmup 18 sec latency 6991.105 ms 1 313 0.60 MB/sec warmup 19 sec latency 7995.250 ms 1 313 0.00 MB/sec execute 1 sec latency 9998.443 ms 1 313 0.00 MB/sec execute 2 sec latency 10999.040 ms 1 313 0.00 MB/sec execute 3 sec latency 11999.433 ms 1 313 0.00 MB/sec execute 4 sec latency 12999.682 ms Failover mds1 to oleg126-server mount facets: mds1 1 313 0.00 MB/sec execute 5 sec latency 14006.143 ms 1 313 0.00 MB/sec execute 6 sec latency 15006.451 ms 1 313 0.00 MB/sec execute 7 sec latency 16008.637 ms 1 313 0.00 MB/sec execute 8 sec latency 17008.919 ms 1 313 0.00 MB/sec execute 9 sec latency 18009.224 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 313 0.00 MB/sec execute 10 sec latency 19010.410 ms 1 313 0.00 MB/sec execute 11 sec latency 20011.458 ms 1 313 0.00 MB/sec execute 12 sec latency 21011.791 ms 1 313 0.00 MB/sec execute 13 sec latency 22014.679 ms 1 313 0.00 MB/sec execute 14 sec latency 23015.462 ms 1 313 0.00 MB/sec execute 15 sec latency 24016.211 ms 1 313 0.00 MB/sec execute 16 sec latency 25016.873 ms 1 313 0.00 MB/sec execute 17 sec latency 26017.142 ms 1 313 0.00 MB/sec execute 18 sec latency 27018.428 ms 1 313 0.00 MB/sec execute 19 sec latency 28023.331 ms 1 313 0.00 MB/sec execute 20 sec latency 29023.691 ms 1 313 0.00 MB/sec execute 21 sec latency 30023.965 ms 1 313 0.00 MB/sec execute 22 sec latency 31024.213 ms 1 313 0.00 MB/sec execute 23 sec latency 32024.684 ms 1 313 0.00 MB/sec execute 24 sec latency 33025.816 ms 1 313 0.00 MB/sec execute 25 sec latency 34026.078 ms 1 313 0.00 MB/sec execute 26 sec latency 35026.786 ms 1 313 0.00 MB/sec execute 27 sec latency 36027.088 ms oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 1 313 0.00 MB/sec execute 28 sec latency 37027.409 ms 1 313 0.00 MB/sec execute 29 sec latency 38028.337 ms pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 1 313 0.00 MB/sec execute 30 sec latency 39031.472 ms Started lustre-MDT0000 14:33:09 (1779129189) targets are mounted 14:33:09 (1779129189) facet_failover done 1 313 0.00 MB/sec execute 31 sec latency 40046.478 ms 1 313 0.00 MB/sec execute 32 sec latency 41046.819 ms 1 323 0.01 MB/sec execute 33 sec latency 41538.470 ms 1 372 0.05 MB/sec execute 34 sec latency 130.202 ms 1 418 0.09 MB/sec execute 35 sec latency 134.739 ms oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 461 0.13 MB/sec execute 36 sec latency 125.626 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 501 0.16 MB/sec execute 37 sec latency 118.977 ms 1 522 0.17 MB/sec execute 38 sec latency 222.742 ms 1 587 0.22 MB/sec execute 39 sec latency 197.262 ms 1 636 0.24 MB/sec execute 40 sec latency 158.704 ms 1 664 0.24 MB/sec execute 41 sec latency 160.629 ms 1 678 0.23 MB/sec execute 42 sec latency 340.710 ms 1 711 0.23 MB/sec execute 43 sec latency 155.509 ms 1 730 0.22 MB/sec execute 44 sec latency 332.245 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3705856 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 9216 3726336 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 46080 7432192 1% /mnt/lustre 1 744 0.22 MB/sec execute 45 sec latency 195.403 ms 1 778 0.21 MB/sec execute 46 sec latency 150.602 ms 1 791 0.21 MB/sec execute 47 sec latency 390.504 ms 1 802 0.21 MB/sec execute 48 sec latency 250.010 ms 1 821 0.20 MB/sec execute 49 sec latency 214.608 ms 1 863 0.20 MB/sec execute 50 sec latency 177.293 ms 1 881 0.20 MB/sec execute 51 sec latency 357.958 ms 1 903 0.20 MB/sec execute 52 sec latency 209.096 ms test_26 fail mds1 2 times 1 924 0.19 MB/sec execute 53 sec latency 212.298 ms 1 952 0.19 MB/sec execute 54 sec latency 196.254 ms 1 969 0.19 MB/sec execute 55 sec latency 166.955 ms Failing mds1 on oleg126-server 1 981 0.19 MB/sec execute 56 sec latency 366.294 ms 1 1005 0.19 MB/sec execute 57 sec latency 181.784 ms Stopping /mnt/lustre-mds1 (opts:) on oleg126-server 1 1048 0.20 MB/sec execute 58 sec latency 136.375 ms 1 1063 0.19 MB/sec execute 59 sec latency 475.309 ms 1 1063 0.19 MB/sec execute 60 sec latency 1475.530 ms 1 1063 0.19 MB/sec execute 61 sec latency 2479.672 ms 1 1063 0.18 MB/sec execute 62 sec latency 3479.935 ms 14:33:41 (1779129221) shut down facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server 1 1063 0.18 MB/sec execute 63 sec latency 4480.073 ms 1 1063 0.18 MB/sec execute 64 sec latency 5480.360 ms 1 1063 0.18 MB/sec execute 65 sec latency 6480.640 ms 1 1063 0.17 MB/sec execute 66 sec latency 7480.947 ms 1 1063 0.17 MB/sec execute 67 sec latency 8481.877 ms 1 1063 0.17 MB/sec execute 68 sec latency 9482.159 ms 1 1063 0.16 MB/sec execute 69 sec latency 10485.800 ms 1 1063 0.16 MB/sec execute 70 sec latency 11486.211 ms 1 1063 0.16 MB/sec execute 71 sec latency 12486.658 ms 1 1063 0.16 MB/sec execute 72 sec latency 13487.214 ms 1 1063 0.16 MB/sec execute 73 sec latency 14487.595 ms Failover mds1 to oleg126-server mount facets: mds1 1 1063 0.15 MB/sec execute 74 sec latency 15497.561 ms 1 1063 0.15 MB/sec execute 75 sec latency 16499.106 ms 1 1063 0.15 MB/sec execute 76 sec latency 17499.482 ms 1 1063 0.15 MB/sec execute 77 sec latency 18500.017 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1063 0.15 MB/sec execute 78 sec latency 19500.798 ms 1 1063 0.14 MB/sec execute 79 sec latency 20509.281 ms 1 1063 0.14 MB/sec execute 80 sec latency 21509.805 ms 1 1063 0.14 MB/sec execute 81 sec latency 22517.065 ms 1 1063 0.14 MB/sec execute 82 sec latency 23517.233 ms 1 1063 0.14 MB/sec execute 83 sec latency 24517.556 ms 1 1063 0.14 MB/sec execute 84 sec latency 25523.072 ms 1 1063 0.13 MB/sec execute 85 sec latency 26523.360 ms 1 1063 0.13 MB/sec execute 86 sec latency 27523.626 ms 1 1063 0.13 MB/sec execute 87 sec latency 28524.018 ms 1 1063 0.13 MB/sec execute 88 sec latency 29525.246 ms 1 1063 0.13 MB/sec execute 89 sec latency 30525.556 ms 1 1063 0.13 MB/sec execute 90 sec latency 31525.882 ms 1 1063 0.13 MB/sec execute 91 sec latency 32526.493 ms 1 1063 0.12 MB/sec execute 92 sec latency 33526.732 ms 1 1063 0.12 MB/sec execute 93 sec latency 34529.582 ms 1 1063 0.12 MB/sec execute 94 sec latency 35534.214 ms 1 1063 0.12 MB/sec execute 95 sec latency 36534.573 ms 1 1063 0.12 MB/sec execute 96 sec latency 37538.922 ms 1 1063 0.12 MB/sec execute 97 sec latency 38542.574 ms oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 1 1063 0.12 MB/sec execute 98 sec latency 39562.903 ms 1 1063 0.11 MB/sec execute 99 sec latency 40563.387 ms pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 1 cleanup 100 sec Started lustre-MDT0000 14:34:19 (1779129259) targets are mounted 14:34:19 (1779129259) facet_failover done 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 107 879.516 41538.451 Close 98 16.920 74.390 Rename 4 177.563 218.072 Unlink 11 48.852 110.021 Qpathinfo 40 39.452 205.909 Qfileinfo 50 2.504 12.349 Qfsinfo 23 7.233 25.231 Sfileinfo 18 74.532 151.195 Find 29 10.912 69.032 WriteX 250 13.291 67.587 ReadX 98 7.600 167.703 Flush 24 194.085 390.488 Throughput 0.114957 MB/sec 1 clients 1 procs max_latency=41538.470 ms stopping dbench on /mnt/lustre at Mon May 18 14:34:23 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon May 18 14:34:30 EDT 2026 waiting for dbench pid 46553 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 29 0.98 MB/sec warmup 1 sec latency 37.919 ms 1 76 1.64 MB/sec warmup 2 sec latency 139.131 ms 1 114 1.74 MB/sec warmup 3 sec latency 191.646 ms 1 147 1.57 MB/sec warmup 4 sec latency 140.066 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 30720 3718144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3720192 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 51200 7438336 1% /mnt/lustre 1 174 1.41 MB/sec warmup 5 sec latency 152.902 ms 1 197 1.30 MB/sec warmup 6 sec latency 187.478 ms 1 226 1.23 MB/sec warmup 7 sec latency 188.754 ms 1 263 1.24 MB/sec warmup 8 sec latency 151.590 ms 1 289 1.19 MB/sec warmup 9 sec latency 166.397 ms 1 315 1.15 MB/sec warmup 10 sec latency 146.674 ms 1 350 1.14 MB/sec warmup 11 sec latency 130.110 ms test_26 fail mds1 3 times 1 389 1.15 MB/sec warmup 12 sec latency 108.723 ms 1 421 1.14 MB/sec warmup 13 sec latency 128.191 ms 1 453 1.13 MB/sec warmup 14 sec latency 150.517 ms Failing mds1 on oleg126-server 1 477 1.11 MB/sec warmup 15 sec latency 176.645 ms Stopping /mnt/lustre-mds1 (opts:) on oleg126-server 1 511 1.10 MB/sec warmup 16 sec latency 199.917 ms 1 542 1.12 MB/sec warmup 17 sec latency 158.012 ms 1 600 1.16 MB/sec warmup 18 sec latency 242.148 ms 1 600 1.10 MB/sec warmup 19 sec latency 1246.785 ms 14:34:51 (1779129291) shut down facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server 1 600 0.00 MB/sec execute 1 sec latency 3247.589 ms 1 600 0.00 MB/sec execute 2 sec latency 4249.026 ms 1 600 0.00 MB/sec execute 3 sec latency 5249.531 ms 1 600 0.00 MB/sec execute 4 sec latency 6250.553 ms 1 600 0.00 MB/sec execute 5 sec latency 7251.425 ms 1 600 0.00 MB/sec execute 6 sec latency 8252.680 ms 1 600 0.00 MB/sec execute 7 sec latency 9252.986 ms 1 600 0.00 MB/sec execute 8 sec latency 10253.190 ms 1 600 0.00 MB/sec execute 9 sec latency 11270.591 ms 1 600 0.00 MB/sec execute 10 sec latency 12271.683 ms Failover mds1 to oleg126-server mount facets: mds1 1 600 0.00 MB/sec execute 11 sec latency 13281.998 ms 1 600 0.00 MB/sec execute 12 sec latency 14282.199 ms 1 600 0.00 MB/sec execute 13 sec latency 15282.530 ms 1 600 0.00 MB/sec execute 14 sec latency 16282.968 ms 1 600 0.00 MB/sec execute 15 sec latency 17283.431 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 600 0.00 MB/sec execute 16 sec latency 18283.761 ms 1 600 0.00 MB/sec execute 17 sec latency 19285.256 ms 1 600 0.00 MB/sec execute 18 sec latency 20288.223 ms 1 600 0.00 MB/sec execute 19 sec latency 21288.525 ms 1 600 0.00 MB/sec execute 20 sec latency 22288.983 ms 1 600 0.00 MB/sec execute 21 sec latency 23293.567 ms 1 600 0.00 MB/sec execute 22 sec latency 24293.847 ms 1 600 0.00 MB/sec execute 23 sec latency 25294.134 ms 1 600 0.00 MB/sec execute 24 sec latency 26294.631 ms 1 600 0.00 MB/sec execute 25 sec latency 27296.062 ms 1 600 0.00 MB/sec execute 26 sec latency 28297.200 ms 1 600 0.00 MB/sec execute 27 sec latency 29299.576 ms 1 600 0.00 MB/sec execute 28 sec latency 30300.693 ms 1 600 0.00 MB/sec execute 29 sec latency 31301.284 ms 1 600 0.00 MB/sec execute 30 sec latency 32302.982 ms 1 600 0.00 MB/sec execute 31 sec latency 33303.832 ms oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all 1 600 0.00 MB/sec execute 32 sec latency 34306.877 ms 1 600 0.00 MB/sec execute 33 sec latency 35308.177 ms pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 1 600 0.00 MB/sec execute 34 sec latency 36308.427 ms Started lustre-MDT0000 14:35:26 (1779129326) targets are mounted 1 600 0.00 MB/sec execute 35 sec latency 37322.409 ms 14:35:26 (1779129326) facet_failover done 1 600 0.00 MB/sec execute 36 sec latency 38325.366 ms 1 600 0.00 MB/sec execute 37 sec latency 39325.996 ms 1 600 0.00 MB/sec execute 38 sec latency 40326.172 ms 1 636 0.00 MB/sec execute 39 sec latency 40510.269 ms 1 664 0.00 MB/sec execute 40 sec latency 311.139 ms 1 686 0.01 MB/sec execute 41 sec latency 325.548 ms oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 717 0.01 MB/sec execute 42 sec latency 120.596 ms 1 728 0.01 MB/sec execute 43 sec latency 504.631 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 759 0.01 MB/sec execute 44 sec latency 161.436 ms 1 778 0.01 MB/sec execute 45 sec latency 174.623 ms 1 796 0.01 MB/sec execute 46 sec latency 273.904 ms 1 810 0.01 MB/sec execute 47 sec latency 159.712 ms 1 863 0.01 MB/sec execute 48 sec latency 125.396 ms 1 892 0.01 MB/sec execute 49 sec latency 261.193 ms 1 916 0.02 MB/sec execute 50 sec latency 289.855 ms 1 936 0.02 MB/sec execute 51 sec latency 193.435 ms 1 970 0.02 MB/sec execute 52 sec latency 140.846 ms 1 986 0.02 MB/sec execute 53 sec latency 258.263 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1010 0.03 MB/sec execute 54 sec latency 156.363 ms 1 1064 0.03 MB/sec execute 55 sec latency 253.376 ms 1 1126 0.04 MB/sec execute 56 sec latency 85.878 ms 1 1155 0.04 MB/sec execute 57 sec latency 175.836 ms 1 1190 0.04 MB/sec execute 58 sec latency 119.886 ms 1 1230 0.04 MB/sec execute 59 sec latency 138.084 ms 1 1268 0.04 MB/sec execute 60 sec latency 102.672 ms 1 1398 0.08 MB/sec execute 61 sec latency 81.246 ms 1 1422 0.08 MB/sec execute 62 sec latency 140.047 ms 1 1468 0.08 MB/sec execute 63 sec latency 148.806 ms tar: Error is not recoverable: exiting now 1 1509 0.08 MB/sec execute 64 sec latency 184.408 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon May 18 14:35:55 EDT 2026 with return code 0 46553 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 46553 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished