== replay-dual test 26: dbench and tar with mds failover ========================================================== 13:38:32 (1783445912) Starting client oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre Started clients oleg238-client.virtnet: 192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44903 Started dbench loop with 44904 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3730432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3755008 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7485440 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jul 7 13:38:37 EDT 2026 waiting for dbench pid 44965 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 34 1.04 MB/sec warmup 1 sec latency 144.071 ms 1 92 2.09 MB/sec warmup 2 sec latency 125.155 ms 1 137 2.01 MB/sec warmup 3 sec latency 147.658 ms 1 181 1.83 MB/sec warmup 4 sec latency 123.779 ms test_26 fail mds1 1 times 1 215 1.67 MB/sec warmup 5 sec latency 125.203 ms 1 253 1.59 MB/sec warmup 6 sec latency 118.132 ms Failing mds1 on oleg238-server 1 280 1.49 MB/sec warmup 7 sec latency 165.470 ms Stopping /mnt/lustre-mds1 (opts:) on oleg238-server 1 315 1.43 MB/sec warmup 8 sec latency 139.056 ms 1 341 1.36 MB/sec warmup 9 sec latency 200.218 ms 1 348 1.25 MB/sec warmup 10 sec latency 858.346 ms 1 348 1.14 MB/sec warmup 11 sec latency 1858.727 ms 13:38:49 (1783445929) shut down facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server 1 348 1.04 MB/sec warmup 12 sec latency 2859.119 ms 1 348 0.96 MB/sec warmup 13 sec latency 3859.293 ms 1 348 0.89 MB/sec warmup 14 sec latency 4859.558 ms 1 348 0.83 MB/sec warmup 15 sec latency 5859.813 ms 1 348 0.78 MB/sec warmup 16 sec latency 6860.307 ms 1 348 0.74 MB/sec warmup 17 sec latency 7860.516 ms 1 348 0.69 MB/sec warmup 18 sec latency 8861.182 ms 1 348 0.66 MB/sec warmup 19 sec latency 9861.381 ms 1 348 0.00 MB/sec execute 1 sec latency 11861.715 ms Failover mds1 to oleg238-server mount facets: mds1 1 348 0.00 MB/sec execute 2 sec latency 12861.838 ms 1 348 0.00 MB/sec execute 3 sec latency 13862.737 ms 1 348 0.00 MB/sec execute 4 sec latency 14863.001 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 348 0.00 MB/sec execute 5 sec latency 15872.254 ms 1 348 0.00 MB/sec execute 6 sec latency 16872.452 ms 1 348 0.00 MB/sec execute 7 sec latency 17872.994 ms 1 348 0.00 MB/sec execute 8 sec latency 18873.188 ms 1 348 0.00 MB/sec execute 9 sec latency 19873.392 ms 1 348 0.00 MB/sec execute 10 sec latency 20883.974 ms oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 1 348 0.00 MB/sec execute 11 sec latency 21889.496 ms 1 365 0.04 MB/sec execute 12 sec latency 22274.991 ms pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 13:39:10 (1783445950) targets are mounted 13:39:10 (1783445950) facet_failover done 1 418 0.17 MB/sec execute 13 sec latency 121.878 ms 1 470 0.28 MB/sec execute 14 sec latency 108.713 ms 1 516 0.35 MB/sec execute 15 sec latency 99.159 ms 1 606 0.53 MB/sec execute 16 sec latency 155.397 ms 1 664 0.51 MB/sec execute 17 sec latency 100.861 ms 1 689 0.48 MB/sec execute 18 sec latency 210.153 ms oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 721 0.46 MB/sec execute 19 sec latency 136.012 ms 1 738 0.44 MB/sec execute 20 sec latency 219.189 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 769 0.42 MB/sec execute 21 sec latency 134.503 ms 1 796 0.41 MB/sec execute 22 sec latency 332.780 ms 1 860 0.39 MB/sec execute 23 sec latency 104.284 ms 1 903 0.38 MB/sec execute 24 sec latency 102.532 ms 1 959 0.37 MB/sec execute 25 sec latency 93.916 ms 1 988 0.36 MB/sec execute 26 sec latency 186.753 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 48128 3702784 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 9216 3713024 1% /mnt/lustre[OST:1] 1 1051 0.38 MB/sec execute 27 sec latency 94.971 ms filesystem_summary: 7542784 57344 7415808 1% /mnt/lustre 1 1091 0.37 MB/sec execute 28 sec latency 128.219 ms 1 1145 0.36 MB/sec execute 29 sec latency 71.958 ms 1 1198 0.35 MB/sec execute 30 sec latency 141.871 ms 1 1244 0.34 MB/sec execute 31 sec latency 63.824 ms 1 1384 0.42 MB/sec execute 32 sec latency 82.278 ms test_26 fail mds1 2 times 1 1428 0.41 MB/sec execute 33 sec latency 117.867 ms Failing mds1 on oleg238-server 1 1474 0.40 MB/sec execute 34 sec latency 69.858 ms Stopping /mnt/lustre-mds1 (opts:) on oleg238-server 1 1514 0.39 MB/sec execute 35 sec latency 186.802 ms 1 1558 0.38 MB/sec execute 36 sec latency 415.493 ms 1 1558 0.37 MB/sec execute 37 sec latency 1416.780 ms 1 1558 0.36 MB/sec execute 38 sec latency 2417.009 ms 1 1558 0.35 MB/sec execute 39 sec latency 3417.149 ms 13:39:37 (1783445977) shut down 1 1558 0.34 MB/sec execute 40 sec latency 4427.705 ms facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server 1 1558 0.33 MB/sec execute 41 sec latency 5429.041 ms 1 1558 0.32 MB/sec execute 42 sec latency 6429.310 ms 1 1558 0.32 MB/sec execute 43 sec latency 7429.660 ms 1 1558 0.31 MB/sec execute 44 sec latency 8432.549 ms 1 1558 0.30 MB/sec execute 45 sec latency 9432.769 ms 1 1558 0.29 MB/sec execute 46 sec latency 10432.993 ms 1 1558 0.29 MB/sec execute 47 sec latency 11433.625 ms 1 1558 0.28 MB/sec execute 48 sec latency 12434.720 ms 1 1558 0.28 MB/sec execute 49 sec latency 13434.924 ms 1 1558 0.27 MB/sec execute 50 sec latency 14435.730 ms Failover mds1 to oleg238-server mount facets: mds1 1 1558 0.27 MB/sec execute 51 sec latency 15435.951 ms 1 1558 0.26 MB/sec execute 52 sec latency 16436.278 ms 1 1558 0.26 MB/sec execute 53 sec latency 17436.421 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1558 0.25 MB/sec execute 54 sec latency 18436.761 ms 1 1558 0.25 MB/sec execute 55 sec latency 19437.854 ms 1 1558 0.24 MB/sec execute 56 sec latency 20439.650 ms 1 1558 0.24 MB/sec execute 57 sec latency 21439.848 ms 1 1558 0.23 MB/sec execute 58 sec latency 22440.031 ms 1 1558 0.23 MB/sec execute 59 sec latency 23442.008 ms 1 1558 0.23 MB/sec execute 60 sec latency 24442.742 ms 1 1558 0.22 MB/sec execute 61 sec latency 25443.295 ms 1 1558 0.22 MB/sec execute 62 sec latency 26443.660 ms 1 1558 0.22 MB/sec execute 63 sec latency 27445.035 ms 1 1558 0.21 MB/sec execute 64 sec latency 28445.218 ms 1 1558 0.21 MB/sec execute 65 sec latency 29446.879 ms 1 1558 0.21 MB/sec execute 66 sec latency 30447.404 ms 1 1558 0.20 MB/sec execute 67 sec latency 31447.699 ms oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 1 1558 0.20 MB/sec execute 68 sec latency 32447.992 ms pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 1 1558 0.20 MB/sec execute 69 sec latency 33448.195 ms Started lustre-MDT0000 13:40:07 (1783446007) targets are mounted 13:40:07 (1783446007) facet_failover done 1 1558 0.19 MB/sec execute 70 sec latency 34448.379 ms 1 1558 0.19 MB/sec execute 71 sec latency 35448.553 ms 1 1558 0.19 MB/sec execute 72 sec latency 36448.816 ms 1 1558 0.19 MB/sec execute 73 sec latency 37449.084 ms 1 1558 0.18 MB/sec execute 74 sec latency 38449.835 ms 1 1558 0.18 MB/sec execute 75 sec latency 39450.680 ms 1 1558 0.18 MB/sec execute 76 sec latency 40450.946 ms 1 1558 0.18 MB/sec execute 77 sec latency 41451.521 ms 1 1558 0.17 MB/sec execute 78 sec latency 42453.349 ms 1 1558 0.17 MB/sec execute 79 sec latency 43453.572 ms 1 1558 0.17 MB/sec execute 80 sec latency 44453.777 ms 1 1558 0.17 MB/sec execute 81 sec latency 45454.113 ms 1 1558 0.17 MB/sec execute 82 sec latency 46454.707 ms 1 1558 0.16 MB/sec execute 83 sec latency 47455.181 ms 1 1558 0.16 MB/sec execute 84 sec latency 48455.406 ms 1 1558 0.16 MB/sec execute 85 sec latency 49455.639 ms 1 1558 0.16 MB/sec execute 86 sec latency 50456.905 ms 1 1558 0.16 MB/sec execute 87 sec latency 51457.155 ms 1 1558 0.15 MB/sec execute 88 sec latency 52457.408 ms 1 1558 0.15 MB/sec execute 89 sec latency 53464.676 ms 1 1591 0.15 MB/sec execute 90 sec latency 53688.349 ms 1 1603 0.15 MB/sec execute 91 sec latency 619.589 ms 1 1646 0.15 MB/sec execute 92 sec latency 220.852 ms 1 1691 0.15 MB/sec execute 93 sec latency 104.055 ms oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1775 0.15 MB/sec execute 94 sec latency 77.623 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1837 0.15 MB/sec execute 95 sec latency 85.847 ms 1 1902 0.14 MB/sec execute 96 sec latency 129.452 ms 1 1996 0.14 MB/sec execute 97 sec latency 79.760 ms 1 2095 0.14 MB/sec execute 98 sec latency 82.208 ms 1 2181 0.14 MB/sec execute 99 sec latency 68.972 ms 1 cleanup 100 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 60416 3704832 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7451648 1% /mnt/lustre 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 341 204.058 53688.333 Close 296 85.718 22274.970 Rename 11 78.113 141.848 Unlink 30 25.319 80.702 Qpathinfo 295 13.411 185.151 Qfileinfo 88 2.049 14.911 Qfsinfo 33 3.590 25.908 Sfileinfo 34 45.793 77.255 Find 92 4.694 63.571 WriteX 323 10.001 67.699 ReadX 362 1.693 98.688 Flush 30 118.968 332.765 Throughput 0.14353 MB/sec 1 clients 1 procs max_latency=53688.349 ms stopping dbench on /mnt/lustre at Tue Jul 7 13:40:41 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt test_26 fail mds1 3 times Failing mds1 on oleg238-server '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jul 7 13:40:46 EDT 2026 waiting for dbench pid 47164 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients Stopping /mnt/lustre-mds1 (opts:) on oleg238-server 1 31 1.05 MB/sec warmup 1 sec latency 182.322 ms 1 31 0.52 MB/sec warmup 2 sec latency 1182.686 ms 13:40:48 (1783446048) shut down facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server 1 31 0.35 MB/sec warmup 3 sec latency 2182.992 ms 1 31 0.26 MB/sec warmup 4 sec latency 3187.418 ms 1 31 0.21 MB/sec warmup 5 sec latency 4187.676 ms 1 31 0.17 MB/sec warmup 6 sec latency 5188.052 ms 1 31 0.15 MB/sec warmup 7 sec latency 6188.280 ms 1 31 0.13 MB/sec warmup 8 sec latency 7188.971 ms 1 31 0.12 MB/sec warmup 9 sec latency 8189.601 ms 1 31 0.10 MB/sec warmup 10 sec latency 9189.825 ms 1 31 0.10 MB/sec warmup 11 sec latency 10190.041 ms 1 31 0.09 MB/sec warmup 12 sec latency 11190.343 ms Failover mds1 to oleg238-server 1 31 0.08 MB/sec warmup 13 sec latency 12190.495 ms mount facets: mds1 1 31 0.07 MB/sec warmup 14 sec latency 13192.136 ms 1 31 0.07 MB/sec warmup 15 sec latency 14194.318 ms 1 31 0.07 MB/sec warmup 16 sec latency 15197.568 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 31 0.06 MB/sec warmup 17 sec latency 16198.351 ms 1 31 0.06 MB/sec warmup 18 sec latency 17198.661 ms 1 31 0.06 MB/sec warmup 19 sec latency 18201.498 ms 1 31 0.00 MB/sec execute 1 sec latency 20202.705 ms oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 1 64 0.85 MB/sec execute 2 sec latency 20860.513 ms pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 1 130 1.57 MB/sec execute 3 sec latency 117.138 ms Started lustre-MDT0000 13:41:09 (1783446069) targets are mounted 13:41:09 (1783446069) facet_failover done 1 197 1.70 MB/sec execute 4 sec latency 80.840 ms 1 255 1.72 MB/sec execute 5 sec latency 71.109 ms 1 313 1.74 MB/sec execute 6 sec latency 90.284 ms 1 365 1.71 MB/sec execute 7 sec latency 85.027 ms oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 413 1.69 MB/sec execute 8 sec latency 143.303 ms 1 469 1.71 MB/sec execute 9 sec latency 96.799 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 517 1.67 MB/sec execute 10 sec latency 104.841 ms 1 636 1.81 MB/sec execute 11 sec latency 149.130 ms 1 676 1.67 MB/sec execute 12 sec latency 180.772 ms 1 721 1.55 MB/sec execute 13 sec latency 97.937 ms 1 789 1.45 MB/sec execute 14 sec latency 164.394 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 28672 3643392 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3628032 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 61440 7271424 1% /mnt/lustre 1 848 1.37 MB/sec execute 15 sec latency 105.407 ms 1 891 1.29 MB/sec execute 16 sec latency 119.791 ms 1 949 1.22 MB/sec execute 17 sec latency 77.019 ms 1 992 1.16 MB/sec execute 18 sec latency 97.442 ms 1 1126 1.15 MB/sec execute 19 sec latency 41.421 ms test_26 fail mds1 4 times 1 1178 1.10 MB/sec execute 20 sec latency 101.634 ms Failing mds1 on oleg238-server 1 1225 1.05 MB/sec execute 21 sec latency 94.367 ms Stopping /mnt/lustre-mds1 (opts:) on oleg238-server 1 1284 1.00 MB/sec execute 22 sec latency 66.067 ms 1 1378 1.08 MB/sec execute 23 sec latency 969.365 ms 1 1378 1.04 MB/sec execute 24 sec latency 1969.605 ms 13:41:30 (1783446090) shut down facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server 1 1378 1.00 MB/sec execute 25 sec latency 2970.115 ms 1 1378 0.96 MB/sec execute 26 sec latency 3970.361 ms 1 1378 0.92 MB/sec execute 27 sec latency 4970.577 ms 1 1378 0.89 MB/sec execute 28 sec latency 5972.788 ms 1 1378 0.86 MB/sec execute 29 sec latency 6978.969 ms 1 1378 0.83 MB/sec execute 30 sec latency 7980.071 ms 1 1378 0.80 MB/sec execute 31 sec latency 8980.312 ms 1 1378 0.78 MB/sec execute 32 sec latency 9981.968 ms 1 1378 0.75 MB/sec execute 33 sec latency 10984.970 ms 1 1378 0.73 MB/sec execute 34 sec latency 11985.175 ms Failover mds1 to oleg238-server mount facets: mds1 1 1378 0.71 MB/sec execute 35 sec latency 12985.366 ms 1 1378 0.69 MB/sec execute 36 sec latency 13985.543 ms 1 1378 0.67 MB/sec execute 37 sec latency 14985.676 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1378 0.66 MB/sec execute 38 sec latency 15991.898 ms 1 1378 0.64 MB/sec execute 39 sec latency 16992.072 ms 1 1378 0.62 MB/sec execute 40 sec latency 17992.284 ms 1 1378 0.61 MB/sec execute 41 sec latency 18992.516 ms 1 1378 0.59 MB/sec execute 42 sec latency 19992.906 ms oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 1 1378 0.58 MB/sec execute 43 sec latency 20993.105 ms Started lustre-MDT0000 13:41:50 (1783446110) targets are mounted 13:41:50 (1783446110) facet_failover done 1 1416 0.57 MB/sec execute 44 sec latency 21150.227 ms 1 1499 0.55 MB/sec execute 45 sec latency 119.613 ms 1 1580 0.54 MB/sec execute 46 sec latency 67.534 ms 1 1634 0.53 MB/sec execute 47 sec latency 199.716 ms 1 1689 0.52 MB/sec execute 48 sec latency 123.184 ms oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1751 0.51 MB/sec execute 49 sec latency 135.631 ms 1 1808 0.50 MB/sec execute 50 sec latency 87.333 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1855 0.50 MB/sec execute 51 sec latency 105.152 ms 1 1902 0.49 MB/sec execute 52 sec latency 190.366 ms 1 2004 0.48 MB/sec execute 53 sec latency 69.906 ms 1 2099 0.47 MB/sec execute 54 sec latency 71.609 ms 1 2166 0.47 MB/sec execute 55 sec latency 116.293 ms 1 2259 0.46 MB/sec execute 56 sec latency 70.246 ms 1 2357 0.47 MB/sec execute 57 sec latency 223.615 ms 1 2416 0.47 MB/sec execute 58 sec latency 119.003 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 2482 0.47 MB/sec execute 59 sec latency 118.862 ms 1 2566 0.46 MB/sec execute 60 sec latency 475.680 ms 1 2586 0.46 MB/sec execute 61 sec latency 1077.121 ms 1 2768 0.50 MB/sec execute 62 sec latency 62.013 ms 1 2896 0.50 MB/sec execute 63 sec latency 234.294 ms 1 2975 0.51 MB/sec execute 64 sec latency 148.632 ms 1 3010 0.50 MB/sec execute 65 sec latency 228.137 ms 1 3198 0.50 MB/sec execute 66 sec latency 213.454 ms 1 3299 0.51 MB/sec execute 67 sec latency 79.240 ms 1 3375 0.50 MB/sec execute 68 sec latency 94.033 ms 1 3539 0.54 MB/sec execute 69 sec latency 76.448 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Jul 7 13:42:16 EDT 2026 with return code 0 1 3633 0.54 MB/sec execute 70 sec latency 68.545 ms 47164 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47164 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished