== replay-dual test 26: dbench and tar with mds failover ========================================================== 11:17:05 (1787843825) Starting client oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre Started clients oleg616-client.virtnet: 192.168.206.116@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45133 Started dbench loop with 45134 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3739648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 17408 7504896 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Aug 27 11:17:11 EDT 2026 waiting for dbench pid 45201 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 34 1.05 MB/sec warmup 1 sec latency 138.735 ms 1 96 2.16 MB/sec warmup 2 sec latency 161.662 ms 1 130 1.92 MB/sec warmup 3 sec latency 184.264 ms 1 165 1.70 MB/sec warmup 4 sec latency 135.361 ms test_26 fail mds1 1 times 1 214 1.67 MB/sec warmup 5 sec latency 140.548 ms Failing mds1 on oleg616-server 1 256 1.61 MB/sec warmup 6 sec latency 125.207 ms Stopping /mnt/lustre-mds1 (opts:) on oleg616-server 1 297 1.56 MB/sec warmup 7 sec latency 138.114 ms 1 331 1.50 MB/sec warmup 8 sec latency 291.169 ms 1 331 1.33 MB/sec warmup 9 sec latency 1291.363 ms 1 331 1.20 MB/sec warmup 10 sec latency 2292.395 ms 1 331 1.09 MB/sec warmup 11 sec latency 3292.596 ms 11:17:23 (1787843843) shut down 1 331 1.00 MB/sec warmup 12 sec latency 4292.766 ms facet: mds1 facet_host: oleg616-server facet_failover_host: oleg616-server 1 331 0.92 MB/sec warmup 13 sec latency 5293.976 ms 1 331 0.86 MB/sec warmup 14 sec latency 6294.161 ms 1 331 0.80 MB/sec warmup 15 sec latency 7294.365 ms 1 331 0.75 MB/sec warmup 16 sec latency 8297.524 ms 1 331 0.71 MB/sec warmup 17 sec latency 9298.132 ms 1 331 0.67 MB/sec warmup 18 sec latency 10298.343 ms 1 331 0.63 MB/sec warmup 19 sec latency 11300.247 ms 1 331 0.00 MB/sec execute 1 sec latency 13300.682 ms 1 331 0.00 MB/sec execute 2 sec latency 14300.953 ms Failover mds1 to oleg616-server mount facets: mds1 1 331 0.00 MB/sec execute 3 sec latency 15301.105 ms 1 331 0.00 MB/sec execute 4 sec latency 16301.313 ms 1 331 0.00 MB/sec execute 5 sec latency 17301.488 ms 1 331 0.00 MB/sec execute 6 sec latency 18303.298 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 331 0.00 MB/sec execute 7 sec latency 19303.480 ms 1 331 0.00 MB/sec execute 8 sec latency 20303.668 ms 1 331 0.00 MB/sec execute 9 sec latency 21304.518 ms 1 331 0.00 MB/sec execute 10 sec latency 22308.261 ms 1 331 0.00 MB/sec execute 11 sec latency 23308.696 ms 1 331 0.00 MB/sec execute 12 sec latency 24311.366 ms 1 331 0.00 MB/sec execute 13 sec latency 25311.571 ms oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all 1 331 0.00 MB/sec execute 14 sec latency 26312.852 ms pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 1 331 0.00 MB/sec execute 15 sec latency 27317.578 ms Started lustre-MDT0000 11:17:46 (1787843866) targets are mounted 11:17:46 (1787843866) facet_failover done 1 331 0.00 MB/sec execute 16 sec latency 28318.642 ms 1 339 0.02 MB/sec execute 17 sec latency 29164.625 ms 1 389 0.10 MB/sec execute 18 sec latency 131.728 ms 1 445 0.19 MB/sec execute 19 sec latency 112.084 ms 1 486 0.25 MB/sec execute 20 sec latency 122.230 ms oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 538 0.33 MB/sec execute 21 sec latency 109.986 ms 1 639 0.41 MB/sec execute 22 sec latency 87.687 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 673 0.40 MB/sec execute 23 sec latency 179.673 ms 1 710 0.38 MB/sec execute 24 sec latency 127.397 ms 1 738 0.37 MB/sec execute 25 sec latency 153.066 ms 1 781 0.36 MB/sec execute 26 sec latency 88.388 ms 1 808 0.35 MB/sec execute 27 sec latency 159.387 ms 1 863 0.34 MB/sec execute 28 sec latency 129.262 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3691520 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3708928 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 48128 7400448 1% /mnt/lustre 1 890 0.33 MB/sec execute 29 sec latency 201.195 ms 1 916 0.33 MB/sec execute 30 sec latency 177.553 ms 1 954 0.32 MB/sec execute 31 sec latency 165.944 ms 1 983 0.31 MB/sec execute 32 sec latency 157.431 ms 1 1058 0.33 MB/sec execute 33 sec latency 95.129 ms test_26 fail mds1 2 times 1 1124 0.32 MB/sec execute 34 sec latency 76.414 ms 1 1156 0.31 MB/sec execute 35 sec latency 144.804 ms Failing mds1 on oleg616-server 1 1193 0.31 MB/sec execute 36 sec latency 88.947 ms Stopping /mnt/lustre-mds1 (opts:) on oleg616-server 1 1232 0.30 MB/sec execute 37 sec latency 99.533 ms 1 1244 0.29 MB/sec execute 38 sec latency 748.878 ms 1 1244 0.28 MB/sec execute 39 sec latency 1749.598 ms 11:18:10 (1787843890) shut down facet: mds1 facet_host: oleg616-server facet_failover_host: oleg616-server 1 1244 0.28 MB/sec execute 40 sec latency 2749.871 ms 1 1244 0.27 MB/sec execute 41 sec latency 3755.724 ms 1 1244 0.26 MB/sec execute 42 sec latency 4755.907 ms 1 1244 0.26 MB/sec execute 43 sec latency 5756.074 ms 1 1244 0.25 MB/sec execute 44 sec latency 6757.245 ms 1 1244 0.25 MB/sec execute 45 sec latency 7757.452 ms 1 1244 0.24 MB/sec execute 46 sec latency 8757.655 ms 1 1244 0.24 MB/sec execute 47 sec latency 9758.336 ms 1 1244 0.23 MB/sec execute 48 sec latency 10758.525 ms 1 1244 0.23 MB/sec execute 49 sec latency 11758.754 ms Failover mds1 to oleg616-server mount facets: mds1 1 1244 0.22 MB/sec execute 50 sec latency 12760.981 ms 1 1244 0.22 MB/sec execute 51 sec latency 13761.324 ms 1 1244 0.21 MB/sec execute 52 sec latency 14761.508 ms 1 1244 0.21 MB/sec execute 53 sec latency 15761.665 ms 1 1244 0.20 MB/sec execute 54 sec latency 16761.863 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1244 0.20 MB/sec execute 55 sec latency 17764.232 ms 1 1244 0.20 MB/sec execute 56 sec latency 18764.512 ms 1 1244 0.19 MB/sec execute 57 sec latency 19764.809 ms 1 1244 0.19 MB/sec execute 58 sec latency 20765.283 ms 1 1244 0.19 MB/sec execute 59 sec latency 21766.740 ms 1 1244 0.18 MB/sec execute 60 sec latency 22766.968 ms 1 1244 0.18 MB/sec execute 61 sec latency 23767.584 ms oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all 1 1244 0.18 MB/sec execute 62 sec latency 24768.009 ms pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 1 1244 0.18 MB/sec execute 63 sec latency 25777.592 ms Started lustre-MDT0000 11:18:34 (1787843914) targets are mounted 11:18:34 (1787843914) facet_failover done 1 1244 0.17 MB/sec execute 64 sec latency 26779.185 ms 1 1244 0.17 MB/sec execute 65 sec latency 27779.966 ms 1 1244 0.17 MB/sec execute 66 sec latency 28781.400 ms 1 1244 0.17 MB/sec execute 67 sec latency 29781.569 ms 1 1244 0.16 MB/sec execute 68 sec latency 30781.749 ms 1 1244 0.16 MB/sec execute 69 sec latency 31784.647 ms 1 1244 0.16 MB/sec execute 70 sec latency 32785.468 ms 1 1244 0.16 MB/sec execute 71 sec latency 33785.906 ms 1 1244 0.15 MB/sec execute 72 sec latency 34786.074 ms 1 1244 0.15 MB/sec execute 73 sec latency 35787.082 ms 1 1244 0.15 MB/sec execute 74 sec latency 36788.981 ms 1 1244 0.15 MB/sec execute 75 sec latency 37789.182 ms 1 1244 0.15 MB/sec execute 76 sec latency 38791.414 ms 1 1244 0.14 MB/sec execute 77 sec latency 39791.631 ms 1 1244 0.14 MB/sec execute 78 sec latency 40793.091 ms 1 1259 0.14 MB/sec execute 79 sec latency 41490.749 ms 1 1393 0.17 MB/sec execute 80 sec latency 77.732 ms 1 1428 0.17 MB/sec execute 81 sec latency 135.367 ms 1 1474 0.17 MB/sec execute 82 sec latency 118.359 ms 1 1502 0.17 MB/sec execute 83 sec latency 355.969 ms 1 1538 0.17 MB/sec execute 84 sec latency 144.883 ms oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1603 0.17 MB/sec execute 85 sec latency 115.661 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1636 0.16 MB/sec execute 86 sec latency 216.773 ms 1 1678 0.16 MB/sec execute 87 sec latency 156.157 ms 1 1707 0.16 MB/sec execute 88 sec latency 158.692 ms 1 1775 0.16 MB/sec execute 89 sec latency 96.658 ms 1 1825 0.16 MB/sec execute 90 sec latency 127.755 ms 1 1884 0.16 MB/sec execute 91 sec latency 93.038 ms 1 1955 0.16 MB/sec execute 92 sec latency 206.456 ms UUID 1K-blocks Used Available Use% Mounted on 1 2002 0.16 MB/sec execute 93 sec latency 93.964 ms lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 47104 3712000 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 26624 3740672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7452672 1% /mnt/lustre 1 2069 0.16 MB/sec execute 94 sec latency 81.134 ms 1 2134 0.15 MB/sec execute 95 sec latency 107.745 ms 1 2166 0.15 MB/sec execute 96 sec latency 102.589 ms 1 2237 0.15 MB/sec execute 97 sec latency 93.696 ms 1 2338 0.16 MB/sec execute 98 sec latency 60.472 ms test_26 fail mds1 3 times 1 2376 0.16 MB/sec execute 99 sec latency 173.178 ms Failing mds1 on oleg616-server 1 cleanup 100 sec Stopping /mnt/lustre-mds1 (opts:) on oleg616-server 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 11:19:15 (1787843955) shut down facet: mds1 facet_host: oleg616-server facet_failover_host: oleg616-server 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec 1 cleanup 114 sec Failover mds1 to oleg616-server mount facets: mds1 1 cleanup 115 sec 1 cleanup 116 sec 1 cleanup 117 sec 1 cleanup 118 sec Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 119 sec 1 cleanup 120 sec 1 cleanup 121 sec 1 cleanup 122 sec 1 cleanup 123 sec 1 cleanup 124 sec oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all 1 cleanup 125 sec pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 1 cleanup 126 sec Started lustre-MDT0000 11:19:38 (1787843978) targets are mounted 11:19:38 (1787843978) facet_failover done 0 cleanup 127 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 358 135.600 29164.610 Close 311 12.278 39.853 Rename 13 3278.226 41490.730 Unlink 33 27.876 96.266 Qpathinfo 317 15.852 89.629 Qfileinfo 97 1.610 9.644 Qfsinfo 39 4.012 32.138 Sfileinfo 42 47.720 87.672 Find 107 5.808 44.249 WriteX 352 10.459 55.873 ReadX 381 1.961 87.230 Flush 32 127.394 355.954 Throughput 0.160974 MB/sec 1 clients 1 procs max_latency=41490.749 ms stopping dbench on /mnt/lustre at Thu Aug 27 11:19:38 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Aug 27 11:19:45 EDT 2026 waiting for dbench pid 47800 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 30 1.05 MB/sec warmup 1 sec latency 45.454 ms 1 71 1.57 MB/sec warmup 2 sec latency 199.314 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 113 1.74 MB/sec warmup 3 sec latency 180.192 ms 1 137 1.50 MB/sec warmup 4 sec latency 206.480 ms 1 157 1.30 MB/sec warmup 5 sec latency 246.302 ms 1 199 1.30 MB/sec warmup 6 sec latency 173.536 ms 1 231 1.26 MB/sec warmup 7 sec latency 164.531 ms 1 276 1.28 MB/sec warmup 8 sec latency 104.219 ms 1 315 1.27 MB/sec warmup 9 sec latency 149.528 ms UUID 1K-blocks Used Available Use% Mounted on 1 348 1.25 MB/sec warmup 10 sec latency 140.695 ms lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3688448 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3713024 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7401472 1% /mnt/lustre 1 386 1.24 MB/sec warmup 11 sec latency 198.038 ms 1 436 1.28 MB/sec warmup 12 sec latency 109.019 ms 1 485 1.30 MB/sec warmup 13 sec latency 88.652 ms 1 532 1.32 MB/sec warmup 14 sec latency 121.126 ms 1 639 1.40 MB/sec warmup 15 sec latency 120.979 ms test_26 fail mds1 4 times 1 671 1.32 MB/sec warmup 16 sec latency 255.752 ms Failing mds1 on oleg616-server 1 708 1.24 MB/sec warmup 17 sec latency 148.738 ms Stopping /mnt/lustre-mds1 (opts:) on oleg616-server 1 726 1.18 MB/sec warmup 18 sec latency 373.746 ms 1 747 1.12 MB/sec warmup 19 sec latency 211.498 ms 1 747 0.00 MB/sec execute 1 sec latency 2213.901 ms 1 747 0.00 MB/sec execute 2 sec latency 3214.632 ms 1 747 0.00 MB/sec execute 3 sec latency 4214.862 ms 11:20:08 (1787844008) shut down facet: mds1 facet_host: oleg616-server facet_failover_host: oleg616-server 1 747 0.00 MB/sec execute 4 sec latency 5215.159 ms 1 747 0.00 MB/sec execute 5 sec latency 6215.588 ms 1 747 0.00 MB/sec execute 6 sec latency 7215.758 ms 1 747 0.00 MB/sec execute 7 sec latency 8215.915 ms 1 747 0.00 MB/sec execute 8 sec latency 9217.573 ms 1 747 0.00 MB/sec execute 9 sec latency 10217.740 ms 1 747 0.00 MB/sec execute 10 sec latency 11220.698 ms 1 747 0.00 MB/sec execute 11 sec latency 12220.870 ms 1 747 0.00 MB/sec execute 12 sec latency 13221.081 ms 1 747 0.00 MB/sec execute 13 sec latency 14221.515 ms Failover mds1 to oleg616-server mount facets: mds1 1 747 0.00 MB/sec execute 14 sec latency 15221.886 ms 1 747 0.00 MB/sec execute 15 sec latency 16222.199 ms 1 747 0.00 MB/sec execute 16 sec latency 17223.335 ms 1 747 0.00 MB/sec execute 17 sec latency 18224.453 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 747 0.00 MB/sec execute 18 sec latency 19224.662 ms 1 747 0.00 MB/sec execute 19 sec latency 20224.854 ms 1 747 0.00 MB/sec execute 20 sec latency 21225.764 ms 1 747 0.00 MB/sec execute 21 sec latency 22225.992 ms 1 747 0.00 MB/sec execute 22 sec latency 23226.283 ms 1 747 0.00 MB/sec execute 23 sec latency 24227.683 ms 1 747 0.00 MB/sec execute 24 sec latency 25227.977 ms 1 747 0.00 MB/sec execute 25 sec latency 26228.154 ms 1 747 0.00 MB/sec execute 26 sec latency 27228.320 ms 1 747 0.00 MB/sec execute 27 sec latency 28228.504 ms 1 747 0.00 MB/sec execute 28 sec latency 29229.714 ms 1 747 0.00 MB/sec execute 29 sec latency 30229.849 ms 1 747 0.00 MB/sec execute 30 sec latency 31230.323 ms 1 747 0.00 MB/sec execute 31 sec latency 32230.502 ms oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all 1 747 0.00 MB/sec execute 32 sec latency 33230.632 ms pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 1 747 0.00 MB/sec execute 33 sec latency 34230.864 ms Started lustre-MDT0000 11:20:38 (1787844038) targets are mounted 11:20:38 (1787844038) facet_failover done 1 747 0.00 MB/sec execute 34 sec latency 35231.386 ms 1 747 0.00 MB/sec execute 35 sec latency 36232.779 ms 1 747 0.00 MB/sec execute 36 sec latency 37232.972 ms 1 747 0.00 MB/sec execute 37 sec latency 38233.205 ms 1 747 0.00 MB/sec execute 38 sec latency 39233.398 ms 1 747 0.00 MB/sec execute 39 sec latency 40234.171 ms 1 752 0.00 MB/sec execute 40 sec latency 41211.390 ms 1 779 0.00 MB/sec execute 41 sec latency 193.132 ms 1 804 0.00 MB/sec execute 42 sec latency 318.550 ms 1 861 0.01 MB/sec execute 43 sec latency 128.051 ms 1 885 0.01 MB/sec execute 44 sec latency 266.800 ms oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 913 0.01 MB/sec execute 45 sec latency 206.120 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 948 0.01 MB/sec execute 46 sec latency 230.887 ms 1 980 0.01 MB/sec execute 47 sec latency 234.126 ms 1 1001 0.02 MB/sec execute 48 sec latency 298.601 ms 1 1051 0.03 MB/sec execute 49 sec latency 173.031 ms 1 1128 0.03 MB/sec execute 50 sec latency 79.840 ms 1 1173 0.03 MB/sec execute 51 sec latency 117.838 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1233 0.03 MB/sec execute 52 sec latency 68.592 ms 1 1274 0.03 MB/sec execute 53 sec latency 128.585 ms 1 1417 0.09 MB/sec execute 54 sec latency 91.126 ms 1 1465 0.09 MB/sec execute 55 sec latency 106.158 ms 1 1499 0.08 MB/sec execute 56 sec latency 174.079 ms 1 1558 0.08 MB/sec execute 57 sec latency 86.081 ms 1 1604 0.08 MB/sec execute 58 sec latency 112.394 ms 1 1646 0.08 MB/sec execute 59 sec latency 308.353 ms 1 1689 0.08 MB/sec execute 60 sec latency 147.710 ms 1 1762 0.08 MB/sec execute 61 sec latency 75.142 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Aug 27 11:21:06 EDT 2026 with return code 0 47800 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47800 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished