== replay-dual test 26: dbench and tar with mds failover ========================================================== 09:36:02 (1787751362) Starting client oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre Started clients oleg153-client.virtnet: 192.168.201.153@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45300 Started dbench loop with 45304 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3738624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7503872 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Aug 26 09:36:09 EDT 2026 waiting for dbench pid 45368 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 24 0.65 MB/sec warmup 1 sec latency 30.759 ms 1 58 1.18 MB/sec warmup 2 sec latency 228.082 ms 1 104 1.61 MB/sec warmup 3 sec latency 212.489 ms 1 134 1.47 MB/sec warmup 4 sec latency 228.317 ms 1 161 1.32 MB/sec warmup 5 sec latency 149.696 ms test_26 fail mds1 1 times 1 190 1.26 MB/sec warmup 6 sec latency 173.171 ms 1 215 1.19 MB/sec warmup 7 sec latency 207.748 ms Failing mds1 on oleg153-server 1 239 1.14 MB/sec warmup 8 sec latency 182.740 ms Stopping /mnt/lustre-mds1 (opts:) on oleg153-server 1 284 1.17 MB/sec warmup 9 sec latency 147.349 ms 1 304 1.12 MB/sec warmup 10 sec latency 453.102 ms 1 304 1.02 MB/sec warmup 11 sec latency 1453.609 ms 1 304 0.93 MB/sec warmup 12 sec latency 2455.517 ms 1 304 0.86 MB/sec warmup 13 sec latency 3456.002 ms 1 304 0.80 MB/sec warmup 14 sec latency 4457.497 ms 09:36:24 (1787751384) shut down facet: mds1 facet_host: oleg153-server facet_failover_host: oleg153-server 1 304 0.75 MB/sec warmup 15 sec latency 5462.034 ms 1 304 0.70 MB/sec warmup 16 sec latency 6462.372 ms 1 304 0.66 MB/sec warmup 17 sec latency 7462.809 ms 1 304 0.62 MB/sec warmup 18 sec latency 8463.207 ms 1 304 0.59 MB/sec warmup 19 sec latency 9463.489 ms 1 304 0.00 MB/sec execute 1 sec latency 11464.343 ms 1 304 0.00 MB/sec execute 2 sec latency 12464.795 ms 1 304 0.00 MB/sec execute 3 sec latency 13465.150 ms 1 304 0.00 MB/sec execute 4 sec latency 14465.435 ms 1 304 0.00 MB/sec execute 5 sec latency 15465.682 ms Failover mds1 to oleg153-server mount facets: mds1 1 304 0.00 MB/sec execute 6 sec latency 16465.945 ms 1 304 0.00 MB/sec execute 7 sec latency 17466.236 ms 1 304 0.00 MB/sec execute 8 sec latency 18469.277 ms 1 304 0.00 MB/sec execute 9 sec latency 19469.933 ms 1 304 0.00 MB/sec execute 10 sec latency 20470.188 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 304 0.00 MB/sec execute 11 sec latency 21470.479 ms 1 304 0.00 MB/sec execute 12 sec latency 22470.683 ms 1 304 0.00 MB/sec execute 13 sec latency 23470.896 ms 1 304 0.00 MB/sec execute 14 sec latency 24471.216 ms 1 304 0.00 MB/sec execute 15 sec latency 25471.729 ms 1 304 0.00 MB/sec execute 16 sec latency 26473.112 ms 1 304 0.00 MB/sec execute 17 sec latency 27473.370 ms 1 304 0.00 MB/sec execute 18 sec latency 28473.612 ms 1 304 0.00 MB/sec execute 19 sec latency 29474.366 ms 1 304 0.00 MB/sec execute 20 sec latency 30474.573 ms 1 304 0.00 MB/sec execute 21 sec latency 31477.267 ms 1 304 0.00 MB/sec execute 22 sec latency 32479.272 ms 1 304 0.00 MB/sec execute 23 sec latency 33480.521 ms 1 304 0.00 MB/sec execute 24 sec latency 34480.753 ms 1 304 0.00 MB/sec execute 25 sec latency 35481.731 ms 1 304 0.00 MB/sec execute 26 sec latency 36482.910 ms 1 304 0.00 MB/sec execute 27 sec latency 37483.379 ms 1 304 0.00 MB/sec execute 28 sec latency 38483.693 ms oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all 1 304 0.00 MB/sec execute 29 sec latency 39486.323 ms 1 304 0.00 MB/sec execute 30 sec latency 40489.318 ms pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 1 304 0.00 MB/sec execute 31 sec latency 41491.356 ms Started lustre-MDT0000 09:37:01 (1787751421) targets are mounted 09:37:01 (1787751421) facet_failover done 1 304 0.00 MB/sec execute 32 sec latency 42491.592 ms 1 304 0.00 MB/sec execute 33 sec latency 43512.545 ms 1 304 0.00 MB/sec execute 34 sec latency 44513.235 ms 1 304 0.00 MB/sec execute 35 sec latency 45513.544 ms 1 304 0.00 MB/sec execute 36 sec latency 46515.381 ms 1 304 0.00 MB/sec execute 37 sec latency 47515.604 ms 1 304 0.00 MB/sec execute 38 sec latency 48516.692 ms 1 304 0.00 MB/sec execute 39 sec latency 49516.941 ms 1 304 0.00 MB/sec execute 40 sec latency 50517.198 ms 1 304 0.00 MB/sec execute 41 sec latency 51517.959 ms 1 304 0.00 MB/sec execute 42 sec latency 52524.102 ms 1 304 0.00 MB/sec execute 43 sec latency 53524.877 ms 1 314 0.01 MB/sec execute 44 sec latency 53922.857 ms 1 341 0.02 MB/sec execute 45 sec latency 148.902 ms 1 374 0.05 MB/sec execute 46 sec latency 156.699 ms 1 407 0.07 MB/sec execute 47 sec latency 144.825 ms 1 437 0.09 MB/sec execute 48 sec latency 166.202 ms 1 477 0.11 MB/sec execute 49 sec latency 137.243 ms oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 507 0.13 MB/sec execute 50 sec latency 154.889 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 542 0.15 MB/sec execute 51 sec latency 224.161 ms 1 618 0.19 MB/sec execute 52 sec latency 93.035 ms 1 664 0.19 MB/sec execute 53 sec latency 139.476 ms 1 677 0.18 MB/sec execute 54 sec latency 322.442 ms 1 699 0.18 MB/sec execute 55 sec latency 193.727 ms 1 726 0.18 MB/sec execute 56 sec latency 179.042 ms 1 765 0.18 MB/sec execute 57 sec latency 104.329 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3695616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 12288 3717120 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 46080 7412736 1% /mnt/lustre 1 789 0.18 MB/sec execute 58 sec latency 249.014 ms 1 811 0.17 MB/sec execute 59 sec latency 304.608 ms 1 864 0.17 MB/sec execute 60 sec latency 99.535 ms 1 888 0.17 MB/sec execute 61 sec latency 275.327 ms 1 913 0.17 MB/sec execute 62 sec latency 135.482 ms 1 936 0.17 MB/sec execute 63 sec latency 331.475 ms test_26 fail mds1 2 times 1 970 0.17 MB/sec execute 64 sec latency 143.678 ms 1 985 0.16 MB/sec execute 65 sec latency 365.438 ms Failing mds1 on oleg153-server 1 1039 0.18 MB/sec execute 66 sec latency 157.836 ms Stopping /mnt/lustre-mds1 (opts:) on oleg153-server 1 1076 0.17 MB/sec execute 67 sec latency 97.360 ms 1 1124 0.17 MB/sec execute 68 sec latency 85.717 ms 1 1128 0.17 MB/sec execute 69 sec latency 754.096 ms 1 1128 0.17 MB/sec execute 70 sec latency 1755.490 ms 1 1128 0.17 MB/sec execute 71 sec latency 2757.127 ms 09:37:41 (1787751461) shut down facet: mds1 facet_host: oleg153-server facet_failover_host: oleg153-server 1 1128 0.16 MB/sec execute 72 sec latency 3757.374 ms 1 1128 0.16 MB/sec execute 73 sec latency 4758.187 ms 1 1128 0.16 MB/sec execute 74 sec latency 5758.396 ms 1 1128 0.16 MB/sec execute 75 sec latency 6758.606 ms 1 1128 0.15 MB/sec execute 76 sec latency 7758.799 ms 1 1128 0.15 MB/sec execute 77 sec latency 8759.111 ms 1 1128 0.15 MB/sec execute 78 sec latency 9759.445 ms 1 1128 0.15 MB/sec execute 79 sec latency 10760.756 ms 1 1128 0.15 MB/sec execute 80 sec latency 11760.985 ms 1 1128 0.15 MB/sec execute 81 sec latency 12761.220 ms Failover mds1 to oleg153-server mount facets: mds1 1 1128 0.14 MB/sec execute 82 sec latency 13768.185 ms 1 1128 0.14 MB/sec execute 83 sec latency 14769.543 ms 1 1128 0.14 MB/sec execute 84 sec latency 15771.749 ms 1 1128 0.14 MB/sec execute 85 sec latency 16772.023 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1128 0.14 MB/sec execute 86 sec latency 17772.399 ms 1 1128 0.14 MB/sec execute 87 sec latency 18776.077 ms 1 1128 0.13 MB/sec execute 88 sec latency 19776.579 ms 1 1128 0.13 MB/sec execute 89 sec latency 20782.603 ms 1 1128 0.13 MB/sec execute 90 sec latency 21785.532 ms 1 1128 0.13 MB/sec execute 91 sec latency 22785.770 ms 1 1128 0.13 MB/sec execute 92 sec latency 23785.968 ms 1 1128 0.13 MB/sec execute 93 sec latency 24786.598 ms 1 1128 0.12 MB/sec execute 94 sec latency 25786.844 ms 1 1128 0.12 MB/sec execute 95 sec latency 26787.258 ms 1 1128 0.12 MB/sec execute 96 sec latency 27787.460 ms 1 1128 0.12 MB/sec execute 97 sec latency 28787.585 ms 1 1128 0.12 MB/sec execute 98 sec latency 29787.815 ms 1 1128 0.12 MB/sec execute 99 sec latency 30788.647 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all 1 cleanup 105 sec pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 1 cleanup 106 sec 1 cleanup 107 sec Started lustre-MDT0000 09:38:17 (1787751497) targets are mounted 09:38:17 (1787751497) facet_failover done 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec 1 cleanup 114 sec 0 cleanup 115 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 116 90.451 239.911 Close 107 519.726 53922.832 Rename 5 120.668 167.235 Unlink 12 28.886 43.943 Qpathinfo 55 798.946 42572.880 Qfileinfo 53 2.313 18.093 Qfsinfo 26 3.338 16.402 Sfileinfo 22 76.104 152.660 Find 32 8.015 71.923 WriteX 260 15.470 98.924 ReadX 113 5.152 87.987 Flush 24 150.688 365.423 Throughput 0.118657 MB/sec 1 clients 1 procs max_latency=53922.857 ms stopping dbench on /mnt/lustre at Wed Aug 26 09:38:25 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Aug 26 09:38:32 EDT 2026 waiting for dbench pid 47344 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 39 1.36 MB/sec warmup 1 sec latency 108.720 ms 1 91 2.08 MB/sec warmup 2 sec latency 111.100 ms 1 126 1.87 MB/sec warmup 3 sec latency 252.056 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3719168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7465984 1% /mnt/lustre 1 147 1.56 MB/sec warmup 4 sec latency 184.411 ms 1 169 1.36 MB/sec warmup 5 sec latency 258.228 ms 1 196 1.30 MB/sec warmup 6 sec latency 195.810 ms 1 215 1.19 MB/sec warmup 7 sec latency 214.202 ms 1 240 1.14 MB/sec warmup 8 sec latency 193.208 ms 1 271 1.12 MB/sec warmup 9 sec latency 194.514 ms 1 296 1.09 MB/sec warmup 10 sec latency 243.471 ms 1 330 1.08 MB/sec warmup 11 sec latency 161.870 ms test_26 fail mds1 3 times 1 357 1.06 MB/sec warmup 12 sec latency 184.543 ms 1 386 1.05 MB/sec warmup 13 sec latency 197.591 ms 1 405 1.02 MB/sec warmup 14 sec latency 232.485 ms Failing mds1 on oleg153-server 1 428 1.00 MB/sec warmup 15 sec latency 242.695 ms 1 454 0.99 MB/sec warmup 16 sec latency 142.030 ms Stopping /mnt/lustre-mds1 (opts:) on oleg153-server 1 485 0.99 MB/sec warmup 17 sec latency 184.263 ms 1 510 0.98 MB/sec warmup 18 sec latency 165.060 ms 1 510 0.93 MB/sec warmup 19 sec latency 1106.974 ms 1 510 0.00 MB/sec execute 1 sec latency 3107.672 ms 1 510 0.00 MB/sec execute 2 sec latency 4109.526 ms 09:38:54 (1787751534) shut down facet: mds1 facet_host: oleg153-server facet_failover_host: oleg153-server 1 510 0.00 MB/sec execute 3 sec latency 5109.831 ms 1 510 0.00 MB/sec execute 4 sec latency 6110.875 ms 1 510 0.00 MB/sec execute 5 sec latency 7111.432 ms 1 510 0.00 MB/sec execute 6 sec latency 8111.704 ms 1 510 0.00 MB/sec execute 7 sec latency 9115.912 ms 1 510 0.00 MB/sec execute 8 sec latency 10119.073 ms 1 510 0.00 MB/sec execute 9 sec latency 11119.701 ms 1 510 0.00 MB/sec execute 10 sec latency 12120.589 ms 1 510 0.00 MB/sec execute 11 sec latency 13120.845 ms 1 510 0.00 MB/sec execute 12 sec latency 14121.380 ms Failover mds1 to oleg153-server mount facets: mds1 1 510 0.00 MB/sec execute 13 sec latency 15128.110 ms 1 510 0.00 MB/sec execute 14 sec latency 16128.997 ms 1 510 0.00 MB/sec execute 15 sec latency 17130.222 ms 1 510 0.00 MB/sec execute 16 sec latency 18133.984 ms 1 510 0.00 MB/sec execute 17 sec latency 19134.177 ms 1 510 0.00 MB/sec execute 18 sec latency 20134.851 ms 1 510 0.00 MB/sec execute 19 sec latency 21136.032 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 510 0.00 MB/sec execute 20 sec latency 22136.326 ms 1 510 0.00 MB/sec execute 21 sec latency 23138.731 ms 1 510 0.00 MB/sec execute 22 sec latency 24139.481 ms 1 510 0.00 MB/sec execute 23 sec latency 25139.838 ms 1 510 0.00 MB/sec execute 24 sec latency 26140.120 ms 1 510 0.00 MB/sec execute 25 sec latency 27156.585 ms 1 510 0.00 MB/sec execute 26 sec latency 28157.157 ms 1 510 0.00 MB/sec execute 27 sec latency 29157.372 ms 1 510 0.00 MB/sec execute 28 sec latency 30157.621 ms 1 510 0.00 MB/sec execute 29 sec latency 31165.979 ms 1 510 0.00 MB/sec execute 30 sec latency 32166.469 ms 1 510 0.00 MB/sec execute 31 sec latency 33166.735 ms 1 510 0.00 MB/sec execute 32 sec latency 34167.012 ms 1 510 0.00 MB/sec execute 33 sec latency 35167.273 ms 1 510 0.00 MB/sec execute 34 sec latency 36168.359 ms oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all 1 510 0.00 MB/sec execute 35 sec latency 37170.956 ms pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 1 510 0.00 MB/sec execute 36 sec latency 38171.192 ms 1 510 0.00 MB/sec execute 37 sec latency 39171.415 ms Started lustre-MDT0000 09:39:30 (1787751570) targets are mounted 09:39:30 (1787751570) facet_failover done 1 511 0.00 MB/sec execute 38 sec latency 40080.638 ms 1 532 0.02 MB/sec execute 39 sec latency 301.493 ms 1 603 0.08 MB/sec execute 40 sec latency 131.900 ms 1 652 0.08 MB/sec execute 41 sec latency 168.977 ms 1 669 0.08 MB/sec execute 42 sec latency 421.154 ms 1 689 0.08 MB/sec execute 43 sec latency 175.561 ms 1 709 0.08 MB/sec execute 44 sec latency 212.290 ms 1 721 0.08 MB/sec execute 45 sec latency 332.062 ms 1 733 0.08 MB/sec execute 46 sec latency 490.348 ms oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 759 0.08 MB/sec execute 47 sec latency 190.065 ms 1 778 0.08 MB/sec execute 48 sec latency 232.328 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 796 0.08 MB/sec execute 49 sec latency 246.403 ms 1 806 0.07 MB/sec execute 50 sec latency 196.536 ms 1 848 0.08 MB/sec execute 51 sec latency 199.243 ms 1 873 0.08 MB/sec execute 52 sec latency 288.349 ms 1 892 0.07 MB/sec execute 53 sec latency 350.054 ms 1 914 0.07 MB/sec execute 54 sec latency 224.189 ms 1 959 0.07 MB/sec execute 55 sec latency 100.572 ms 1 988 0.07 MB/sec execute 56 sec latency 241.645 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1056 0.09 MB/sec execute 57 sec latency 90.962 ms 1 1117 0.09 MB/sec execute 58 sec latency 111.627 ms 1 1155 0.09 MB/sec execute 59 sec latency 170.553 ms 1 1190 0.09 MB/sec execute 60 sec latency 93.188 ms 1 1223 0.09 MB/sec execute 61 sec latency 105.818 ms 1 1232 0.09 MB/sec execute 62 sec latency 408.632 ms 1 1255 0.08 MB/sec execute 63 sec latency 179.446 ms 1 1381 0.13 MB/sec execute 64 sec latency 132.989 ms 1 1426 0.13 MB/sec execute 65 sec latency 110.000 ms 1 1470 0.13 MB/sec execute 66 sec latency 89.405 ms 1 1493 0.12 MB/sec execute 67 sec latency 391.524 ms 1 1532 0.12 MB/sec execute 68 sec latency 146.143 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Aug 26 09:40:01 EDT 2026 with return code 0 47344 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47344 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished