== replay-dual test 26: dbench and tar with mds failover ========================================================== 11:51:00 (1785081060) Starting client oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre Started clients oleg351-client.virtnet: 192.168.203.151@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 46502 Started dbench loop with 46503 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Jul 26 11:51:04 EDT 2026 waiting for dbench pid 46542 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1716 1269588 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21888 3579848 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 23300 7185456 1% /mnt/lustre 1 31 1.05 MB/sec warmup 1 sec latency 120.215 ms 1 94 2.10 MB/sec warmup 2 sec latency 129.795 ms 1 137 2.01 MB/sec warmup 3 sec latency 150.790 ms 1 169 1.72 MB/sec warmup 4 sec latency 161.331 ms 1 197 1.57 MB/sec warmup 5 sec latency 452.106 ms 1 198 1.31 MB/sec warmup 6 sec latency 926.704 ms 1 231 1.27 MB/sec warmup 7 sec latency 965.151 ms 1 275 1.28 MB/sec warmup 8 sec latency 121.421 ms 1 317 1.27 MB/sec warmup 9 sec latency 99.834 ms test_26 fail mds1 1 times 1 361 1.28 MB/sec warmup 10 sec latency 117.173 ms Failing mds1 on oleg351-server 1 406 1.30 MB/sec warmup 11 sec latency 130.134 ms Stopping /mnt/lustre-mds1 (opts:) on oleg351-server 1 459 1.34 MB/sec warmup 12 sec latency 103.252 ms 1 485 1.30 MB/sec warmup 13 sec latency 443.351 ms 1 485 1.21 MB/sec warmup 14 sec latency 1444.000 ms 11:51:19 (1785081079) shut down facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server 1 485 1.13 MB/sec warmup 15 sec latency 2445.684 ms 1 485 1.06 MB/sec warmup 16 sec latency 3447.074 ms 1 485 1.00 MB/sec warmup 17 sec latency 4447.885 ms 1 485 0.94 MB/sec warmup 18 sec latency 5448.178 ms 1 485 0.89 MB/sec warmup 19 sec latency 6449.670 ms 1 485 0.00 MB/sec execute 1 sec latency 8450.621 ms 1 485 0.00 MB/sec execute 2 sec latency 9453.092 ms 1 485 0.00 MB/sec execute 3 sec latency 10460.576 ms 1 485 0.00 MB/sec execute 4 sec latency 11461.740 ms 1 485 0.00 MB/sec execute 5 sec latency 12462.235 ms Failover mds1 to oleg351-server mount facets: mds1 1 485 0.00 MB/sec execute 6 sec latency 13462.424 ms 1 485 0.00 MB/sec execute 7 sec latency 14463.442 ms 1 485 0.00 MB/sec execute 8 sec latency 15464.285 ms 1 485 0.00 MB/sec execute 9 sec latency 16466.438 ms 1 485 0.00 MB/sec execute 10 sec latency 17467.180 ms 1 485 0.00 MB/sec execute 11 sec latency 18467.579 ms 1 485 0.00 MB/sec execute 12 sec latency 19467.886 ms 1 485 0.00 MB/sec execute 13 sec latency 20468.083 ms 1 485 0.00 MB/sec execute 14 sec latency 21471.439 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 485 0.00 MB/sec execute 15 sec latency 22471.628 ms 1 485 0.00 MB/sec execute 16 sec latency 23471.823 ms 1 485 0.00 MB/sec execute 17 sec latency 24472.944 ms 1 485 0.00 MB/sec execute 18 sec latency 25473.564 ms 1 485 0.00 MB/sec execute 19 sec latency 26473.740 ms 1 485 0.00 MB/sec execute 20 sec latency 27474.810 ms 1 485 0.00 MB/sec execute 21 sec latency 28475.185 ms 1 485 0.00 MB/sec execute 22 sec latency 29476.789 ms 1 485 0.00 MB/sec execute 23 sec latency 30477.005 ms 1 485 0.00 MB/sec execute 24 sec latency 31477.268 ms 1 485 0.00 MB/sec execute 25 sec latency 32477.575 ms 1 485 0.00 MB/sec execute 26 sec latency 33479.252 ms 1 485 0.00 MB/sec execute 27 sec latency 34480.415 ms 1 485 0.00 MB/sec execute 28 sec latency 35483.998 ms oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 1 485 0.00 MB/sec execute 29 sec latency 36485.379 ms pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 1 485 0.00 MB/sec execute 30 sec latency 37485.618 ms Started lustre-MDT0000 11:51:55 (1785081115) targets are mounted 11:51:55 (1785081115) facet_failover done 1 485 0.00 MB/sec execute 31 sec latency 38487.704 ms 1 485 0.00 MB/sec execute 32 sec latency 39488.914 ms 1 485 0.00 MB/sec execute 33 sec latency 40489.650 ms 1 485 0.00 MB/sec execute 34 sec latency 41489.901 ms 1 493 0.01 MB/sec execute 35 sec latency 42265.053 ms 1 524 0.03 MB/sec execute 36 sec latency 164.495 ms 1 607 0.11 MB/sec execute 37 sec latency 129.269 ms 1 654 0.11 MB/sec execute 38 sec latency 121.898 ms 1 686 0.11 MB/sec execute 39 sec latency 159.760 ms oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 718 0.11 MB/sec execute 40 sec latency 119.863 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 740 0.11 MB/sec execute 41 sec latency 128.740 ms 1 777 0.11 MB/sec execute 42 sec latency 144.089 ms 1 803 0.11 MB/sec execute 43 sec latency 132.545 ms 1 860 0.11 MB/sec execute 44 sec latency 102.465 ms 1 884 0.10 MB/sec execute 45 sec latency 246.623 ms 1 884 0.10 MB/sec execute 46 sec latency 1257.435 ms 1 914 0.10 MB/sec execute 47 sec latency 1335.264 ms 1 949 0.10 MB/sec execute 48 sec latency 129.655 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1820 1269484 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 8076 3596516 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 46992 3557840 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 55068 7154356 1% /mnt/lustre 1 980 0.10 MB/sec execute 49 sec latency 136.791 ms 1 1038 0.12 MB/sec execute 50 sec latency 84.323 ms 1 1076 0.12 MB/sec execute 51 sec latency 101.089 ms 1 1129 0.12 MB/sec execute 52 sec latency 68.487 ms 1 1162 0.11 MB/sec execute 53 sec latency 127.839 ms 1 1198 0.11 MB/sec execute 54 sec latency 87.619 ms 1 1216 0.11 MB/sec execute 55 sec latency 566.119 ms 1 1243 0.11 MB/sec execute 56 sec latency 128.821 ms 1 1296 0.11 MB/sec execute 57 sec latency 116.712 ms test_26 fail mds1 2 times 1 1410 0.16 MB/sec execute 58 sec latency 104.571 ms 1 1446 0.15 MB/sec execute 59 sec latency 129.532 ms Failing mds1 on oleg351-server 1 1474 0.15 MB/sec execute 60 sec latency 314.180 ms Stopping /mnt/lustre-mds1 (opts:) on oleg351-server 1 1509 0.15 MB/sec execute 61 sec latency 98.803 ms 1 1540 0.15 MB/sec execute 62 sec latency 171.174 ms 1 1540 0.15 MB/sec execute 63 sec latency 1100.510 ms 1 1540 0.14 MB/sec execute 64 sec latency 2101.350 ms 11:52:29 (1785081149) shut down facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server 1 1540 0.14 MB/sec execute 65 sec latency 3101.893 ms 1 1540 0.14 MB/sec execute 66 sec latency 4102.229 ms 1 1540 0.14 MB/sec execute 67 sec latency 5102.424 ms 1 1540 0.13 MB/sec execute 68 sec latency 6103.253 ms 1 1540 0.13 MB/sec execute 69 sec latency 7103.538 ms 1 1540 0.13 MB/sec execute 70 sec latency 8104.077 ms 1 1540 0.13 MB/sec execute 71 sec latency 9106.531 ms 1 1540 0.13 MB/sec execute 72 sec latency 10107.742 ms 1 1540 0.13 MB/sec execute 73 sec latency 11112.631 ms 1 1540 0.12 MB/sec execute 74 sec latency 12112.944 ms Failover mds1 to oleg351-server 1 1540 0.12 MB/sec execute 75 sec latency 13113.211 ms mount facets: mds1 1 1540 0.12 MB/sec execute 76 sec latency 14114.352 ms 1 1540 0.12 MB/sec execute 77 sec latency 15114.591 ms 1 1540 0.12 MB/sec execute 78 sec latency 16114.783 ms 1 1540 0.12 MB/sec execute 79 sec latency 17115.234 ms 1 1540 0.11 MB/sec execute 80 sec latency 18121.301 ms 1 1540 0.11 MB/sec execute 81 sec latency 19126.186 ms 1 1540 0.11 MB/sec execute 82 sec latency 20127.956 ms 1 1540 0.11 MB/sec execute 83 sec latency 21128.150 ms 1 1540 0.11 MB/sec execute 84 sec latency 22128.375 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1540 0.11 MB/sec execute 85 sec latency 23128.636 ms 1 1540 0.11 MB/sec execute 86 sec latency 24128.840 ms 1 1540 0.11 MB/sec execute 87 sec latency 25132.171 ms 1 1540 0.10 MB/sec execute 88 sec latency 26133.092 ms 1 1540 0.10 MB/sec execute 89 sec latency 27134.010 ms 1 1540 0.10 MB/sec execute 90 sec latency 28134.231 ms 1 1540 0.10 MB/sec execute 91 sec latency 29135.490 ms 1 1540 0.10 MB/sec execute 92 sec latency 30135.716 ms 1 1540 0.10 MB/sec execute 93 sec latency 31136.055 ms 1 1540 0.10 MB/sec execute 94 sec latency 32136.254 ms 1 1540 0.10 MB/sec execute 95 sec latency 33143.350 ms 1 1540 0.10 MB/sec execute 96 sec latency 34145.955 ms oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 1 1540 0.09 MB/sec execute 97 sec latency 35147.101 ms pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 1 1540 0.09 MB/sec execute 98 sec latency 36147.286 ms Started lustre-MDT0000 11:53:03 (1785081183) targets are mounted 11:53:03 (1785081183) facet_failover done 1 1540 0.09 MB/sec execute 99 sec latency 37148.035 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 0 cleanup 108 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 159 334.925 42265.038 Close 134 17.662 566.096 Rename 10 101.715 134.459 Unlink 20 30.915 52.762 Qpathinfo 126 364.395 42991.631 Qfileinfo 44 2.237 10.210 Qfsinfo 33 3.184 19.979 Sfileinfo 34 65.013 132.530 Find 54 7.571 92.844 WriteX 180 13.758 87.142 ReadX 237 3.214 74.708 Flush 25 60.956 159.738 Throughput 0.0923538 MB/sec 1 clients 1 procs max_latency=42265.053 ms stopping dbench on /mnt/lustre at Sun Jul 26 11:53:13 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Jul 26 11:53:19 EDT 2026 waiting for dbench pid 48629 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 59.755 ms 1 94 2.09 MB/sec warmup 2 sec latency 113.679 ms 1 137 2.01 MB/sec warmup 3 sec latency 150.088 ms 1 165 1.70 MB/sec warmup 4 sec latency 154.453 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1269340 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 10556 3596304 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36404 3569364 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 46960 7165668 1% /mnt/lustre 1 189 1.52 MB/sec warmup 5 sec latency 436.150 ms 1 218 1.39 MB/sec warmup 6 sec latency 148.141 ms 1 252 1.36 MB/sec warmup 7 sec latency 155.209 ms 1 289 1.34 MB/sec warmup 8 sec latency 134.306 ms 1 316 1.27 MB/sec warmup 9 sec latency 158.393 ms 1 331 1.20 MB/sec warmup 10 sec latency 668.794 ms 1 331 1.09 MB/sec warmup 11 sec latency 1669.003 ms 1 349 1.04 MB/sec warmup 12 sec latency 1753.660 ms 1 381 1.04 MB/sec warmup 13 sec latency 203.607 ms 1 413 1.04 MB/sec warmup 14 sec latency 158.941 ms test_26 fail mds1 3 times 1 446 1.04 MB/sec warmup 15 sec latency 140.789 ms 1 477 1.04 MB/sec warmup 16 sec latency 150.762 ms Failing mds1 on oleg351-server 1 510 1.04 MB/sec warmup 17 sec latency 153.402 ms Stopping /mnt/lustre-mds1 (opts:) on oleg351-server 1 553 1.06 MB/sec warmup 18 sec latency 123.332 ms 1 606 1.10 MB/sec warmup 19 sec latency 504.481 ms 1 606 0.00 MB/sec execute 1 sec latency 2507.299 ms 11:53:40 (1785081220) shut down facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server 1 606 0.00 MB/sec execute 2 sec latency 3507.615 ms 1 606 0.00 MB/sec execute 3 sec latency 4507.995 ms 1 606 0.00 MB/sec execute 4 sec latency 5508.260 ms 1 606 0.00 MB/sec execute 5 sec latency 6510.118 ms 1 606 0.00 MB/sec execute 6 sec latency 7512.788 ms 1 606 0.00 MB/sec execute 7 sec latency 8513.064 ms 1 606 0.00 MB/sec execute 8 sec latency 9513.281 ms 1 606 0.00 MB/sec execute 9 sec latency 10513.536 ms 1 606 0.00 MB/sec execute 10 sec latency 11513.761 ms 1 606 0.00 MB/sec execute 11 sec latency 12513.971 ms Failover mds1 to oleg351-server mount facets: mds1 1 606 0.00 MB/sec execute 12 sec latency 13514.254 ms 1 606 0.00 MB/sec execute 13 sec latency 14519.650 ms 1 606 0.00 MB/sec execute 14 sec latency 15520.017 ms 1 606 0.00 MB/sec execute 15 sec latency 16520.207 ms 1 606 0.00 MB/sec execute 16 sec latency 17520.706 ms 1 606 0.00 MB/sec execute 17 sec latency 18521.256 ms 1 606 0.00 MB/sec execute 18 sec latency 19521.462 ms 1 606 0.00 MB/sec execute 19 sec latency 20524.267 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 606 0.00 MB/sec execute 20 sec latency 21524.506 ms 1 606 0.00 MB/sec execute 21 sec latency 22525.095 ms 1 606 0.00 MB/sec execute 22 sec latency 23527.766 ms 1 606 0.00 MB/sec execute 23 sec latency 24527.952 ms 1 606 0.00 MB/sec execute 24 sec latency 25530.892 ms 1 606 0.00 MB/sec execute 25 sec latency 26531.509 ms 1 606 0.00 MB/sec execute 26 sec latency 27531.812 ms 1 606 0.00 MB/sec execute 27 sec latency 28532.110 ms 1 606 0.00 MB/sec execute 28 sec latency 29537.567 ms 1 606 0.00 MB/sec execute 29 sec latency 30537.951 ms 1 606 0.00 MB/sec execute 30 sec latency 31538.244 ms 1 606 0.00 MB/sec execute 31 sec latency 32540.784 ms oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 1 606 0.00 MB/sec execute 32 sec latency 33541.022 ms 1 606 0.00 MB/sec execute 33 sec latency 34541.276 ms pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 1 606 0.00 MB/sec execute 34 sec latency 35543.932 ms Started lustre-MDT0000 11:54:14 (1785081254) targets are mounted 11:54:14 (1785081254) facet_failover done 1 606 0.00 MB/sec execute 35 sec latency 36545.950 ms 1 606 0.00 MB/sec execute 36 sec latency 37551.767 ms 1 606 0.00 MB/sec execute 37 sec latency 38554.003 ms 1 606 0.00 MB/sec execute 38 sec latency 39554.697 ms 1 606 0.00 MB/sec execute 39 sec latency 40555.188 ms 1 606 0.00 MB/sec execute 40 sec latency 41555.442 ms 1 606 0.00 MB/sec execute 41 sec latency 42557.887 ms 1 606 0.00 MB/sec execute 42 sec latency 43558.751 ms 1 606 0.00 MB/sec execute 43 sec latency 44559.036 ms 1 616 0.00 MB/sec execute 44 sec latency 44601.694 ms 1 652 0.00 MB/sec execute 45 sec latency 218.694 ms 1 671 0.00 MB/sec execute 46 sec latency 246.965 ms 1 695 0.00 MB/sec execute 47 sec latency 152.753 ms 1 718 0.01 MB/sec execute 48 sec latency 206.372 ms 1 735 0.01 MB/sec execute 49 sec latency 150.467 ms 1 769 0.01 MB/sec execute 50 sec latency 146.962 ms 1 789 0.01 MB/sec execute 51 sec latency 276.698 ms oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 814 0.01 MB/sec execute 52 sec latency 155.796 ms 1 851 0.01 MB/sec execute 53 sec latency 147.221 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 876 0.01 MB/sec execute 54 sec latency 189.382 ms 1 884 0.01 MB/sec execute 55 sec latency 539.044 ms 1 885 0.01 MB/sec execute 56 sec latency 1386.341 ms 1 892 0.01 MB/sec execute 57 sec latency 658.763 ms 1 913 0.01 MB/sec execute 58 sec latency 171.713 ms 1 926 0.01 MB/sec execute 59 sec latency 173.490 ms 1 952 0.01 MB/sec execute 60 sec latency 132.792 ms 1 981 0.02 MB/sec execute 61 sec latency 148.682 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1011 0.02 MB/sec execute 62 sec latency 197.028 ms 1 1077 0.03 MB/sec execute 63 sec latency 203.623 ms 1 1132 0.03 MB/sec execute 64 sec latency 124.202 ms 1 1178 0.03 MB/sec execute 65 sec latency 162.923 ms 1 1215 0.03 MB/sec execute 66 sec latency 109.197 ms 1 1244 0.03 MB/sec execute 67 sec latency 150.826 ms 1 1379 0.07 MB/sec execute 68 sec latency 161.856 ms 1 1410 0.07 MB/sec execute 69 sec latency 96.350 ms 1 1447 0.07 MB/sec execute 70 sec latency 117.515 ms 1 1487 0.07 MB/sec execute 71 sec latency 204.915 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sun Jul 26 11:54:51 EDT 2026 with return code 0 1 1544 0.07 MB/sec execute 72 sec latency 78.560 ms 48629 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48629 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished