== replay-dual test 26: dbench and tar with mds failover ========================================================== 00:37:31 (1788669451) Starting client oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre Started clients oleg249-client.virtnet: 192.168.202.149@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65674 Started dbench loop with 65676 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1976 1269328 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1804 1269500 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3588920 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3108 7194396 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Sep 6 00:37:38 EDT 2026 waiting for dbench pid 65752 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 38 1.31 MB/sec warmup 1 sec latency 83.259 ms 1 95 2.13 MB/sec warmup 2 sec latency 117.215 ms 1 122 1.83 MB/sec warmup 3 sec latency 237.105 ms 1 155 1.63 MB/sec warmup 4 sec latency 176.098 ms 1 181 1.46 MB/sec warmup 5 sec latency 234.314 ms 1 189 1.26 MB/sec warmup 6 sec latency 609.140 ms 1 239 1.30 MB/sec warmup 7 sec latency 187.341 ms test_26 fail mds1 1 times 1 279 1.30 MB/sec warmup 8 sec latency 126.906 ms 1 299 1.21 MB/sec warmup 9 sec latency 232.166 ms 1 328 1.18 MB/sec warmup 10 sec latency 154.144 ms Failing mds1 on oleg249-server 1 357 1.16 MB/sec warmup 11 sec latency 184.161 ms Stopping /mnt/lustre-mds1 (opts:) on oleg249-server 1 390 1.15 MB/sec warmup 12 sec latency 199.235 ms 1 420 1.14 MB/sec warmup 13 sec latency 162.538 ms 1 420 1.06 MB/sec warmup 14 sec latency 1076.488 ms 1 420 0.99 MB/sec warmup 15 sec latency 2076.863 ms 00:37:53 (1788669473) shut down facet: mds1 facet_host: oleg249-server facet_failover_host: oleg249-server 1 420 0.93 MB/sec warmup 16 sec latency 3077.115 ms 1 420 0.87 MB/sec warmup 17 sec latency 4077.541 ms 1 420 0.82 MB/sec warmup 18 sec latency 5080.430 ms 1 420 0.78 MB/sec warmup 19 sec latency 6080.701 ms 1 420 0.00 MB/sec execute 1 sec latency 8082.539 ms 1 420 0.00 MB/sec execute 2 sec latency 9083.113 ms 1 420 0.00 MB/sec execute 3 sec latency 10083.366 ms 1 420 0.00 MB/sec execute 4 sec latency 11083.577 ms 1 420 0.00 MB/sec execute 5 sec latency 12086.499 ms 1 420 0.00 MB/sec execute 6 sec latency 13086.702 ms Failover mds1 to oleg249-server mount facets: mds1 1 420 0.00 MB/sec execute 7 sec latency 14086.952 ms 1 420 0.00 MB/sec execute 8 sec latency 15087.258 ms 1 420 0.00 MB/sec execute 9 sec latency 16087.563 ms 1 420 0.00 MB/sec execute 10 sec latency 17101.764 ms 1 420 0.00 MB/sec execute 11 sec latency 18106.002 ms 1 420 0.00 MB/sec execute 12 sec latency 19106.738 ms 1 420 0.00 MB/sec execute 13 sec latency 20108.584 ms 1 420 0.00 MB/sec execute 14 sec latency 21110.836 ms 1 420 0.00 MB/sec execute 15 sec latency 22111.443 ms 1 420 0.00 MB/sec execute 16 sec latency 23113.100 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 420 0.00 MB/sec execute 17 sec latency 24113.608 ms 1 420 0.00 MB/sec execute 18 sec latency 25113.899 ms 1 420 0.00 MB/sec execute 19 sec latency 26114.314 ms 1 420 0.00 MB/sec execute 20 sec latency 27118.099 ms 1 420 0.00 MB/sec execute 21 sec latency 28132.063 ms 1 420 0.00 MB/sec execute 22 sec latency 29133.593 ms 1 420 0.00 MB/sec execute 23 sec latency 30146.043 ms 1 420 0.00 MB/sec execute 24 sec latency 31146.718 ms oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all 1 420 0.00 MB/sec execute 25 sec latency 32147.835 ms 1 420 0.00 MB/sec execute 26 sec latency 33148.465 ms pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 1 420 0.00 MB/sec execute 27 sec latency 34150.409 ms Started lustre-MDT0000 00:38:25 (1788669505) targets are mounted 00:38:25 (1788669505) facet_failover done 1 420 0.00 MB/sec execute 28 sec latency 35151.921 ms 1 430 0.01 MB/sec execute 29 sec latency 35643.348 ms 1 470 0.05 MB/sec execute 30 sec latency 158.138 ms 1 503 0.08 MB/sec execute 31 sec latency 150.559 ms 1 523 0.10 MB/sec execute 32 sec latency 208.960 ms 1 552 0.13 MB/sec execute 33 sec latency 259.542 ms 1 598 0.18 MB/sec execute 34 sec latency 674.343 ms oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 639 0.18 MB/sec execute 35 sec latency 134.252 ms 1 666 0.17 MB/sec execute 36 sec latency 156.299 ms 1 695 0.17 MB/sec execute 37 sec latency 140.677 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 713 0.17 MB/sec execute 38 sec latency 226.116 ms 1 736 0.16 MB/sec execute 39 sec latency 220.911 ms 1 762 0.16 MB/sec execute 40 sec latency 179.886 ms 1 780 0.16 MB/sec execute 41 sec latency 173.866 ms 1 808 0.16 MB/sec execute 42 sec latency 134.918 ms 1 856 0.16 MB/sec execute 43 sec latency 105.161 ms 1 888 0.15 MB/sec execute 44 sec latency 103.662 ms 1 914 0.15 MB/sec execute 45 sec latency 159.244 ms 1 936 0.15 MB/sec execute 46 sec latency 143.413 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2088 1269216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1900 1269404 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 35792 3565412 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13200 3584832 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 48992 7150244 1% /mnt/lustre 1 969 0.15 MB/sec execute 47 sec latency 140.117 ms 1 990 0.15 MB/sec execute 48 sec latency 123.440 ms 1 1048 0.16 MB/sec execute 49 sec latency 106.955 ms 1 1077 0.16 MB/sec execute 50 sec latency 122.185 ms 1 1126 0.16 MB/sec execute 51 sec latency 76.122 ms 1 1148 0.16 MB/sec execute 52 sec latency 177.125 ms 1 1172 0.15 MB/sec execute 53 sec latency 458.213 ms 1 1177 0.15 MB/sec execute 54 sec latency 1168.880 ms 1 1207 0.15 MB/sec execute 55 sec latency 109.025 ms 1 1244 0.15 MB/sec execute 56 sec latency 87.716 ms 1 1380 0.19 MB/sec execute 57 sec latency 100.446 ms test_26 fail mds2 2 times 1 1410 0.19 MB/sec execute 58 sec latency 121.281 ms 1 1442 0.19 MB/sec execute 59 sec latency 150.993 ms Failing mds2 on oleg249-server 1 1481 0.19 MB/sec execute 60 sec latency 120.337 ms 1 1509 0.18 MB/sec execute 61 sec latency 119.970 ms Stopping /mnt/lustre-mds2 (opts:) on oleg249-server 1 1554 0.18 MB/sec execute 62 sec latency 116.715 ms 1 1609 0.18 MB/sec execute 63 sec latency 82.166 ms 1 1657 0.18 MB/sec execute 64 sec latency 87.693 ms 1 1698 0.17 MB/sec execute 65 sec latency 99.374 ms 1 1758 0.17 MB/sec execute 66 sec latency 105.294 ms 1 1813 0.17 MB/sec execute 67 sec latency 112.216 ms 1 1884 0.17 MB/sec execute 68 sec latency 73.717 ms 1 1947 0.17 MB/sec execute 69 sec latency 159.748 ms 1 2000 0.17 MB/sec execute 70 sec latency 97.500 ms 00:39:08 (1788669548) shut down facet: mds2 facet_host: oleg249-server facet_failover_host: oleg249-server 1 2088 0.17 MB/sec execute 71 sec latency 99.980 ms 1 2156 0.16 MB/sec execute 72 sec latency 84.715 ms 1 2239 0.16 MB/sec execute 73 sec latency 66.870 ms 1 2259 0.16 MB/sec execute 74 sec latency 825.667 ms 1 2259 0.16 MB/sec execute 75 sec latency 1827.318 ms 1 2259 0.16 MB/sec execute 76 sec latency 2827.662 ms 1 2259 0.15 MB/sec execute 77 sec latency 3827.971 ms 1 2259 0.15 MB/sec execute 78 sec latency 4828.220 ms 1 2259 0.15 MB/sec execute 79 sec latency 5828.588 ms 1 2259 0.15 MB/sec execute 80 sec latency 6828.909 ms Failover mds2 to oleg249-server mount facets: mds2 1 2259 0.15 MB/sec execute 81 sec latency 7829.217 ms 1 2259 0.15 MB/sec execute 82 sec latency 8830.742 ms 1 2259 0.14 MB/sec execute 83 sec latency 9832.194 ms 1 2259 0.14 MB/sec execute 84 sec latency 10832.752 ms 1 2259 0.14 MB/sec execute 85 sec latency 11833.025 ms 1 2259 0.14 MB/sec execute 86 sec latency 12833.315 ms 1 2259 0.14 MB/sec execute 87 sec latency 13833.535 ms 1 2259 0.14 MB/sec execute 88 sec latency 14837.343 ms 1 2259 0.13 MB/sec execute 89 sec latency 15837.667 ms 1 2259 0.13 MB/sec execute 90 sec latency 16837.972 ms 1 2259 0.13 MB/sec execute 91 sec latency 17838.347 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.13 MB/sec execute 92 sec latency 18840.767 ms 1 2259 0.13 MB/sec execute 93 sec latency 19841.747 ms 1 2259 0.13 MB/sec execute 94 sec latency 20842.308 ms 1 2259 0.13 MB/sec execute 95 sec latency 21842.466 ms 1 2259 0.12 MB/sec execute 96 sec latency 22843.192 ms 1 2259 0.12 MB/sec execute 97 sec latency 23843.422 ms 1 2259 0.12 MB/sec execute 98 sec latency 24843.683 ms oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all 1 2259 0.12 MB/sec execute 99 sec latency 25843.935 ms pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0001 1 cleanup 100 sec 00:39:38 (1788669578) targets are mounted 00:39:38 (1788669578) facet_failover done 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 330 142.777 27148.905 Close 287 139.516 35643.319 Rename 11 113.809 177.091 Unlink 30 32.876 87.426 Qpathinfo 295 15.911 122.117 Qfileinfo 79 2.847 22.105 Qfsinfo 33 4.957 18.246 Sfileinfo 34 77.846 220.889 Find 92 14.104 93.783 WriteX 271 15.713 81.403 ReadX 348 3.113 116.685 Flush 30 66.733 159.701 Throughput 0.120162 MB/sec 1 clients 1 procs max_latency=35643.348 ms stopping dbench on /mnt/lustre at Sun Sep 6 00:39:42 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Sep 6 00:39:53 EDT 2026 waiting for dbench pid 67864 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 59 2.42 MB/sec warmup 1 sec latency 46.039 ms 1 133 2.91 MB/sec warmup 2 sec latency 92.006 ms 1 157 2.18 MB/sec warmup 3 sec latency 214.637 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2316 1268988 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2224 1269080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 10616 3596172 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36600 3568920 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 47216 7165092 1% /mnt/lustre 1 184 1.85 MB/sec warmup 4 sec latency 171.824 ms 1 199 1.57 MB/sec warmup 5 sec latency 594.778 ms 1 228 1.46 MB/sec warmup 6 sec latency 191.770 ms 1 255 1.38 MB/sec warmup 7 sec latency 130.466 ms 1 283 1.31 MB/sec warmup 8 sec latency 169.088 ms 1 315 1.27 MB/sec warmup 9 sec latency 149.655 ms 1 322 1.17 MB/sec warmup 10 sec latency 783.692 ms 1 342 1.11 MB/sec warmup 11 sec latency 1096.216 ms 1 382 1.13 MB/sec warmup 12 sec latency 138.938 ms 1 433 1.17 MB/sec warmup 13 sec latency 86.208 ms test_26 fail mds1 3 times 1 477 1.19 MB/sec warmup 14 sec latency 122.292 ms 1 504 1.16 MB/sec warmup 15 sec latency 165.400 ms Failing mds1 on oleg249-server 1 542 1.19 MB/sec warmup 16 sec latency 210.323 ms Stopping /mnt/lustre-mds1 (opts:) on oleg249-server 1 616 1.23 MB/sec warmup 17 sec latency 163.829 ms 1 664 1.17 MB/sec warmup 18 sec latency 164.942 ms 1 695 1.11 MB/sec warmup 19 sec latency 214.972 ms 1 799 0.13 MB/sec execute 1 sec latency 123.594 ms 1 845 0.11 MB/sec execute 2 sec latency 139.691 ms 1 874 0.10 MB/sec execute 3 sec latency 128.576 ms 00:40:17 (1788669617) shut down facet: mds1 facet_host: oleg249-server facet_failover_host: oleg249-server 1 884 0.08 MB/sec execute 4 sec latency 571.708 ms 1 885 0.06 MB/sec execute 5 sec latency 1388.979 ms 1 921 0.07 MB/sec execute 6 sec latency 256.095 ms 1 970 0.07 MB/sec execute 7 sec latency 102.485 ms 1 995 0.07 MB/sec execute 8 sec latency 349.182 ms 1 995 0.06 MB/sec execute 9 sec latency 1349.595 ms 1 995 0.06 MB/sec execute 10 sec latency 2349.898 ms 1 995 0.05 MB/sec execute 11 sec latency 3350.698 ms 1 995 0.05 MB/sec execute 12 sec latency 4350.933 ms 1 995 0.04 MB/sec execute 13 sec latency 5351.187 ms 1 995 0.04 MB/sec execute 14 sec latency 6351.774 ms Failover mds1 to oleg249-server mount facets: mds1 1 995 0.04 MB/sec execute 15 sec latency 7353.286 ms 1 995 0.04 MB/sec execute 16 sec latency 8354.904 ms 1 995 0.03 MB/sec execute 17 sec latency 9355.675 ms 1 995 0.03 MB/sec execute 18 sec latency 10356.033 ms 1 995 0.03 MB/sec execute 19 sec latency 11364.490 ms 1 995 0.03 MB/sec execute 20 sec latency 12365.641 ms 1 995 0.03 MB/sec execute 21 sec latency 13365.835 ms 1 995 0.03 MB/sec execute 22 sec latency 14379.161 ms 1 995 0.02 MB/sec execute 23 sec latency 15380.112 ms 1 995 0.02 MB/sec execute 24 sec latency 16384.609 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.02 MB/sec execute 25 sec latency 17385.177 ms 1 995 0.02 MB/sec execute 26 sec latency 18386.613 ms 1 995 0.02 MB/sec execute 27 sec latency 19387.957 ms 1 995 0.02 MB/sec execute 28 sec latency 20388.205 ms 1 995 0.02 MB/sec execute 29 sec latency 21390.071 ms 1 995 0.02 MB/sec execute 30 sec latency 22392.162 ms 1 995 0.02 MB/sec execute 31 sec latency 23393.173 ms 1 995 0.02 MB/sec execute 32 sec latency 24400.542 ms 1 995 0.02 MB/sec execute 33 sec latency 25402.341 ms 1 995 0.02 MB/sec execute 34 sec latency 26402.862 ms 1 995 0.02 MB/sec execute 35 sec latency 27403.288 ms 1 995 0.02 MB/sec execute 36 sec latency 28407.769 ms 1 995 0.02 MB/sec execute 37 sec latency 29408.001 ms oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all 1 995 0.01 MB/sec execute 38 sec latency 30408.480 ms pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 1 995 0.01 MB/sec execute 39 sec latency 31412.311 ms Started lustre-MDT0000 00:40:53 (1788669653) targets are mounted 00:40:53 (1788669653) facet_failover done 1 995 0.01 MB/sec execute 40 sec latency 32413.901 ms 1 1009 0.02 MB/sec execute 41 sec latency 32811.101 ms 1 1041 0.04 MB/sec execute 42 sec latency 264.559 ms 1 1080 0.04 MB/sec execute 43 sec latency 123.474 ms 1 1129 0.04 MB/sec execute 44 sec latency 102.794 ms 1 1181 0.04 MB/sec execute 45 sec latency 91.430 ms 1 1223 0.04 MB/sec execute 46 sec latency 108.677 ms 1 1252 0.04 MB/sec execute 47 sec latency 164.574 ms oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1285 0.04 MB/sec execute 48 sec latency 137.947 ms 1 1404 0.09 MB/sec execute 49 sec latency 127.256 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1445 0.09 MB/sec execute 50 sec latency 129.712 ms 1 1485 0.09 MB/sec execute 51 sec latency 127.076 ms 1 1523 0.09 MB/sec execute 52 sec latency 112.819 ms 1 1570 0.09 MB/sec execute 53 sec latency 109.196 ms 1 1609 0.09 MB/sec execute 54 sec latency 100.026 ms 1 1654 0.09 MB/sec execute 55 sec latency 103.057 ms 1 1689 0.09 MB/sec execute 56 sec latency 167.236 ms 1 1729 0.09 MB/sec execute 57 sec latency 127.999 ms 1 1783 0.09 MB/sec execute 58 sec latency 137.696 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1834 0.08 MB/sec execute 59 sec latency 102.221 ms 1 1884 0.08 MB/sec execute 60 sec latency 88.528 ms 1 1964 0.08 MB/sec execute 61 sec latency 144.364 ms 1 1986 0.08 MB/sec execute 62 sec latency 186.811 ms 1 2022 0.08 MB/sec execute 63 sec latency 160.209 ms 1 2087 0.08 MB/sec execute 64 sec latency 117.793 ms 1 2141 0.08 MB/sec execute 65 sec latency 100.566 ms 1 2168 0.08 MB/sec execute 66 sec latency 177.442 ms 1 2247 0.08 MB/sec execute 67 sec latency 78.140 ms 1 2334 0.10 MB/sec execute 68 sec latency 94.722 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sun Sep 6 00:41:22 EDT 2026 with return code 0 1 2373 0.09 MB/sec execute 69 sec latency 103.785 ms 67864 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 67864 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished