== replay-dual test 26: dbench and tar with mds failover ========================================================== 16:45:49 (1790973949) Starting client oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre Started clients oleg303-client.virtnet: 192.168.203.103@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45160 Started dbench loop with 45161 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3730432 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 24576 7492608 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Oct 2 16:45:54 EDT 2026 waiting for dbench pid 45219 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 26 0.85 MB/sec warmup 1 sec latency 219.549 ms 1 75 1.60 MB/sec warmup 2 sec latency 178.337 ms 1 119 1.81 MB/sec warmup 3 sec latency 204.034 ms 1 155 1.63 MB/sec warmup 4 sec latency 118.598 ms 1 189 1.51 MB/sec warmup 5 sec latency 165.604 ms test_26 fail mds1 1 times 1 228 1.45 MB/sec warmup 6 sec latency 95.061 ms 1 262 1.41 MB/sec warmup 7 sec latency 137.820 ms Failing mds1 on oleg303-server 1 287 1.33 MB/sec warmup 8 sec latency 255.801 ms Stopping /mnt/lustre-mds1 (opts:) on oleg303-server 1 322 1.30 MB/sec warmup 9 sec latency 158.182 ms 1 339 1.22 MB/sec warmup 10 sec latency 607.439 ms 1 339 1.11 MB/sec warmup 11 sec latency 1607.574 ms 16:46:06 (1790973966) shut down 1 339 1.02 MB/sec warmup 12 sec latency 2608.170 ms facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server 1 339 0.94 MB/sec warmup 13 sec latency 3608.513 ms 1 339 0.87 MB/sec warmup 14 sec latency 4610.494 ms 1 339 0.82 MB/sec warmup 15 sec latency 5611.063 ms 1 339 0.77 MB/sec warmup 16 sec latency 6611.248 ms 1 339 0.72 MB/sec warmup 17 sec latency 7611.635 ms 1 339 0.68 MB/sec warmup 18 sec latency 8611.955 ms 1 339 0.64 MB/sec warmup 19 sec latency 9613.546 ms 1 339 0.00 MB/sec execute 1 sec latency 11614.437 ms 1 339 0.00 MB/sec execute 2 sec latency 12615.922 ms Failover mds1 to oleg303-server mount facets: mds1 1 339 0.00 MB/sec execute 3 sec latency 13616.151 ms 1 339 0.00 MB/sec execute 4 sec latency 14616.355 ms 1 339 0.00 MB/sec execute 5 sec latency 15616.553 ms 1 339 0.00 MB/sec execute 6 sec latency 16616.777 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 339 0.00 MB/sec execute 7 sec latency 17617.315 ms 1 339 0.00 MB/sec execute 8 sec latency 18617.481 ms 1 339 0.00 MB/sec execute 9 sec latency 19617.664 ms 1 339 0.00 MB/sec execute 10 sec latency 20617.854 ms 1 339 0.00 MB/sec execute 11 sec latency 21618.036 ms 1 339 0.00 MB/sec execute 12 sec latency 22618.349 ms 1 339 0.00 MB/sec execute 13 sec latency 23619.150 ms 1 339 0.00 MB/sec execute 14 sec latency 24619.356 ms 1 339 0.00 MB/sec execute 15 sec latency 25620.178 ms 1 339 0.00 MB/sec execute 16 sec latency 26620.410 ms 1 339 0.00 MB/sec execute 17 sec latency 27621.918 ms 1 339 0.00 MB/sec execute 18 sec latency 28622.081 ms 1 339 0.00 MB/sec execute 19 sec latency 29622.666 ms 1 339 0.00 MB/sec execute 20 sec latency 30625.881 ms 1 339 0.00 MB/sec execute 21 sec latency 31626.336 ms 1 339 0.00 MB/sec execute 22 sec latency 32626.500 ms oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all 1 339 0.00 MB/sec execute 23 sec latency 33628.914 ms pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 1 339 0.00 MB/sec execute 24 sec latency 34629.921 ms Started lustre-MDT0000 16:46:38 (1790973998) targets are mounted 16:46:38 (1790973998) facet_failover done 1 339 0.00 MB/sec execute 25 sec latency 35630.096 ms 1 339 0.00 MB/sec execute 26 sec latency 36630.676 ms 1 339 0.00 MB/sec execute 27 sec latency 37630.838 ms 1 339 0.00 MB/sec execute 28 sec latency 38631.077 ms 1 339 0.00 MB/sec execute 29 sec latency 39631.322 ms 1 339 0.00 MB/sec execute 30 sec latency 40633.414 ms 1 339 0.00 MB/sec execute 31 sec latency 41633.532 ms 1 369 0.03 MB/sec execute 32 sec latency 41776.201 ms 1 403 0.06 MB/sec execute 33 sec latency 129.957 ms 1 443 0.10 MB/sec execute 34 sec latency 110.021 ms 1 481 0.13 MB/sec execute 35 sec latency 124.379 ms oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 514 0.15 MB/sec execute 36 sec latency 131.169 ms 1 606 0.24 MB/sec execute 37 sec latency 131.019 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 654 0.23 MB/sec execute 38 sec latency 113.745 ms 1 674 0.23 MB/sec execute 39 sec latency 290.350 ms 1 710 0.22 MB/sec execute 40 sec latency 117.522 ms 1 733 0.22 MB/sec execute 41 sec latency 228.197 ms 1 777 0.22 MB/sec execute 42 sec latency 95.562 ms 1 801 0.21 MB/sec execute 43 sec latency 234.831 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3712000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 36864 3673088 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 47104 7385088 1% /mnt/lustre 1 847 0.21 MB/sec execute 44 sec latency 139.526 ms 1 876 0.21 MB/sec execute 45 sec latency 206.685 ms 1 897 0.20 MB/sec execute 46 sec latency 164.202 ms 1 918 0.20 MB/sec execute 47 sec latency 161.688 ms 1 951 0.20 MB/sec execute 48 sec latency 121.960 ms 1 983 0.20 MB/sec execute 49 sec latency 253.730 ms 1 1059 0.21 MB/sec execute 50 sec latency 72.300 ms test_26 fail mds1 2 times 1 1104 0.21 MB/sec execute 51 sec latency 94.488 ms Failing mds1 on oleg303-server 1 1146 0.21 MB/sec execute 52 sec latency 85.137 ms Stopping /mnt/lustre-mds1 (opts:) on oleg303-server 1 1182 0.20 MB/sec execute 53 sec latency 122.181 ms 1 1223 0.20 MB/sec execute 54 sec latency 100.005 ms 1 1223 0.20 MB/sec execute 55 sec latency 1079.897 ms 1 1223 0.19 MB/sec execute 56 sec latency 2084.613 ms 16:47:11 (1790974031) shut down facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server 1 1223 0.19 MB/sec execute 57 sec latency 3085.637 ms 1 1223 0.19 MB/sec execute 58 sec latency 4085.796 ms 1 1223 0.18 MB/sec execute 59 sec latency 5086.103 ms 1 1223 0.18 MB/sec execute 60 sec latency 6086.287 ms 1 1223 0.18 MB/sec execute 61 sec latency 7086.476 ms 1 1223 0.17 MB/sec execute 62 sec latency 8086.718 ms 1 1223 0.17 MB/sec execute 63 sec latency 9090.138 ms 1 1223 0.17 MB/sec execute 64 sec latency 10090.349 ms 1 1223 0.17 MB/sec execute 65 sec latency 11090.588 ms 1 1223 0.16 MB/sec execute 66 sec latency 12091.687 ms 1 1223 0.16 MB/sec execute 67 sec latency 13091.928 ms Failover mds1 to oleg303-server mount facets: mds1 1 1223 0.16 MB/sec execute 68 sec latency 14092.129 ms 1 1223 0.16 MB/sec execute 69 sec latency 15093.169 ms 1 1223 0.15 MB/sec execute 70 sec latency 16094.589 ms 1 1223 0.15 MB/sec execute 71 sec latency 17096.570 ms 1 1223 0.15 MB/sec execute 72 sec latency 18098.056 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1223 0.15 MB/sec execute 73 sec latency 19098.299 ms 1 1223 0.15 MB/sec execute 74 sec latency 20098.463 ms 1 1223 0.14 MB/sec execute 75 sec latency 21098.732 ms 1 1223 0.14 MB/sec execute 76 sec latency 22098.938 ms 1 1223 0.14 MB/sec execute 77 sec latency 23099.119 ms 1 1223 0.14 MB/sec execute 78 sec latency 24102.265 ms 1 1223 0.14 MB/sec execute 79 sec latency 25102.692 ms 1 1223 0.13 MB/sec execute 80 sec latency 26106.327 ms oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all 1 1244 0.13 MB/sec execute 81 sec latency 26216.762 ms pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 1 1274 0.13 MB/sec execute 82 sec latency 172.055 ms Started lustre-MDT0000 1 1382 0.17 MB/sec execute 83 sec latency 415.418 ms 16:47:37 (1790974057) targets are mounted 16:47:37 (1790974057) facet_failover done 1 1418 0.16 MB/sec execute 84 sec latency 103.451 ms 1 1451 0.16 MB/sec execute 85 sec latency 152.130 ms 1 1486 0.16 MB/sec execute 86 sec latency 197.143 ms 1 1522 0.16 MB/sec execute 87 sec latency 113.751 ms 1 1570 0.16 MB/sec execute 88 sec latency 96.292 ms 1 1618 0.16 MB/sec execute 89 sec latency 97.990 ms oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1653 0.15 MB/sec execute 90 sec latency 260.202 ms 1 1689 0.15 MB/sec execute 91 sec latency 158.530 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1729 0.15 MB/sec execute 92 sec latency 121.854 ms 1 1775 0.15 MB/sec execute 93 sec latency 149.952 ms 1 1820 0.15 MB/sec execute 94 sec latency 123.047 ms 1 1861 0.15 MB/sec execute 95 sec latency 106.411 ms 1 1911 0.15 MB/sec execute 96 sec latency 169.336 ms 1 1978 0.15 MB/sec execute 97 sec latency 118.941 ms 1 2014 0.15 MB/sec execute 98 sec latency 117.799 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3734528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 60416 3680256 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 72704 7414784 1% /mnt/lustre 1 2065 0.14 MB/sec execute 99 sec latency 107.438 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 312 198.720 41776.188 Close 264 13.945 54.769 Rename 11 102.410 172.032 Unlink 27 30.921 79.542 Qpathinfo 264 17.709 166.478 Qfileinfo 83 1.761 21.753 Qfsinfo 33 3.331 28.702 Sfileinfo 34 831.167 26216.747 Find 86 7.572 79.826 WriteX 315 12.211 71.406 ReadX 328 1.817 78.750 Flush 30 140.859 290.335 Throughput 0.144855 MB/sec 1 clients 1 procs max_latency=41776.201 ms stopping dbench on /mnt/lustre at Fri Oct 2 16:47:58 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished test_26 fail mds1 3 times Failing mds1 on oleg303-server looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt Stopping /mnt/lustre-mds1 (opts:) on oleg303-server 16:48:06 (1790974086) shut down facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server Failover mds1 to oleg303-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 16:48:38 (1790974118) targets are mounted 16:48:38 (1790974118) facet_failover done '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Oct 2 16:48:42 EDT 2026 waiting for dbench pid 47747 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 44 1.70 MB/sec warmup 1 sec latency 121.521 ms 1 92 2.10 MB/sec warmup 2 sec latency 218.986 ms 1 121 1.83 MB/sec warmup 3 sec latency 285.777 ms 1 147 1.57 MB/sec warmup 4 sec latency 213.987 ms oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 174 1.41 MB/sec warmup 5 sec latency 131.989 ms 1 200 1.31 MB/sec warmup 6 sec latency 190.052 ms 1 224 1.23 MB/sec warmup 7 sec latency 194.021 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 255 1.21 MB/sec warmup 8 sec latency 181.820 ms 1 286 1.19 MB/sec warmup 9 sec latency 161.594 ms 1 313 1.15 MB/sec warmup 10 sec latency 187.747 ms 1 340 1.11 MB/sec warmup 11 sec latency 189.025 ms 1 369 1.09 MB/sec warmup 12 sec latency 237.438 ms 1 414 1.12 MB/sec warmup 13 sec latency 93.898 ms 1 446 1.11 MB/sec warmup 14 sec latency 141.734 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3723264 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 55296 3670016 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 62464 7393280 1% /mnt/lustre 1 481 1.11 MB/sec warmup 15 sec latency 168.839 ms 1 510 1.10 MB/sec warmup 16 sec latency 197.062 ms 1 571 1.12 MB/sec warmup 17 sec latency 210.314 ms 1 641 1.17 MB/sec warmup 18 sec latency 129.831 ms 1 676 1.11 MB/sec warmup 19 sec latency 237.863 ms test_26 fail mds1 4 times 1 738 0.02 MB/sec execute 1 sec latency 190.121 ms 1 778 0.03 MB/sec execute 2 sec latency 110.772 ms Failing mds1 on oleg303-server 1 796 0.06 MB/sec execute 3 sec latency 252.308 ms Stopping /mnt/lustre-mds1 (opts:) on oleg303-server 1 845 0.07 MB/sec execute 4 sec latency 121.414 ms 1 864 0.06 MB/sec execute 5 sec latency 447.752 ms 1 864 0.05 MB/sec execute 6 sec latency 1450.145 ms 16:49:09 (1790974149) shut down 1 864 0.04 MB/sec execute 7 sec latency 2450.899 ms facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server 1 864 0.04 MB/sec execute 8 sec latency 3456.558 ms 1 864 0.03 MB/sec execute 9 sec latency 4456.746 ms 1 864 0.03 MB/sec execute 10 sec latency 5456.955 ms 1 864 0.03 MB/sec execute 11 sec latency 6457.361 ms 1 864 0.02 MB/sec execute 12 sec latency 7457.575 ms 1 864 0.02 MB/sec execute 13 sec latency 8457.771 ms 1 864 0.02 MB/sec execute 14 sec latency 9457.966 ms 1 864 0.02 MB/sec execute 15 sec latency 10458.546 ms 1 864 0.02 MB/sec execute 16 sec latency 11458.789 ms 1 864 0.02 MB/sec execute 17 sec latency 12459.052 ms Failover mds1 to oleg303-server mount facets: mds1 1 864 0.02 MB/sec execute 18 sec latency 13459.298 ms 1 864 0.01 MB/sec execute 19 sec latency 14459.477 ms 1 864 0.01 MB/sec execute 20 sec latency 15463.296 ms 1 864 0.01 MB/sec execute 21 sec latency 16465.536 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 864 0.01 MB/sec execute 22 sec latency 17466.479 ms 1 864 0.01 MB/sec execute 23 sec latency 18466.624 ms 1 864 0.01 MB/sec execute 24 sec latency 19466.817 ms 1 864 0.01 MB/sec execute 25 sec latency 20468.356 ms 1 864 0.01 MB/sec execute 26 sec latency 21468.518 ms 1 864 0.01 MB/sec execute 27 sec latency 22469.484 ms 1 864 0.01 MB/sec execute 28 sec latency 23469.826 ms 1 864 0.01 MB/sec execute 29 sec latency 24470.091 ms 1 864 0.01 MB/sec execute 30 sec latency 25470.278 ms 1 864 0.01 MB/sec execute 31 sec latency 26471.866 ms 1 864 0.01 MB/sec execute 32 sec latency 27472.436 ms 1 864 0.01 MB/sec execute 33 sec latency 28474.341 ms 1 864 0.01 MB/sec execute 34 sec latency 29479.184 ms 1 864 0.01 MB/sec execute 35 sec latency 30479.366 ms 1 864 0.01 MB/sec execute 36 sec latency 31480.212 ms 1 864 0.01 MB/sec execute 37 sec latency 32480.680 ms oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all 1 864 0.01 MB/sec execute 38 sec latency 33481.005 ms pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 1 864 0.01 MB/sec execute 39 sec latency 34482.538 ms Started lustre-MDT0000 16:49:42 (1790974182) targets are mounted 16:49:42 (1790974182) facet_failover done 1 864 0.01 MB/sec execute 40 sec latency 35484.708 ms 1 864 0.01 MB/sec execute 41 sec latency 36487.035 ms 1 864 0.01 MB/sec execute 42 sec latency 37487.772 ms 1 864 0.01 MB/sec execute 43 sec latency 38487.946 ms 1 864 0.01 MB/sec execute 44 sec latency 39489.422 ms 1 864 0.01 MB/sec execute 45 sec latency 40500.840 ms 1 885 0.01 MB/sec execute 46 sec latency 40570.084 ms 1 910 0.01 MB/sec execute 47 sec latency 158.357 ms 1 929 0.01 MB/sec execute 48 sec latency 233.413 ms 1 967 0.01 MB/sec execute 49 sec latency 123.535 ms oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 988 0.01 MB/sec execute 50 sec latency 259.175 ms 1 1052 0.03 MB/sec execute 51 sec latency 121.821 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1086 0.03 MB/sec execute 52 sec latency 116.294 ms 1 1132 0.03 MB/sec execute 53 sec latency 115.988 ms 1 1176 0.03 MB/sec execute 54 sec latency 135.901 ms 1 1215 0.03 MB/sec execute 55 sec latency 137.405 ms 1 1253 0.03 MB/sec execute 56 sec latency 144.639 ms 1 1389 0.08 MB/sec execute 57 sec latency 110.582 ms 1 1438 0.08 MB/sec execute 58 sec latency 103.834 ms 1 1489 0.08 MB/sec execute 59 sec latency 268.737 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1538 0.08 MB/sec execute 60 sec latency 96.763 ms 1 1605 0.08 MB/sec execute 61 sec latency 76.920 ms 1 1655 0.08 MB/sec execute 62 sec latency 157.903 ms 1 1696 0.08 MB/sec execute 63 sec latency 105.884 ms 1 1770 0.08 MB/sec execute 64 sec latency 112.314 ms 1 1834 0.08 MB/sec execute 65 sec latency 98.177 ms 1 1884 0.08 MB/sec execute 66 sec latency 156.832 ms 1 1944 0.08 MB/sec execute 67 sec latency 350.701 ms 1 2014 0.08 MB/sec execute 68 sec latency 83.768 ms 1 2082 0.08 MB/sec execute 69 sec latency 106.488 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Oct 2 16:50:12 EDT 2026 with return code 0 1 2141 0.08 MB/sec execute 70 sec latency 114.965 ms 47747 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47747 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished