== replay-dual test 26: dbench and tar with mds failover ========================================================== 01:12:57 (1788844377) Starting client oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre Started clients oleg315-client.virtnet: 192.168.203.115@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45237 Started dbench loop with 45238 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 20480 7498752 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Sep 8 01:13:03 EDT 2026 waiting for dbench pid 45307 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 32 1.05 MB/sec warmup 1 sec latency 165.875 ms 1 95 2.13 MB/sec warmup 2 sec latency 124.254 ms 1 129 1.92 MB/sec warmup 3 sec latency 198.782 ms 1 157 1.63 MB/sec warmup 4 sec latency 166.191 ms 1 184 1.47 MB/sec warmup 5 sec latency 200.451 ms test_26 fail mds1 1 times 1 216 1.39 MB/sec warmup 6 sec latency 230.895 ms Failing mds1 on oleg315-server 1 249 1.34 MB/sec warmup 7 sec latency 163.462 ms Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 1 287 1.33 MB/sec warmup 8 sec latency 115.064 ms 1 312 1.27 MB/sec warmup 9 sec latency 153.609 ms 1 312 1.15 MB/sec warmup 10 sec latency 1142.529 ms 1 312 1.04 MB/sec warmup 11 sec latency 2142.942 ms 01:13:14 (1788844394) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server 1 312 0.96 MB/sec warmup 12 sec latency 3144.737 ms 1 312 0.88 MB/sec warmup 13 sec latency 4145.030 ms 1 312 0.82 MB/sec warmup 14 sec latency 5145.439 ms 1 312 0.76 MB/sec warmup 15 sec latency 6145.743 ms 1 312 0.72 MB/sec warmup 16 sec latency 7146.073 ms 1 312 0.67 MB/sec warmup 17 sec latency 8152.852 ms 1 312 0.64 MB/sec warmup 18 sec latency 9153.837 ms 1 312 0.60 MB/sec warmup 19 sec latency 10154.095 ms 1 312 0.00 MB/sec execute 1 sec latency 12158.729 ms Failover mds1 to oleg315-server 1 312 0.00 MB/sec execute 2 sec latency 13159.868 ms mount facets: mds1 1 312 0.00 MB/sec execute 3 sec latency 14161.876 ms 1 312 0.00 MB/sec execute 4 sec latency 15162.054 ms 1 312 0.00 MB/sec execute 5 sec latency 16162.291 ms 1 312 0.00 MB/sec execute 6 sec latency 17162.561 ms 1 312 0.00 MB/sec execute 7 sec latency 18163.222 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 312 0.00 MB/sec execute 8 sec latency 19163.482 ms 1 312 0.00 MB/sec execute 9 sec latency 20163.675 ms 1 312 0.00 MB/sec execute 10 sec latency 21164.250 ms 1 312 0.00 MB/sec execute 11 sec latency 22164.912 ms 1 312 0.00 MB/sec execute 12 sec latency 23165.193 ms 1 312 0.00 MB/sec execute 13 sec latency 24165.380 ms oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all 1 312 0.00 MB/sec execute 14 sec latency 25168.914 ms pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 1 312 0.00 MB/sec execute 15 sec latency 26173.938 ms Started lustre-MDT0000 01:13:38 (1788844418) targets are mounted 01:13:38 (1788844418) facet_failover done 1 312 0.00 MB/sec execute 16 sec latency 27174.233 ms 1 312 0.00 MB/sec execute 17 sec latency 28174.596 ms 1 312 0.00 MB/sec execute 18 sec latency 29174.884 ms 1 312 0.00 MB/sec execute 19 sec latency 30175.167 ms 1 312 0.00 MB/sec execute 20 sec latency 31177.653 ms 1 312 0.00 MB/sec execute 21 sec latency 32177.950 ms 1 312 0.00 MB/sec execute 22 sec latency 33178.438 ms 1 312 0.00 MB/sec execute 23 sec latency 34179.356 ms 1 312 0.00 MB/sec execute 24 sec latency 35179.578 ms 1 312 0.00 MB/sec execute 25 sec latency 36179.871 ms 1 312 0.00 MB/sec execute 26 sec latency 37180.090 ms 1 312 0.00 MB/sec execute 27 sec latency 38180.442 ms 1 312 0.00 MB/sec execute 28 sec latency 39185.447 ms 1 312 0.00 MB/sec execute 29 sec latency 40189.232 ms 1 316 0.00 MB/sec execute 30 sec latency 40662.675 ms 1 344 0.03 MB/sec execute 31 sec latency 161.400 ms 1 373 0.06 MB/sec execute 32 sec latency 171.032 ms 1 409 0.09 MB/sec execute 33 sec latency 148.372 ms 1 445 0.12 MB/sec execute 34 sec latency 148.656 ms 1 469 0.14 MB/sec execute 35 sec latency 132.574 ms oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 501 0.17 MB/sec execute 36 sec latency 158.695 ms 1 522 0.18 MB/sec execute 37 sec latency 166.376 ms 1 586 0.23 MB/sec execute 38 sec latency 195.346 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 618 0.24 MB/sec execute 39 sec latency 162.055 ms 1 664 0.24 MB/sec execute 40 sec latency 109.854 ms 1 699 0.24 MB/sec execute 41 sec latency 177.699 ms 1 731 0.23 MB/sec execute 42 sec latency 142.454 ms 1 778 0.23 MB/sec execute 43 sec latency 156.424 ms 1 796 0.23 MB/sec execute 44 sec latency 349.394 ms 1 825 0.22 MB/sec execute 45 sec latency 99.293 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3717120 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 12288 3691520 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 49152 7408640 1% /mnt/lustre 1 865 0.22 MB/sec execute 46 sec latency 155.728 ms 1 878 0.22 MB/sec execute 47 sec latency 411.838 ms 1 902 0.21 MB/sec execute 48 sec latency 173.709 ms 1 916 0.21 MB/sec execute 49 sec latency 291.204 ms 1 938 0.21 MB/sec execute 50 sec latency 192.515 ms 1 965 0.20 MB/sec execute 51 sec latency 149.592 ms 1 983 0.20 MB/sec execute 52 sec latency 267.905 ms test_26 fail mds1 2 times 1 1040 0.21 MB/sec execute 53 sec latency 92.774 ms 1 1077 0.21 MB/sec execute 54 sec latency 125.286 ms Failing mds1 on oleg315-server 1 1124 0.21 MB/sec execute 55 sec latency 99.437 ms 1 1147 0.21 MB/sec execute 56 sec latency 170.467 ms Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 1 1182 0.20 MB/sec execute 57 sec latency 175.189 ms 1 1192 0.20 MB/sec execute 58 sec latency 768.573 ms 1 1192 0.20 MB/sec execute 59 sec latency 1771.174 ms 1 1192 0.19 MB/sec execute 60 sec latency 2771.473 ms 01:14:23 (1788844463) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server 1 1192 0.19 MB/sec execute 61 sec latency 3772.545 ms 1 1192 0.19 MB/sec execute 62 sec latency 4772.875 ms 1 1192 0.18 MB/sec execute 63 sec latency 5773.387 ms 1 1192 0.18 MB/sec execute 64 sec latency 6773.674 ms 1 1192 0.18 MB/sec execute 65 sec latency 7773.999 ms 1 1192 0.17 MB/sec execute 66 sec latency 8774.571 ms 1 1192 0.17 MB/sec execute 67 sec latency 9775.132 ms 1 1192 0.17 MB/sec execute 68 sec latency 10775.411 ms 1 1192 0.17 MB/sec execute 69 sec latency 11775.924 ms 1 1192 0.16 MB/sec execute 70 sec latency 12777.825 ms Failover mds1 to oleg315-server mount facets: mds1 1 1192 0.16 MB/sec execute 71 sec latency 13777.998 ms 1 1192 0.16 MB/sec execute 72 sec latency 14778.251 ms 1 1192 0.16 MB/sec execute 73 sec latency 15778.955 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1192 0.16 MB/sec execute 74 sec latency 16783.945 ms 1 1192 0.15 MB/sec execute 75 sec latency 17784.410 ms 1 1192 0.15 MB/sec execute 76 sec latency 18784.616 ms 1 1192 0.15 MB/sec execute 77 sec latency 19785.662 ms 1 1192 0.15 MB/sec execute 78 sec latency 20787.722 ms 1 1192 0.15 MB/sec execute 79 sec latency 21793.529 ms 1 1192 0.14 MB/sec execute 80 sec latency 22793.676 ms oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all 1 1214 0.14 MB/sec execute 81 sec latency 23115.296 ms pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 1 1244 0.14 MB/sec execute 82 sec latency 106.843 ms Started lustre-MDT0000 01:14:46 (1788844486) targets are mounted 01:14:46 (1788844486) facet_failover done 1 1277 0.14 MB/sec execute 83 sec latency 119.744 ms 1 1409 0.17 MB/sec execute 84 sec latency 112.015 ms 1 1451 0.17 MB/sec execute 85 sec latency 123.570 ms 1 1497 0.17 MB/sec execute 86 sec latency 156.052 ms 1 1532 0.17 MB/sec execute 87 sec latency 195.677 ms 1 1578 0.17 MB/sec execute 88 sec latency 101.860 ms 1 1619 0.16 MB/sec execute 89 sec latency 102.117 ms oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1653 0.16 MB/sec execute 90 sec latency 228.457 ms 1 1689 0.16 MB/sec execute 91 sec latency 119.553 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1748 0.16 MB/sec execute 92 sec latency 111.045 ms 1 1805 0.16 MB/sec execute 93 sec latency 130.745 ms 1 1861 0.16 MB/sec execute 94 sec latency 88.919 ms 1 1911 0.16 MB/sec execute 95 sec latency 206.506 ms 1 1986 0.16 MB/sec execute 96 sec latency 72.673 ms 1 2024 0.16 MB/sec execute 97 sec latency 100.923 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3713024 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3714048 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7427072 1% /mnt/lustre 1 2088 0.15 MB/sec execute 98 sec latency 105.721 ms 1 2148 0.15 MB/sec execute 99 sec latency 104.297 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 330 68.473 335.225 Close 280 159.280 40662.645 Rename 11 122.771 199.207 Unlink 28 30.651 69.509 Qpathinfo 266 18.257 162.027 Qfileinfo 86 2.603 17.099 Qfsinfo 33 5.294 46.490 Sfileinfo 34 753.416 23115.273 Find 88 7.426 77.916 WriteX 330 16.182 116.819 ReadX 343 2.596 124.100 Flush 30 134.155 411.816 Throughput 0.153373 MB/sec 1 clients 1 procs max_latency=40662.675 ms stopping dbench on /mnt/lustre at Tue Sep 8 01:15:06 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished test_26 fail mds1 3 times Failing mds1 on oleg315-server looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 01:15:13 (1788844513) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 01:15:38 (1788844538) targets are mounted 01:15:38 (1788844538) facet_failover done '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Sep 8 01:15:52 EDT 2026 waiting for dbench pid 47819 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 117.504 ms 1 90 2.10 MB/sec warmup 2 sec latency 150.602 ms 1 129 1.92 MB/sec warmup 3 sec latency 164.816 ms 1 165 1.70 MB/sec warmup 4 sec latency 170.798 ms oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 197 1.57 MB/sec warmup 5 sec latency 150.881 ms 1 231 1.48 MB/sec warmup 6 sec latency 225.194 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 271 1.45 MB/sec warmup 7 sec latency 139.092 ms 1 305 1.40 MB/sec warmup 8 sec latency 155.353 ms 1 331 1.33 MB/sec warmup 9 sec latency 141.179 ms 1 362 1.29 MB/sec warmup 10 sec latency 146.053 ms 1 405 1.30 MB/sec warmup 11 sec latency 147.160 ms 1 437 1.28 MB/sec warmup 12 sec latency 197.906 ms 1 469 1.26 MB/sec warmup 13 sec latency 124.849 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 37888 3694592 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 26624 3720192 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 64512 7414784 1% /mnt/lustre 1 496 1.23 MB/sec warmup 14 sec latency 163.614 ms 1 517 1.18 MB/sec warmup 15 sec latency 217.456 ms 1 608 1.31 MB/sec warmup 16 sec latency 121.820 ms 1 661 1.24 MB/sec warmup 17 sec latency 144.662 ms 1 674 1.17 MB/sec warmup 18 sec latency 321.039 ms 1 708 1.11 MB/sec warmup 19 sec latency 149.762 ms test_26 fail mds1 4 times 1 769 0.05 MB/sec execute 1 sec latency 160.155 ms 1 791 0.07 MB/sec execute 2 sec latency 231.323 ms Failing mds1 on oleg315-server 1 806 0.05 MB/sec execute 3 sec latency 196.146 ms Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 1 863 0.07 MB/sec execute 4 sec latency 113.790 ms 1 874 0.07 MB/sec execute 5 sec latency 551.168 ms 1 874 0.06 MB/sec execute 6 sec latency 1552.508 ms 1 874 0.05 MB/sec execute 7 sec latency 2552.726 ms 01:16:19 (1788844579) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server 1 874 0.04 MB/sec execute 8 sec latency 3553.189 ms 1 874 0.04 MB/sec execute 9 sec latency 4553.492 ms 1 874 0.03 MB/sec execute 10 sec latency 5553.878 ms 1 874 0.03 MB/sec execute 11 sec latency 6555.722 ms 1 874 0.03 MB/sec execute 12 sec latency 7556.691 ms 1 874 0.03 MB/sec execute 13 sec latency 8560.626 ms 1 874 0.02 MB/sec execute 14 sec latency 9561.142 ms 1 874 0.02 MB/sec execute 15 sec latency 10561.564 ms 1 874 0.02 MB/sec execute 16 sec latency 11562.966 ms 1 874 0.02 MB/sec execute 17 sec latency 12566.236 ms 1 874 0.02 MB/sec execute 18 sec latency 13566.396 ms Failover mds1 to oleg315-server mount facets: mds1 1 874 0.02 MB/sec execute 19 sec latency 14566.636 ms 1 874 0.02 MB/sec execute 20 sec latency 15566.929 ms 1 874 0.02 MB/sec execute 21 sec latency 16567.824 ms 1 874 0.02 MB/sec execute 22 sec latency 17568.269 ms 1 874 0.01 MB/sec execute 23 sec latency 18568.688 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 874 0.01 MB/sec execute 24 sec latency 19568.947 ms 1 874 0.01 MB/sec execute 25 sec latency 20569.892 ms 1 874 0.01 MB/sec execute 26 sec latency 21570.171 ms 1 874 0.01 MB/sec execute 27 sec latency 22571.010 ms 1 874 0.01 MB/sec execute 28 sec latency 23571.284 ms 1 874 0.01 MB/sec execute 29 sec latency 24571.575 ms 1 874 0.01 MB/sec execute 30 sec latency 25572.372 ms 1 874 0.01 MB/sec execute 31 sec latency 26573.971 ms 1 874 0.01 MB/sec execute 32 sec latency 27574.351 ms 1 874 0.01 MB/sec execute 33 sec latency 28574.648 ms 1 874 0.01 MB/sec execute 34 sec latency 29575.133 ms 1 874 0.01 MB/sec execute 35 sec latency 30575.373 ms 1 874 0.01 MB/sec execute 36 sec latency 31581.635 ms 1 874 0.01 MB/sec execute 37 sec latency 32581.815 ms 1 874 0.01 MB/sec execute 38 sec latency 33582.102 ms 1 874 0.01 MB/sec execute 39 sec latency 34582.312 ms oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all 1 874 0.01 MB/sec execute 40 sec latency 35582.766 ms pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 1 874 0.01 MB/sec execute 41 sec latency 36584.909 ms Started lustre-MDT0000 01:16:54 (1788844614) targets are mounted 01:16:54 (1788844614) facet_failover done 1 874 0.01 MB/sec execute 42 sec latency 37585.113 ms 1 874 0.01 MB/sec execute 43 sec latency 38585.758 ms 1 874 0.01 MB/sec execute 44 sec latency 39585.965 ms 1 874 0.01 MB/sec execute 45 sec latency 40586.086 ms 1 892 0.01 MB/sec execute 46 sec latency 40965.189 ms 1 916 0.01 MB/sec execute 47 sec latency 284.825 ms 1 955 0.01 MB/sec execute 48 sec latency 147.473 ms 1 980 0.01 MB/sec execute 49 sec latency 237.283 ms oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1038 0.03 MB/sec execute 50 sec latency 252.695 ms 1 1084 0.03 MB/sec execute 51 sec latency 76.111 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1136 0.03 MB/sec execute 52 sec latency 102.773 ms 1 1172 0.03 MB/sec execute 53 sec latency 183.990 ms 1 1215 0.03 MB/sec execute 54 sec latency 90.683 ms 1 1244 0.03 MB/sec execute 55 sec latency 124.168 ms 1 1382 0.08 MB/sec execute 56 sec latency 137.963 ms 1 1428 0.08 MB/sec execute 57 sec latency 80.563 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1474 0.08 MB/sec execute 58 sec latency 101.515 ms 1 1509 0.08 MB/sec execute 59 sec latency 225.427 ms 1 1576 0.08 MB/sec execute 60 sec latency 80.109 ms 1 1632 0.08 MB/sec execute 61 sec latency 92.780 ms 1 1660 0.08 MB/sec execute 62 sec latency 295.665 ms 1 1698 0.08 MB/sec execute 63 sec latency 153.443 ms 1 1773 0.08 MB/sec execute 64 sec latency 95.120 ms 1 1832 0.08 MB/sec execute 65 sec latency 71.928 ms 1 1884 0.08 MB/sec execute 66 sec latency 262.147 ms 1 1950 0.08 MB/sec execute 67 sec latency 290.160 ms 1 2004 0.08 MB/sec execute 68 sec latency 106.271 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Sep 8 01:17:21 EDT 2026 with return code 0 1 2083 0.08 MB/sec execute 69 sec latency 83.998 ms 47819 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47819 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished