== replay-dual test 26: dbench and tar with mds failover ========================================================== 15:47:13 (1761248833) Starting client oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre Started clients oleg219-client.virtnet: 192.168.202.119@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65083 Started dbench loop with 65084 striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Oct 23 15:47:17 EDT 2025 waiting for dbench pid 65133 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1988 1269316 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1564 3605456 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26120 3579736 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27684 7185192 1% /mnt/lustre 1 22 0.52 MB/sec warmup 1 sec latency 19.166 ms 1 106 2.49 MB/sec warmup 2 sec latency 90.843 ms 1 140 2.01 MB/sec warmup 3 sec latency 165.521 ms 1 173 1.76 MB/sec warmup 4 sec latency 213.063 ms 1 207 1.62 MB/sec warmup 5 sec latency 355.157 ms 1 247 1.57 MB/sec warmup 6 sec latency 144.055 ms 1 289 1.53 MB/sec warmup 7 sec latency 121.912 ms 1 323 1.47 MB/sec warmup 8 sec latency 153.967 ms test_26 fail mds1 1 times 1 377 1.48 MB/sec warmup 9 sec latency 100.853 ms 1 414 1.46 MB/sec warmup 10 sec latency 120.614 ms Failing mds1 on oleg219-server 1 454 1.44 MB/sec warmup 11 sec latency 151.271 ms Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 1 501 1.45 MB/sec warmup 12 sec latency 110.350 ms 1 540 1.46 MB/sec warmup 13 sec latency 195.267 ms 1 606 1.50 MB/sec warmup 14 sec latency 367.104 ms 15:47:32 (1761248852) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server 1 606 1.40 MB/sec warmup 15 sec latency 1367.402 ms 1 606 1.31 MB/sec warmup 16 sec latency 2371.266 ms 1 606 1.23 MB/sec warmup 17 sec latency 3372.269 ms 1 606 1.16 MB/sec warmup 18 sec latency 4372.417 ms 1 606 1.10 MB/sec warmup 19 sec latency 5372.974 ms 1 606 0.00 MB/sec execute 1 sec latency 7373.392 ms 1 606 0.00 MB/sec execute 2 sec latency 8376.153 ms 1 606 0.00 MB/sec execute 3 sec latency 9376.409 ms 1 606 0.00 MB/sec execute 4 sec latency 10376.779 ms Failover mds1 to oleg219-server 1 606 0.00 MB/sec execute 5 sec latency 11384.370 ms mount facets: mds1 1 606 0.00 MB/sec execute 6 sec latency 12384.563 ms 1 606 0.00 MB/sec execute 7 sec latency 13386.145 ms 1 606 0.00 MB/sec execute 8 sec latency 14387.440 ms 1 606 0.00 MB/sec execute 9 sec latency 15387.593 ms 1 606 0.00 MB/sec execute 10 sec latency 16388.456 ms 1 606 0.00 MB/sec execute 11 sec latency 17389.303 ms 1 606 0.00 MB/sec execute 12 sec latency 18392.788 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 606 0.00 MB/sec execute 13 sec latency 19393.282 ms 1 606 0.00 MB/sec execute 14 sec latency 20393.952 ms 1 606 0.00 MB/sec execute 15 sec latency 21394.237 ms 1 606 0.00 MB/sec execute 16 sec latency 22394.498 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 1 606 0.00 MB/sec execute 17 sec latency 23394.800 ms pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 15:47:55 (1761248875) targets are mounted 15:47:55 (1761248875) facet_failover done 1 606 0.00 MB/sec execute 18 sec latency 24395.514 ms 1 606 0.00 MB/sec execute 19 sec latency 25397.586 ms 1 606 0.00 MB/sec execute 20 sec latency 26406.875 ms 1 608 0.00 MB/sec execute 21 sec latency 27339.994 ms 1 661 0.01 MB/sec execute 22 sec latency 112.958 ms 1 695 0.01 MB/sec execute 23 sec latency 167.553 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 723 0.01 MB/sec execute 24 sec latency 164.402 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 762 0.02 MB/sec execute 25 sec latency 122.400 ms 1 798 0.02 MB/sec execute 26 sec latency 105.120 ms 1 849 0.02 MB/sec execute 27 sec latency 149.201 ms 1 884 0.02 MB/sec execute 28 sec latency 168.151 ms 1 891 0.02 MB/sec execute 29 sec latency 927.979 ms 1 934 0.03 MB/sec execute 30 sec latency 88.752 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2128 1269176 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1872 1269432 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13872 3581028 1% /mnt/lustre[OST:0] 1 983 0.03 MB/sec execute 31 sec latency 73.734 ms lustre-OST0001_UUID 3833116 36880 3569944 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50752 7150972 1% /mnt/lustre 1 1059 0.06 MB/sec execute 32 sec latency 91.151 ms 1 1119 0.06 MB/sec execute 33 sec latency 68.310 ms 1 1176 0.06 MB/sec execute 34 sec latency 100.803 ms 1 1225 0.06 MB/sec execute 35 sec latency 61.282 ms 1 1269 0.06 MB/sec execute 36 sec latency 94.013 ms 1 1406 0.14 MB/sec execute 37 sec latency 73.355 ms 1 1467 0.13 MB/sec execute 38 sec latency 79.771 ms test_26 fail mds2 2 times 1 1508 0.13 MB/sec execute 39 sec latency 228.706 ms 1 1564 0.13 MB/sec execute 40 sec latency 86.030 ms Failing mds2 on oleg219-server Stopping /mnt/lustre-mds2 (opts:) on oleg219-server 1 1609 0.13 MB/sec execute 41 sec latency 82.046 ms 1 1660 0.12 MB/sec execute 42 sec latency 158.625 ms 1 1696 0.12 MB/sec execute 43 sec latency 147.939 ms 15:48:21 (1761248901) shut down facet: mds2 facet_host: oleg219-server facet_failover_host: oleg219-server 1 1771 0.12 MB/sec execute 44 sec latency 87.287 ms 1 1838 0.12 MB/sec execute 45 sec latency 98.175 ms 1 1913 0.12 MB/sec execute 46 sec latency 75.331 ms 1 2000 0.12 MB/sec execute 47 sec latency 73.040 ms 1 2056 0.12 MB/sec execute 48 sec latency 73.135 ms 1 2131 0.12 MB/sec execute 49 sec latency 84.446 ms 1 2185 0.12 MB/sec execute 50 sec latency 97.208 ms 1 2259 0.11 MB/sec execute 51 sec latency 300.917 ms 1 2259 0.11 MB/sec execute 52 sec latency 1301.085 ms 1 2259 0.11 MB/sec execute 53 sec latency 2303.584 ms 1 2259 0.11 MB/sec execute 54 sec latency 3303.767 ms Failover mds2 to oleg219-server mount facets: mds2 1 2259 0.11 MB/sec execute 55 sec latency 4304.386 ms 1 2259 0.10 MB/sec execute 56 sec latency 5306.095 ms 1 2259 0.10 MB/sec execute 57 sec latency 6306.653 ms 1 2259 0.10 MB/sec execute 58 sec latency 7309.763 ms 1 2259 0.10 MB/sec execute 59 sec latency 8310.170 ms 1 2259 0.10 MB/sec execute 60 sec latency 9312.693 ms 1 2259 0.10 MB/sec execute 61 sec latency 10312.983 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.09 MB/sec execute 62 sec latency 11313.268 ms 1 2259 0.09 MB/sec execute 63 sec latency 12313.532 ms 1 2259 0.09 MB/sec execute 64 sec latency 13314.823 ms 1 2259 0.09 MB/sec execute 65 sec latency 14315.640 ms 1 2259 0.09 MB/sec execute 66 sec latency 15316.402 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 1 2259 0.09 MB/sec execute 67 sec latency 16316.603 ms pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 1 2259 0.09 MB/sec execute 68 sec latency 17317.716 ms Started lustre-MDT0001 15:48:46 (1761248926) targets are mounted 15:48:46 (1761248926) facet_failover done 1 2259 0.08 MB/sec execute 69 sec latency 18318.008 ms 1 2259 0.08 MB/sec execute 70 sec latency 19324.728 ms 1 2334 0.10 MB/sec execute 71 sec latency 19382.907 ms 1 2377 0.10 MB/sec execute 72 sec latency 153.714 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 2417 0.10 MB/sec execute 73 sec latency 181.713 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2466 0.10 MB/sec execute 74 sec latency 94.499 ms 1 2559 0.11 MB/sec execute 75 sec latency 76.606 ms 1 2608 0.11 MB/sec execute 76 sec latency 77.684 ms 1 2770 0.14 MB/sec execute 77 sec latency 90.415 ms 1 2917 0.16 MB/sec execute 78 sec latency 66.769 ms 1 2993 0.16 MB/sec execute 79 sec latency 84.741 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1268920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2136 1269168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 32168 3573980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37064 3567768 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 69232 7141748 1% /mnt/lustre 1 3175 0.16 MB/sec execute 80 sec latency 107.146 ms 1 3245 0.17 MB/sec execute 81 sec latency 88.241 ms 1 3339 0.18 MB/sec execute 82 sec latency 88.834 ms 1 3399 0.17 MB/sec execute 83 sec latency 65.397 ms 1 3543 0.21 MB/sec execute 84 sec latency 81.523 ms 1 3626 0.21 MB/sec execute 85 sec latency 68.305 ms 1 3685 0.21 MB/sec execute 86 sec latency 68.208 ms 1 3754 0.21 MB/sec execute 87 sec latency 74.889 ms test_26 fail mds1 3 times 1 3800 0.21 MB/sec execute 88 sec latency 114.782 ms 1 3840 0.21 MB/sec execute 89 sec latency 106.464 ms Failing mds1 on oleg219-server 1 3885 0.21 MB/sec execute 90 sec latency 70.967 ms Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 1 3939 0.21 MB/sec execute 91 sec latency 115.392 ms 1 3991 0.21 MB/sec execute 92 sec latency 123.841 ms 15:49:11 (1761248951) shut down 1 4036 0.20 MB/sec execute 93 sec latency 82.066 ms facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server 1 4083 0.20 MB/sec execute 94 sec latency 213.305 ms 1 4083 0.20 MB/sec execute 95 sec latency 1214.837 ms 1 4083 0.20 MB/sec execute 96 sec latency 2217.648 ms 1 4083 0.20 MB/sec execute 97 sec latency 3217.908 ms 1 4083 0.19 MB/sec execute 98 sec latency 4218.172 ms 1 4083 0.19 MB/sec execute 99 sec latency 5218.419 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec Failover mds1 to oleg219-server mount facets: mds1 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec 1 cleanup 116 sec 1 cleanup 117 sec 1 cleanup 118 sec 1 cleanup 119 sec 1 cleanup 120 sec 1 cleanup 121 sec 1 cleanup 122 sec 1 cleanup 123 sec 1 cleanup 124 sec oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 1 cleanup 125 sec pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 1 cleanup 126 sec Started lustre-MDT0000 15:49:44 (1761248984) targets are mounted 15:49:44 (1761248984) facet_failover done 1 cleanup 127 sec 1 cleanup 128 sec 1 cleanup 129 sec 1 cleanup 130 sec 1 cleanup 131 sec oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 cleanup 132 sec 0 cleanup 133 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 613 73.174 19382.892 Close 448 10.269 43.906 Rename 26 72.243 147.902 Unlink 96 21.589 42.539 Qpathinfo 534 130.159 35217.221 Qfileinfo 97 1.248 14.825 Qfsinfo 98 3.189 79.007 Sfileinfo 50 54.615 181.699 Find 204 9.419 86.665 WriteX 303 9.864 81.160 ReadX 962 1.184 74.791 LockX 2 7.603 8.572 UnlockX 2 11.800 16.920 Flush 43 51.103 167.528 Throughput 0.192601 MB/sec 1 clients 1 procs max_latency=27339.994 ms stopping dbench on /mnt/lustre at Thu Oct 23 15:49:51 EDT 2025 with return code 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2640 1268664 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2336 1268968 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21036 3565104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 3052 3603700 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 24088 7168804 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Oct 23 15:49:59 EDT 2025 waiting for dbench pid 68325 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 28 0.92 MB/sec warmup 1 sec latency 25.378 ms 1 75 1.60 MB/sec warmup 2 sec latency 110.875 ms 1 121 1.83 MB/sec warmup 3 sec latency 137.154 ms 1 155 1.63 MB/sec warmup 4 sec latency 175.159 ms 1 155 1.31 MB/sec warmup 5 sec latency 1039.585 ms 1 165 1.13 MB/sec warmup 6 sec latency 1647.725 ms 1 186 1.06 MB/sec warmup 7 sec latency 485.325 ms 1 231 1.11 MB/sec warmup 8 sec latency 120.050 ms test_26 fail mds2 4 times 1 279 1.16 MB/sec warmup 9 sec latency 101.006 ms 1 313 1.15 MB/sec warmup 10 sec latency 162.832 ms Failing mds2 on oleg219-server 1 348 1.14 MB/sec warmup 11 sec latency 139.081 ms Stopping /mnt/lustre-mds2 (opts:) on oleg219-server 1 398 1.17 MB/sec warmup 12 sec latency 100.351 ms 1 450 1.21 MB/sec warmup 13 sec latency 153.446 ms 1 501 1.25 MB/sec warmup 14 sec latency 287.908 ms 1 501 1.16 MB/sec warmup 15 sec latency 1288.433 ms 1 501 1.09 MB/sec warmup 16 sec latency 2290.188 ms 1 501 1.03 MB/sec warmup 17 sec latency 3292.947 ms 1 501 0.97 MB/sec warmup 18 sec latency 4293.241 ms 1 501 0.92 MB/sec warmup 19 sec latency 5293.654 ms 15:50:20 (1761249020) shut down 1 501 0.00 MB/sec execute 1 sec latency 7300.202 ms facet: mds2 facet_host: oleg219-server facet_failover_host: oleg219-server 1 501 0.00 MB/sec execute 2 sec latency 8300.432 ms 1 501 0.00 MB/sec execute 3 sec latency 9302.029 ms 1 501 0.00 MB/sec execute 4 sec latency 10302.462 ms 1 501 0.00 MB/sec execute 5 sec latency 11303.218 ms 1 501 0.00 MB/sec execute 6 sec latency 12303.434 ms 1 501 0.00 MB/sec execute 7 sec latency 13303.718 ms 1 501 0.00 MB/sec execute 8 sec latency 14303.915 ms 1 501 0.00 MB/sec execute 9 sec latency 15305.512 ms 1 501 0.00 MB/sec execute 10 sec latency 16305.702 ms 1 501 0.00 MB/sec execute 11 sec latency 17305.931 ms Failover mds2 to oleg219-server mount facets: mds2 1 501 0.00 MB/sec execute 12 sec latency 18308.533 ms 1 501 0.00 MB/sec execute 13 sec latency 19308.896 ms 1 501 0.00 MB/sec execute 14 sec latency 20312.246 ms 1 501 0.00 MB/sec execute 15 sec latency 21312.595 ms 1 501 0.00 MB/sec execute 16 sec latency 22313.523 ms 1 501 0.00 MB/sec execute 17 sec latency 23314.624 ms 1 501 0.00 MB/sec execute 18 sec latency 24318.999 ms 1 501 0.00 MB/sec execute 19 sec latency 25319.209 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 501 0.00 MB/sec execute 20 sec latency 26322.955 ms 1 501 0.00 MB/sec execute 21 sec latency 27323.164 ms 1 501 0.00 MB/sec execute 22 sec latency 28323.331 ms 1 501 0.00 MB/sec execute 23 sec latency 29324.048 ms 1 501 0.00 MB/sec execute 24 sec latency 30324.314 ms oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 1 501 0.00 MB/sec execute 25 sec latency 31325.200 ms pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 1 501 0.00 MB/sec execute 26 sec latency 32325.434 ms Started lustre-MDT0001 15:50:46 (1761249046) targets are mounted 15:50:46 (1761249046) facet_failover done 1 501 0.00 MB/sec execute 27 sec latency 33326.212 ms 1 501 0.00 MB/sec execute 28 sec latency 34329.963 ms 1 516 0.01 MB/sec execute 29 sec latency 34680.134 ms 1 612 0.12 MB/sec execute 30 sec latency 99.513 ms oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 664 0.12 MB/sec execute 31 sec latency 119.684 ms 1 699 0.12 MB/sec execute 32 sec latency 131.800 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 730 0.12 MB/sec execute 33 sec latency 131.612 ms 1 769 0.11 MB/sec execute 34 sec latency 86.703 ms 1 801 0.11 MB/sec execute 35 sec latency 163.117 ms 1 863 0.11 MB/sec execute 36 sec latency 84.468 ms 1 884 0.11 MB/sec execute 37 sec latency 183.166 ms 1 887 0.11 MB/sec execute 38 sec latency 952.098 ms 1 918 0.11 MB/sec execute 39 sec latency 101.817 ms 1 970 0.11 MB/sec execute 40 sec latency 100.199 ms 1 1005 0.11 MB/sec execute 41 sec latency 97.187 ms 1 1074 0.13 MB/sec execute 42 sec latency 60.201 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1128 0.13 MB/sec execute 43 sec latency 89.306 ms 1 1164 0.13 MB/sec execute 44 sec latency 130.482 ms 1 1201 0.12 MB/sec execute 45 sec latency 100.577 ms 1 1251 0.12 MB/sec execute 46 sec latency 88.241 ms 1 1380 0.18 MB/sec execute 47 sec latency 81.941 ms 1 1410 0.18 MB/sec execute 48 sec latency 204.333 ms 1 1465 0.17 MB/sec execute 49 sec latency 136.508 ms 1 1497 0.17 MB/sec execute 50 sec latency 321.277 ms 1 1545 0.17 MB/sec execute 51 sec latency 123.796 ms 1 1603 0.17 MB/sec execute 52 sec latency 84.387 ms 1 1654 0.16 MB/sec execute 53 sec latency 108.305 ms 1 1691 0.16 MB/sec execute 54 sec latency 119.400 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Oct 23 15:51:13 EDT 2025 with return code 0 68325 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 68325 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished