== replay-dual test 26: dbench and tar with mds failover ========================================================== 01:03:02 (1782363782) Starting client oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre Started clients oleg408-client.virtnet: 192.168.204.108@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65471 Started dbench loop with 65472 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jun 25 01:03:04 EDT 2026 waiting for dbench pid 65513 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2100 1269204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1964 1269340 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26140 3579544 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27684 7185020 1% /mnt/lustre 1 128 5.76 MB/sec warmup 1 sec latency 44.133 ms 1 231 4.44 MB/sec warmup 2 sec latency 150.076 ms 1 363 4.33 MB/sec warmup 3 sec latency 49.501 ms 1 477 4.17 MB/sec warmup 4 sec latency 50.943 ms 1 648 4.21 MB/sec warmup 5 sec latency 51.400 ms test_26 fail mds1 1 times Failing mds1 on oleg408-server 1 759 3.56 MB/sec warmup 6 sec latency 40.219 ms Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 1 863 3.08 MB/sec warmup 7 sec latency 65.184 ms 01:03:11 (1782363791) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server 1 902 2.71 MB/sec warmup 8 sec latency 511.735 ms 1 995 2.43 MB/sec warmup 9 sec latency 544.185 ms 1 995 2.19 MB/sec warmup 10 sec latency 1544.394 ms 1 995 1.99 MB/sec warmup 11 sec latency 2544.552 ms 1 995 1.83 MB/sec warmup 12 sec latency 3544.679 ms 1 995 1.68 MB/sec warmup 13 sec latency 4544.846 ms 1 995 1.56 MB/sec warmup 14 sec latency 5545.006 ms 1 995 1.46 MB/sec warmup 15 sec latency 6545.147 ms 1 995 1.37 MB/sec warmup 16 sec latency 7545.300 ms 1 995 1.29 MB/sec warmup 17 sec latency 8545.451 ms Failover mds1 to oleg408-server mount facets: mds1 1 995 1.22 MB/sec warmup 18 sec latency 9545.602 ms 1 995 1.15 MB/sec warmup 19 sec latency 10546.989 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.00 MB/sec execute 1 sec latency 12547.271 ms 1 995 0.00 MB/sec execute 2 sec latency 13547.438 ms oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 1 995 0.00 MB/sec execute 3 sec latency 14547.558 ms pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 01:03:27 (1782363807) targets are mounted 01:03:27 (1782363807) facet_failover done 1 995 0.00 MB/sec execute 4 sec latency 15547.826 ms 1 995 0.00 MB/sec execute 5 sec latency 16547.953 ms 1 995 0.00 MB/sec execute 6 sec latency 17548.092 ms 1 995 0.00 MB/sec execute 7 sec latency 18548.223 ms 1 995 0.00 MB/sec execute 8 sec latency 19548.381 ms 1 995 0.00 MB/sec execute 9 sec latency 20548.604 ms 1 995 0.00 MB/sec execute 10 sec latency 21548.752 ms 1 995 0.00 MB/sec execute 11 sec latency 22548.897 ms 1 995 0.00 MB/sec execute 12 sec latency 23549.032 ms 1 995 0.00 MB/sec execute 13 sec latency 24549.173 ms 1 995 0.00 MB/sec execute 14 sec latency 25549.338 ms 1 995 0.00 MB/sec execute 15 sec latency 26549.587 ms 1 995 0.00 MB/sec execute 16 sec latency 27549.764 ms 1 995 0.00 MB/sec execute 17 sec latency 28549.949 ms 1 995 0.00 MB/sec execute 18 sec latency 29550.125 ms 1 995 0.00 MB/sec execute 19 sec latency 30550.272 ms 1 995 0.00 MB/sec execute 20 sec latency 31550.429 ms 1 995 0.00 MB/sec execute 21 sec latency 32550.575 ms 1 995 0.00 MB/sec execute 22 sec latency 33550.810 ms 1 995 0.00 MB/sec execute 23 sec latency 34550.980 ms 1 995 0.00 MB/sec execute 24 sec latency 35551.148 ms 1 995 0.00 MB/sec execute 25 sec latency 36551.307 ms 1 995 0.00 MB/sec execute 26 sec latency 37551.457 ms 1 995 0.00 MB/sec execute 27 sec latency 38551.622 ms 1 995 0.00 MB/sec execute 28 sec latency 39551.851 ms 1 995 0.00 MB/sec execute 29 sec latency 40552.043 ms 1 995 0.00 MB/sec execute 30 sec latency 41552.200 ms 1 995 0.00 MB/sec execute 31 sec latency 42552.405 ms 1 995 0.00 MB/sec execute 32 sec latency 43552.618 ms 1 995 0.00 MB/sec execute 33 sec latency 44552.840 ms 1 995 0.00 MB/sec execute 34 sec latency 45553.038 ms 1 995 0.00 MB/sec execute 35 sec latency 46553.228 ms 1 995 0.00 MB/sec execute 36 sec latency 47553.478 ms 1 995 0.00 MB/sec execute 37 sec latency 48553.697 ms 1 995 0.00 MB/sec execute 38 sec latency 49553.952 ms 1 995 0.00 MB/sec execute 39 sec latency 50554.218 ms 1 995 0.00 MB/sec execute 40 sec latency 51554.447 ms 1 995 0.00 MB/sec execute 41 sec latency 52554.683 ms 1 995 0.00 MB/sec execute 42 sec latency 53554.918 ms 1 995 0.00 MB/sec execute 43 sec latency 54555.148 ms 1 995 0.00 MB/sec execute 44 sec latency 55555.377 ms 1 995 0.00 MB/sec execute 45 sec latency 56555.689 ms 1 995 0.00 MB/sec execute 46 sec latency 57556.049 ms 1 995 0.00 MB/sec execute 47 sec latency 58556.257 ms 1 995 0.00 MB/sec execute 48 sec latency 59556.481 ms 1 995 0.00 MB/sec execute 49 sec latency 60556.880 ms 1 995 0.00 MB/sec execute 50 sec latency 61557.199 ms 1 995 0.00 MB/sec execute 51 sec latency 62557.546 ms 1 995 0.00 MB/sec execute 52 sec latency 63557.829 ms 1 995 0.00 MB/sec execute 53 sec latency 64558.011 ms 1 995 0.00 MB/sec execute 54 sec latency 65558.216 ms 1 995 0.00 MB/sec execute 55 sec latency 66558.444 ms 1 995 0.00 MB/sec execute 56 sec latency 67558.721 ms 1 995 0.00 MB/sec execute 57 sec latency 68559.012 ms 1 995 0.00 MB/sec execute 58 sec latency 69559.245 ms 1 995 0.00 MB/sec execute 59 sec latency 70559.468 ms 1 995 0.00 MB/sec execute 60 sec latency 71559.682 ms 1 995 0.00 MB/sec execute 61 sec latency 72559.850 ms 1 995 0.00 MB/sec execute 62 sec latency 73560.118 ms 1 995 0.00 MB/sec execute 63 sec latency 74560.352 ms 1 995 0.00 MB/sec execute 64 sec latency 75560.659 ms 1 995 0.00 MB/sec execute 65 sec latency 76560.966 ms 1 995 0.00 MB/sec execute 66 sec latency 77561.224 ms 1 995 0.00 MB/sec execute 67 sec latency 78561.490 ms 1 995 0.00 MB/sec execute 68 sec latency 79561.853 ms 1 995 0.00 MB/sec execute 69 sec latency 80562.036 ms 1 995 0.00 MB/sec execute 70 sec latency 81562.249 ms 1 995 0.00 MB/sec execute 71 sec latency 82562.507 ms 1 995 0.00 MB/sec execute 72 sec latency 83562.742 ms 1 995 0.00 MB/sec execute 73 sec latency 84563.031 ms 1 995 0.00 MB/sec execute 74 sec latency 85563.298 ms 1 995 0.00 MB/sec execute 75 sec latency 86563.528 ms 1 995 0.00 MB/sec execute 76 sec latency 87563.823 ms 1 995 0.00 MB/sec execute 77 sec latency 88564.063 ms 1 995 0.00 MB/sec execute 78 sec latency 89564.330 ms 1 995 0.00 MB/sec execute 79 sec latency 90564.562 ms 1 995 0.00 MB/sec execute 80 sec latency 91564.822 ms 1 995 0.00 MB/sec execute 81 sec latency 92565.034 ms 1 995 0.00 MB/sec execute 82 sec latency 93565.307 ms 1 995 0.00 MB/sec execute 83 sec latency 94565.579 ms 1 995 0.00 MB/sec execute 84 sec latency 95565.900 ms 1 995 0.00 MB/sec execute 85 sec latency 96566.168 ms 1 995 0.00 MB/sec execute 86 sec latency 97566.395 ms 1 995 0.00 MB/sec execute 87 sec latency 98566.623 ms 1 995 0.00 MB/sec execute 88 sec latency 99566.890 ms 1 995 0.00 MB/sec execute 89 sec latency 100567.122 ms 1 995 0.00 MB/sec execute 90 sec latency 101567.330 ms 1 995 0.00 MB/sec execute 91 sec latency 102567.567 ms 1 995 0.00 MB/sec execute 92 sec latency 103567.836 ms 1 995 0.00 MB/sec execute 93 sec latency 104568.191 ms 1 995 0.00 MB/sec execute 94 sec latency 105568.405 ms 1 995 0.00 MB/sec execute 95 sec latency 106568.704 ms 1 995 0.00 MB/sec execute 96 sec latency 107568.927 ms 1 995 0.00 MB/sec execute 97 sec latency 108569.191 ms 1 995 0.00 MB/sec execute 98 sec latency 109569.427 ms 1 995 0.00 MB/sec execute 99 sec latency 110569.860 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec 1 cleanup 116 sec 1 cleanup 117 sec 1 cleanup 118 sec 1 cleanup 119 sec 1 cleanup 120 sec 1 cleanup 121 sec 1 cleanup 122 sec 1 cleanup 123 sec 1 cleanup 124 sec 1 cleanup 125 sec 1 cleanup 126 sec 1 cleanup 127 sec 1 cleanup 128 sec 1 cleanup 129 sec 1 cleanup 130 sec 1 cleanup 131 sec 1 cleanup 132 sec 1 cleanup 133 sec 1 cleanup 134 sec 1 cleanup 135 sec 1 cleanup 136 sec 1 cleanup 137 sec 1 cleanup 138 sec 1 cleanup 139 sec 1 cleanup 140 sec 1 cleanup 141 sec 1 cleanup 142 sec 1 cleanup 143 sec 1 cleanup 144 sec 1 cleanup 145 sec 1 cleanup 146 sec 1 cleanup 147 sec 1 cleanup 148 sec 1 cleanup 149 sec 1 cleanup 150 sec 1 cleanup 151 sec 1 cleanup 152 sec 1 cleanup 153 sec 1 cleanup 154 sec 1 cleanup 155 sec 1 cleanup 156 sec 1 cleanup 157 sec 1 cleanup 158 sec 1 cleanup 159 sec 1 cleanup 160 sec 1 cleanup 161 sec 1 cleanup 162 sec 1 cleanup 163 sec 1 cleanup 164 sec 1 cleanup 165 sec 1 cleanup 166 sec 1 cleanup 167 sec 1 cleanup 168 sec 1 cleanup 169 sec 1 cleanup 170 sec 1 cleanup 171 sec 1 cleanup 172 sec 1 cleanup 173 sec 1 cleanup 174 sec 1 cleanup 175 sec 1 cleanup 176 sec 1 cleanup 177 sec 1 cleanup 178 sec 1 cleanup 179 sec 1 cleanup 180 sec 1 cleanup 181 sec 1 cleanup 182 sec 1 cleanup 183 sec tar: etc/udev/rules.d: Cannot mkdir: Input/output error 1 cleanup 184 sec 1 cleanup 185 sec 1 cleanup 186 sec 1 cleanup 187 sec 1 cleanup 188 sec 1 cleanup 189 sec 1 cleanup 190 sec 1 cleanup 191 sec 1 cleanup 192 sec 1 cleanup 193 sec 1 cleanup 194 sec 1 cleanup 195 sec 1 cleanup 196 sec 1 cleanup 197 sec 1 cleanup 198 sec