== replay-dual test 26: dbench and tar with mds failover ========================================================== 19:15:25 (1763511325) Starting client oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre Started clients oleg449-client.virtnet: 192.168.204.149@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65042 Started dbench loop with 65043 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1940 1269364 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13852 3588972 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 15396 7194448 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Nov 18 19:15:31 EST 2025 waiting for dbench pid 65121 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 74.003 ms 1 110 2.62 MB/sec warmup 2 sec latency 109.378 ms 1 147 2.09 MB/sec warmup 3 sec latency 122.698 ms 1 184 1.85 MB/sec warmup 4 sec latency 284.165 ms 1 220 1.70 MB/sec warmup 5 sec latency 403.766 ms 1 271 1.70 MB/sec warmup 6 sec latency 101.351 ms test_26 fail mds1 1 times 1 313 1.64 MB/sec warmup 7 sec latency 137.852 ms 1 354 1.58 MB/sec warmup 8 sec latency 111.876 ms Failing mds1 on oleg449-server 1 397 1.56 MB/sec warmup 9 sec latency 121.977 ms Stopping /mnt/lustre-mds1 (opts:) on oleg449-server 1 453 1.59 MB/sec warmup 10 sec latency 83.966 ms 1 453 1.45 MB/sec warmup 11 sec latency 1069.186 ms 1 453 1.32 MB/sec warmup 12 sec latency 2069.392 ms 19:15:44 (1763511344) shut down facet: mds1 facet_host: oleg449-server facet_failover_host: oleg449-server 1 453 1.22 MB/sec warmup 13 sec latency 3069.740 ms 1 453 1.14 MB/sec warmup 14 sec latency 4070.111 ms 1 453 1.06 MB/sec warmup 15 sec latency 5070.331 ms 1 453 0.99 MB/sec warmup 16 sec latency 6070.902 ms 1 453 0.94 MB/sec warmup 17 sec latency 7072.638 ms 1 453 0.88 MB/sec warmup 18 sec latency 8072.841 ms 1 453 0.84 MB/sec warmup 19 sec latency 9073.045 ms 1 453 0.00 MB/sec execute 1 sec latency 11074.312 ms 1 453 0.00 MB/sec execute 2 sec latency 12075.575 ms Failover mds1 to oleg449-server mount facets: mds1 1 453 0.00 MB/sec execute 3 sec latency 13075.771 ms 1 453 0.00 MB/sec execute 4 sec latency 14076.890 ms 1 453 0.00 MB/sec execute 5 sec latency 15077.144 ms 1 453 0.00 MB/sec execute 6 sec latency 16077.917 ms 1 453 0.00 MB/sec execute 7 sec latency 17078.183 ms 1 453 0.00 MB/sec execute 8 sec latency 18078.446 ms 1 453 0.00 MB/sec execute 9 sec latency 19078.716 ms 1 453 0.00 MB/sec execute 10 sec latency 20078.967 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 453 0.00 MB/sec execute 11 sec latency 21082.607 ms 1 453 0.00 MB/sec execute 12 sec latency 22084.737 ms 1 453 0.00 MB/sec execute 13 sec latency 23090.217 ms 1 453 0.00 MB/sec execute 14 sec latency 24090.475 ms 1 453 0.00 MB/sec execute 15 sec latency 25090.796 ms 1 453 0.00 MB/sec execute 16 sec latency 26091.259 ms 1 453 0.00 MB/sec execute 17 sec latency 27091.579 ms 1 453 0.00 MB/sec execute 18 sec latency 28091.916 ms 1 453 0.00 MB/sec execute 19 sec latency 29095.608 ms 1 453 0.00 MB/sec execute 20 sec latency 30096.985 ms 1 453 0.00 MB/sec execute 21 sec latency 31097.320 ms oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 1 453 0.00 MB/sec execute 22 sec latency 32097.593 ms pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 1 453 0.00 MB/sec execute 23 sec latency 33097.813 ms Started lustre-MDT0000 19:16:15 (1763511375) targets are mounted 19:16:15 (1763511375) facet_failover done 1 453 0.00 MB/sec execute 24 sec latency 34098.022 ms 1 453 0.00 MB/sec execute 25 sec latency 35101.977 ms 1 485 0.04 MB/sec execute 26 sec latency 35278.407 ms 1 522 0.08 MB/sec execute 27 sec latency 132.295 ms oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 616 0.18 MB/sec execute 28 sec latency 95.061 ms 1 673 0.18 MB/sec execute 29 sec latency 98.153 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 710 0.18 MB/sec execute 30 sec latency 120.015 ms 1 762 0.18 MB/sec execute 31 sec latency 120.757 ms 1 803 0.17 MB/sec execute 32 sec latency 87.924 ms 1 849 0.17 MB/sec execute 33 sec latency 109.171 ms 1 884 0.17 MB/sec execute 34 sec latency 312.214 ms 1 890 0.16 MB/sec execute 35 sec latency 953.531 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2048 1269256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 39268 3559684 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 10700 3594656 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49968 7154340 1% /mnt/lustre 1 923 0.16 MB/sec execute 36 sec latency 123.646 ms 1 969 0.16 MB/sec execute 37 sec latency 103.155 ms 1 1035 0.18 MB/sec execute 38 sec latency 95.830 ms 1 1092 0.18 MB/sec execute 39 sec latency 83.019 ms 1 1156 0.18 MB/sec execute 40 sec latency 74.543 ms 1 1198 0.17 MB/sec execute 41 sec latency 89.899 ms 1 1246 0.17 MB/sec execute 42 sec latency 93.700 ms 1 1385 0.23 MB/sec execute 43 sec latency 101.443 ms test_26 fail mds2 2 times 1 1419 0.23 MB/sec execute 44 sec latency 102.560 ms Failing mds2 on oleg449-server 1 1467 0.22 MB/sec execute 45 sec latency 113.719 ms Stopping /mnt/lustre-mds2 (opts:) on oleg449-server 1 1502 0.22 MB/sec execute 46 sec latency 263.798 ms 1 1560 0.22 MB/sec execute 47 sec latency 77.149 ms 1 1625 0.21 MB/sec execute 48 sec latency 60.832 ms 1 1684 0.21 MB/sec execute 49 sec latency 125.232 ms 1 1762 0.21 MB/sec execute 50 sec latency 90.787 ms 1 1834 0.20 MB/sec execute 51 sec latency 73.412 ms 1 1905 0.20 MB/sec execute 52 sec latency 78.987 ms 1 1982 0.20 MB/sec execute 53 sec latency 80.648 ms 19:16:45 (1763511405) shut down facet: mds2 facet_host: oleg449-server facet_failover_host: oleg449-server 1 2033 0.20 MB/sec execute 54 sec latency 85.831 ms 1 2138 0.20 MB/sec execute 55 sec latency 60.828 ms 1 2191 0.19 MB/sec execute 56 sec latency 72.680 ms 1 2259 0.19 MB/sec execute 57 sec latency 381.988 ms 1 2259 0.19 MB/sec execute 58 sec latency 1382.617 ms 1 2259 0.18 MB/sec execute 59 sec latency 2382.849 ms 1 2259 0.18 MB/sec execute 60 sec latency 3385.626 ms 1 2259 0.18 MB/sec execute 61 sec latency 4385.841 ms 1 2259 0.18 MB/sec execute 62 sec latency 5386.049 ms 1 2259 0.17 MB/sec execute 63 sec latency 6386.285 ms 1 2259 0.17 MB/sec execute 64 sec latency 7386.917 ms Failover mds2 to oleg449-server mount facets: mds2 1 2259 0.17 MB/sec execute 65 sec latency 8387.128 ms 1 2259 0.16 MB/sec execute 66 sec latency 9387.606 ms 1 2259 0.16 MB/sec execute 67 sec latency 10388.770 ms 1 2259 0.16 MB/sec execute 68 sec latency 11388.997 ms 1 2259 0.16 MB/sec execute 69 sec latency 12390.583 ms 1 2259 0.16 MB/sec execute 70 sec latency 13390.820 ms 1 2259 0.15 MB/sec execute 71 sec latency 14394.197 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.15 MB/sec execute 72 sec latency 15400.294 ms 1 2259 0.15 MB/sec execute 73 sec latency 16400.764 ms 1 2259 0.15 MB/sec execute 74 sec latency 17401.101 ms 1 2259 0.14 MB/sec execute 75 sec latency 18401.337 ms 1 2259 0.14 MB/sec execute 76 sec latency 19402.651 ms oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 1 2259 0.14 MB/sec execute 77 sec latency 20406.572 ms pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 1 2259 0.14 MB/sec execute 78 sec latency 21406.819 ms Started lustre-MDT0001 19:17:10 (1763511430) targets are mounted 19:17:10 (1763511430) facet_failover done 1 2259 0.14 MB/sec execute 79 sec latency 22408.499 ms 1 2259 0.14 MB/sec execute 80 sec latency 23409.140 ms 1 2297 0.15 MB/sec execute 81 sec latency 23934.774 ms 1 2368 0.15 MB/sec execute 82 sec latency 63.515 ms oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 2416 0.15 MB/sec execute 83 sec latency 97.712 ms 1 2447 0.15 MB/sec execute 84 sec latency 126.640 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2487 0.15 MB/sec execute 85 sec latency 95.726 ms 1 2566 0.15 MB/sec execute 86 sec latency 135.797 ms 1 2608 0.15 MB/sec execute 87 sec latency 107.399 ms 1 2763 0.18 MB/sec execute 88 sec latency 125.444 ms 1 2876 0.18 MB/sec execute 89 sec latency 167.959 ms 1 2922 0.19 MB/sec execute 90 sec latency 67.724 ms 1 2982 0.19 MB/sec execute 91 sec latency 119.283 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2328 1268976 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2112 1269192 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 52660 3549184 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12428 3593380 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 65088 7142564 1% /mnt/lustre 1 3009 0.19 MB/sec execute 92 sec latency 121.825 ms 1 3171 0.20 MB/sec execute 93 sec latency 110.962 ms 1 3208 0.20 MB/sec execute 94 sec latency 145.500 ms 1 3276 0.20 MB/sec execute 95 sec latency 122.517 ms 1 3339 0.20 MB/sec execute 96 sec latency 136.135 ms 1 3373 0.20 MB/sec execute 97 sec latency 88.424 ms 1 3498 0.23 MB/sec execute 98 sec latency 132.505 ms 1 3534 0.23 MB/sec execute 99 sec latency 120.940 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec test_26 fail mds1 3 times 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 456 177.163 35278.384 Close 381 11.395 55.871 Rename 22 87.078 135.773 Unlink 57 25.817 54.992 Qpathinfo 442 16.882 137.085 Qfileinfo 116 1.298 13.776 Qfsinfo 84 3.219 24.992 Sfileinfo 50 48.867 91.202 Find 187 10.435 76.262 WriteX 372 11.888 90.643 ReadX 906 1.263 82.123 LockX 2 19.369 19.407 UnlockX 2 15.504 20.936 Flush 43 60.319 167.942 Throughput 0.226538 MB/sec 1 clients 1 procs max_latency=35278.407 ms stopping dbench on /mnt/lustre at Tue Nov 18 19:17:36 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished Failing mds1 on oleg449-server Stopping /mnt/lustre-mds1 (opts:) on oleg449-server striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench 19:17:42 (1763511462) shut down facet: mds1 facet_host: oleg449-server facet_failover_host: oleg449-server Failover mds1 to oleg449-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0000 19:18:14 (1763511494) targets are mounted 19:18:14 (1763511494) facet_failover done looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Nov 18 19:18:19 EST 2025 waiting for dbench pid 67988 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 51 2.09 MB/sec warmup 1 sec latency 102.219 ms 1 129 2.88 MB/sec warmup 2 sec latency 131.863 ms 1 173 2.35 MB/sec warmup 3 sec latency 112.041 ms 1 207 2.02 MB/sec warmup 4 sec latency 370.104 ms 1 267 2.00 MB/sec warmup 5 sec latency 91.156 ms 1 313 1.91 MB/sec warmup 6 sec latency 99.712 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2664 1268640 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2404 1268900 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 47624 3552752 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5144 3596464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 52768 7149216 1% /mnt/lustre 1 357 1.83 MB/sec warmup 7 sec latency 108.309 ms 1 413 1.82 MB/sec warmup 8 sec latency 92.047 ms 1 462 1.79 MB/sec warmup 9 sec latency 88.385 ms 1 502 1.74 MB/sec warmup 10 sec latency 195.299 ms 1 502 1.59 MB/sec warmup 11 sec latency 1195.894 ms 1 532 1.54 MB/sec warmup 12 sec latency 1326.292 ms 1 640 1.62 MB/sec warmup 13 sec latency 96.866 ms 1 695 1.51 MB/sec warmup 14 sec latency 89.331 ms test_26 fail mds2 4 times 1 736 1.42 MB/sec warmup 15 sec latency 95.150 ms Failing mds2 on oleg449-server 1 779 1.33 MB/sec warmup 16 sec latency 144.553 ms Stopping /mnt/lustre-mds2 (opts:) on oleg449-server 1 825 1.26 MB/sec warmup 17 sec latency 97.038 ms 1 878 1.20 MB/sec warmup 18 sec latency 94.863 ms 1 884 1.14 MB/sec warmup 19 sec latency 889.606 ms 19:18:38 (1763511518) shut down facet: mds2 facet_host: oleg449-server facet_failover_host: oleg449-server 1 970 0.05 MB/sec execute 1 sec latency 961.049 ms 1 995 0.07 MB/sec execute 2 sec latency 394.502 ms 1 995 0.05 MB/sec execute 3 sec latency 1395.375 ms 1 995 0.04 MB/sec execute 4 sec latency 2396.772 ms 1 995 0.03 MB/sec execute 5 sec latency 3398.787 ms 1 995 0.02 MB/sec execute 6 sec latency 4399.295 ms 1 995 0.02 MB/sec execute 7 sec latency 5401.346 ms 1 995 0.02 MB/sec execute 8 sec latency 6402.034 ms 1 995 0.02 MB/sec execute 9 sec latency 7402.796 ms Failover mds2 to oleg449-server mount facets: mds2 1 995 0.01 MB/sec execute 10 sec latency 8403.041 ms 1 995 0.01 MB/sec execute 11 sec latency 9404.098 ms 1 995 0.01 MB/sec execute 12 sec latency 10411.291 ms 1 995 0.01 MB/sec execute 13 sec latency 11411.491 ms 1 995 0.01 MB/sec execute 14 sec latency 12411.751 ms 1 995 0.01 MB/sec execute 15 sec latency 13412.011 ms 1 995 0.01 MB/sec execute 16 sec latency 14412.344 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 995 0.01 MB/sec execute 17 sec latency 15414.880 ms 1 995 0.01 MB/sec execute 18 sec latency 16415.084 ms 1 995 0.01 MB/sec execute 19 sec latency 17417.356 ms 1 995 0.01 MB/sec execute 20 sec latency 18421.330 ms 1 995 0.01 MB/sec execute 21 sec latency 19424.084 ms oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 1 995 0.01 MB/sec execute 22 sec latency 20424.911 ms pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 1 995 0.01 MB/sec execute 23 sec latency 21425.299 ms Started lustre-MDT0001 19:19:02 (1763511542) targets are mounted 19:19:02 (1763511542) facet_failover done 1 995 0.01 MB/sec execute 24 sec latency 22429.744 ms 1 997 0.01 MB/sec execute 25 sec latency 23386.734 ms 1 1059 0.04 MB/sec execute 26 sec latency 158.017 ms oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1124 0.05 MB/sec execute 27 sec latency 99.999 ms 1 1158 0.04 MB/sec execute 28 sec latency 130.652 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1208 0.04 MB/sec execute 29 sec latency 86.262 ms 1 1240 0.04 MB/sec execute 30 sec latency 163.124 ms 1 1274 0.04 MB/sec execute 31 sec latency 145.650 ms 1 1403 0.13 MB/sec execute 32 sec latency 100.246 ms 1 1449 0.13 MB/sec execute 33 sec latency 102.834 ms 1 1489 0.13 MB/sec execute 34 sec latency 358.936 ms 1 1538 0.12 MB/sec execute 35 sec latency 76.736 ms 1 1603 0.12 MB/sec execute 36 sec latency 72.835 ms 1 1647 0.12 MB/sec execute 37 sec latency 120.874 ms 1 1689 0.12 MB/sec execute 38 sec latency 110.214 ms 1 1753 0.12 MB/sec execute 39 sec latency 95.344 ms 1 1805 0.11 MB/sec execute 40 sec latency 120.982 ms 1 1845 0.11 MB/sec execute 41 sec latency 143.033 ms 1 1879 0.11 MB/sec execute 42 sec latency 126.141 ms 1 1965 0.11 MB/sec execute 43 sec latency 86.939 ms 1 2006 0.11 MB/sec execute 44 sec latency 100.804 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 2069 0.11 MB/sec execute 45 sec latency 83.595 ms 1 2140 0.11 MB/sec execute 46 sec latency 79.049 ms 1 2176 0.11 MB/sec execute 47 sec latency 92.964 ms 1 2241 0.10 MB/sec execute 48 sec latency 76.158 ms 1 2338 0.12 MB/sec execute 49 sec latency 61.684 ms 1 2377 0.12 MB/sec execute 50 sec latency 139.969 ms 1 2417 0.12 MB/sec execute 51 sec latency 141.183 ms 1 2464 0.13 MB/sec execute 52 sec latency 121.359 ms 1 2558 0.13 MB/sec execute 53 sec latency 89.205 ms 1 2602 0.13 MB/sec execute 54 sec latency 86.982 ms 1 2775 0.19 MB/sec execute 55 sec latency 72.374 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Nov 18 19:19:35 EST 2025 with return code 0 67988 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 67988 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished