== replay-dual test 26: dbench and tar with mds failover ========================================================== 02:50:13 (1778655013) Starting client oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre Started clients oleg108-client.virtnet: 192.168.201.108@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44497 Started dbench loop with 44498 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3733504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7495680 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed May 13 02:50:19 EDT 2026 waiting for dbench pid 44567 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 42 1.57 MB/sec warmup 1 sec latency 92.103 ms 1 110 2.62 MB/sec warmup 2 sec latency 135.052 ms 1 148 2.09 MB/sec warmup 3 sec latency 156.791 ms 1 189 1.89 MB/sec warmup 4 sec latency 119.178 ms test_26 fail mds1 1 times 1 215 1.67 MB/sec warmup 5 sec latency 211.576 ms 1 244 1.54 MB/sec warmup 6 sec latency 131.375 ms 1 279 1.49 MB/sec warmup 7 sec latency 156.770 ms Failing mds1 on oleg108-server 1 306 1.40 MB/sec warmup 8 sec latency 186.723 ms Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 331 1.33 MB/sec warmup 9 sec latency 176.190 ms 1 341 1.22 MB/sec warmup 10 sec latency 511.386 ms 1 341 1.11 MB/sec warmup 11 sec latency 1513.358 ms 1 341 1.02 MB/sec warmup 12 sec latency 2513.575 ms 1 341 0.94 MB/sec warmup 13 sec latency 3513.978 ms 02:50:32 (1778655032) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 341 0.87 MB/sec warmup 14 sec latency 4515.438 ms 1 341 0.82 MB/sec warmup 15 sec latency 5515.722 ms 1 341 0.77 MB/sec warmup 16 sec latency 6515.909 ms 1 341 0.72 MB/sec warmup 17 sec latency 7517.536 ms 1 341 0.68 MB/sec warmup 18 sec latency 8517.826 ms 1 341 0.64 MB/sec warmup 19 sec latency 9518.742 ms 1 341 0.00 MB/sec execute 1 sec latency 11528.209 ms 1 341 0.00 MB/sec execute 2 sec latency 12530.377 ms 1 341 0.00 MB/sec execute 3 sec latency 13530.609 ms 1 341 0.00 MB/sec execute 4 sec latency 14534.462 ms Failover mds1 to oleg108-server mount facets: mds1 1 341 0.00 MB/sec execute 5 sec latency 15535.188 ms 1 341 0.00 MB/sec execute 6 sec latency 16537.117 ms 1 341 0.00 MB/sec execute 7 sec latency 17538.085 ms 1 341 0.00 MB/sec execute 8 sec latency 18538.677 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 341 0.00 MB/sec execute 9 sec latency 19539.024 ms 1 341 0.00 MB/sec execute 10 sec latency 20539.289 ms 1 341 0.00 MB/sec execute 11 sec latency 21539.578 ms 1 341 0.00 MB/sec execute 12 sec latency 22540.092 ms 1 341 0.00 MB/sec execute 13 sec latency 23540.305 ms 1 341 0.00 MB/sec execute 14 sec latency 24542.103 ms 1 341 0.00 MB/sec execute 15 sec latency 25546.023 ms 1 341 0.00 MB/sec execute 16 sec latency 26546.500 ms 1 341 0.00 MB/sec execute 17 sec latency 27549.604 ms 1 341 0.00 MB/sec execute 18 sec latency 28550.054 ms 1 341 0.00 MB/sec execute 19 sec latency 29551.186 ms 1 341 0.00 MB/sec execute 20 sec latency 30551.454 ms 1 341 0.00 MB/sec execute 21 sec latency 31551.731 ms 1 341 0.00 MB/sec execute 22 sec latency 32552.488 ms 1 341 0.00 MB/sec execute 23 sec latency 33555.140 ms 1 341 0.00 MB/sec execute 24 sec latency 34555.491 ms 1 341 0.00 MB/sec execute 25 sec latency 35558.429 ms oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all 1 341 0.00 MB/sec execute 26 sec latency 36558.758 ms pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 1 341 0.00 MB/sec execute 27 sec latency 37559.577 ms Started lustre-MDT0000 02:51:06 (1778655066) targets are mounted 02:51:06 (1778655066) facet_failover done 1 341 0.00 MB/sec execute 28 sec latency 38560.069 ms 1 341 0.00 MB/sec execute 29 sec latency 39560.293 ms 1 341 0.00 MB/sec execute 30 sec latency 40561.764 ms 1 341 0.00 MB/sec execute 31 sec latency 41561.911 ms 1 341 0.00 MB/sec execute 32 sec latency 42564.633 ms 1 341 0.00 MB/sec execute 33 sec latency 43567.749 ms 1 341 0.00 MB/sec execute 34 sec latency 44568.245 ms 1 341 0.00 MB/sec execute 35 sec latency 45576.796 ms 1 365 0.02 MB/sec execute 36 sec latency 45651.537 ms 1 389 0.04 MB/sec execute 37 sec latency 207.091 ms 1 411 0.06 MB/sec execute 38 sec latency 180.693 ms 1 429 0.07 MB/sec execute 39 sec latency 260.755 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 445 0.08 MB/sec execute 40 sec latency 279.700 ms 1 462 0.09 MB/sec execute 41 sec latency 252.368 ms 1 485 0.11 MB/sec execute 42 sec latency 199.192 ms 1 503 0.12 MB/sec execute 43 sec latency 191.210 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 514 0.12 MB/sec execute 44 sec latency 410.015 ms 1 532 0.14 MB/sec execute 45 sec latency 265.843 ms 1 578 0.16 MB/sec execute 46 sec latency 294.642 ms 1 610 0.18 MB/sec execute 47 sec latency 138.355 ms 1 639 0.18 MB/sec execute 48 sec latency 293.579 ms 1 663 0.18 MB/sec execute 49 sec latency 199.672 ms 1 673 0.18 MB/sec execute 50 sec latency 387.580 ms 1 695 0.17 MB/sec execute 51 sec latency 233.020 ms 1 721 0.17 MB/sec execute 52 sec latency 150.321 ms 1 738 0.17 MB/sec execute 53 sec latency 248.973 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3705856 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 11264 3747840 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 48128 7453696 1% /mnt/lustre 1 765 0.17 MB/sec execute 54 sec latency 169.727 ms 1 778 0.16 MB/sec execute 55 sec latency 234.838 ms 1 789 0.16 MB/sec execute 56 sec latency 298.439 ms 1 796 0.16 MB/sec execute 57 sec latency 585.273 ms 1 799 0.16 MB/sec execute 58 sec latency 667.635 ms 1 818 0.16 MB/sec execute 59 sec latency 182.148 ms 1 854 0.15 MB/sec execute 60 sec latency 213.346 ms 1 874 0.15 MB/sec execute 61 sec latency 296.388 ms 1 902 0.15 MB/sec execute 62 sec latency 211.532 ms test_26 fail mds1 2 times 1 913 0.15 MB/sec execute 63 sec latency 312.360 ms 1 919 0.15 MB/sec execute 64 sec latency 685.285 ms 1 949 0.15 MB/sec execute 65 sec latency 169.742 ms Failing mds1 on oleg108-server 1 969 0.14 MB/sec execute 66 sec latency 211.667 ms 1 981 0.14 MB/sec execute 67 sec latency 492.849 ms Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 998 0.14 MB/sec execute 68 sec latency 301.535 ms 1 1028 0.15 MB/sec execute 69 sec latency 663.823 ms 1 1028 0.15 MB/sec execute 70 sec latency 1664.582 ms 02:51:50 (1778655110) shut down 1 1028 0.15 MB/sec execute 71 sec latency 2686.259 ms facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 1028 0.15 MB/sec execute 72 sec latency 3690.958 ms 1 1028 0.15 MB/sec execute 73 sec latency 4691.175 ms 1 1028 0.14 MB/sec execute 74 sec latency 5691.449 ms 1 1028 0.14 MB/sec execute 75 sec latency 6693.368 ms 1 1028 0.14 MB/sec execute 76 sec latency 7693.649 ms 1 1028 0.14 MB/sec execute 77 sec latency 8696.634 ms 1 1028 0.14 MB/sec execute 78 sec latency 9699.034 ms 1 1028 0.13 MB/sec execute 79 sec latency 10699.233 ms 1 1028 0.13 MB/sec execute 80 sec latency 11702.518 ms 1 1028 0.13 MB/sec execute 81 sec latency 12702.994 ms Failover mds1 to oleg108-server mount facets: mds1 1 1028 0.13 MB/sec execute 82 sec latency 13703.188 ms 1 1028 0.13 MB/sec execute 83 sec latency 14703.478 ms 1 1028 0.13 MB/sec execute 84 sec latency 15703.694 ms 1 1028 0.12 MB/sec execute 85 sec latency 16704.128 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1028 0.12 MB/sec execute 86 sec latency 17704.415 ms 1 1028 0.12 MB/sec execute 87 sec latency 18705.821 ms 1 1028 0.12 MB/sec execute 88 sec latency 19706.081 ms 1 1028 0.12 MB/sec execute 89 sec latency 20706.342 ms 1 1028 0.12 MB/sec execute 90 sec latency 21706.729 ms 1 1028 0.12 MB/sec execute 91 sec latency 22707.283 ms 1 1028 0.12 MB/sec execute 92 sec latency 23707.640 ms 1 1028 0.11 MB/sec execute 93 sec latency 24710.153 ms 1 1028 0.11 MB/sec execute 94 sec latency 25711.671 ms 1 1028 0.11 MB/sec execute 95 sec latency 26718.812 ms 1 1028 0.11 MB/sec execute 96 sec latency 27720.470 ms 1 1028 0.11 MB/sec execute 97 sec latency 28720.621 ms 1 1028 0.11 MB/sec execute 98 sec latency 29722.451 ms 1 1028 0.11 MB/sec execute 99 sec latency 30732.608 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 1 cleanup 107 sec 1 cleanup 108 sec Started lustre-MDT0000 02:52:28 (1778655148) targets are mounted 02:52:28 (1778655148) facet_failover done 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec 0 cleanup 116 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 98 608.238 45651.522 Close 86 23.490 77.754 Rename 4 196.634 272.722 Unlink 10 36.318 66.275 Qpathinfo 30 1390.479 40950.596 Qfileinfo 46 3.954 20.104 Qfsinfo 18 10.907 77.188 Sfileinfo 18 95.440 208.309 Find 25 14.250 138.340 WriteX 233 26.835 199.972 ReadX 96 5.882 143.129 Flush 24 257.733 685.268 Throughput 0.106985 MB/sec 1 clients 1 procs max_latency=45651.537 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid stopping dbench on /mnt/lustre at Wed May 13 02:52:35 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed May 13 02:52:45 EDT 2026 waiting for dbench pid 46509 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 15 0.07 MB/sec warmup 1 sec latency 34.154 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 27648 3732480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7486464 1% /mnt/lustre 1 44 0.85 MB/sec warmup 2 sec latency 242.437 ms 1 79 1.16 MB/sec warmup 3 sec latency 220.543 ms 1 113 1.31 MB/sec warmup 4 sec latency 264.199 ms 1 147 1.25 MB/sec warmup 5 sec latency 140.845 ms 1 182 1.22 MB/sec warmup 6 sec latency 196.134 ms 1 207 1.16 MB/sec warmup 7 sec latency 236.895 ms 1 246 1.17 MB/sec warmup 8 sec latency 140.260 ms test_26 fail mds1 3 times 1 287 1.19 MB/sec warmup 9 sec latency 136.652 ms 1 313 1.15 MB/sec warmup 10 sec latency 170.418 ms Failing mds1 on oleg108-server 1 339 1.11 MB/sec warmup 11 sec latency 219.047 ms Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 381 1.13 MB/sec warmup 12 sec latency 95.697 ms 1 389 1.06 MB/sec warmup 13 sec latency 846.388 ms 1 389 0.99 MB/sec warmup 14 sec latency 1846.586 ms 1 389 0.92 MB/sec warmup 15 sec latency 2848.851 ms 02:53:01 (1778655181) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 389 0.86 MB/sec warmup 16 sec latency 3849.048 ms 1 389 0.81 MB/sec warmup 17 sec latency 4849.494 ms 1 389 0.77 MB/sec warmup 18 sec latency 5849.873 ms 1 389 0.73 MB/sec warmup 19 sec latency 6851.143 ms 1 389 0.00 MB/sec execute 1 sec latency 8852.980 ms 1 389 0.00 MB/sec execute 2 sec latency 9853.981 ms 1 389 0.00 MB/sec execute 3 sec latency 10854.232 ms 1 389 0.00 MB/sec execute 4 sec latency 11856.789 ms 1 389 0.00 MB/sec execute 5 sec latency 12857.282 ms Failover mds1 to oleg108-server mount facets: mds1 1 389 0.00 MB/sec execute 6 sec latency 13857.517 ms 1 389 0.00 MB/sec execute 7 sec latency 14857.678 ms 1 389 0.00 MB/sec execute 8 sec latency 15857.884 ms 1 389 0.00 MB/sec execute 9 sec latency 16858.301 ms 1 389 0.00 MB/sec execute 10 sec latency 17858.756 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 389 0.00 MB/sec execute 11 sec latency 18858.985 ms 1 389 0.00 MB/sec execute 12 sec latency 19859.196 ms 1 389 0.00 MB/sec execute 13 sec latency 20861.361 ms 1 389 0.00 MB/sec execute 14 sec latency 21861.634 ms 1 389 0.00 MB/sec execute 15 sec latency 22861.886 ms 1 389 0.00 MB/sec execute 16 sec latency 23862.348 ms 1 389 0.00 MB/sec execute 17 sec latency 24862.547 ms 1 389 0.00 MB/sec execute 18 sec latency 25862.741 ms oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all 1 389 0.00 MB/sec execute 19 sec latency 26865.125 ms pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 1 389 0.00 MB/sec execute 20 sec latency 27865.302 ms Started lustre-MDT0000 02:53:26 (1778655206) targets are mounted 02:53:26 (1778655206) facet_failover done 1 389 0.00 MB/sec execute 21 sec latency 28876.323 ms 1 389 0.00 MB/sec execute 22 sec latency 29877.766 ms 1 389 0.00 MB/sec execute 23 sec latency 30878.014 ms 1 389 0.00 MB/sec execute 24 sec latency 31880.471 ms 1 389 0.00 MB/sec execute 25 sec latency 32881.010 ms 1 389 0.00 MB/sec execute 26 sec latency 33881.294 ms 1 389 0.00 MB/sec execute 27 sec latency 34881.490 ms 1 389 0.00 MB/sec execute 28 sec latency 35881.689 ms 1 389 0.00 MB/sec execute 29 sec latency 36881.912 ms 1 389 0.00 MB/sec execute 30 sec latency 37888.756 ms 1 389 0.00 MB/sec execute 31 sec latency 38892.314 ms 1 389 0.00 MB/sec execute 32 sec latency 39892.556 ms 1 413 0.02 MB/sec execute 33 sec latency 40324.675 ms 1 462 0.07 MB/sec execute 34 sec latency 103.863 ms 1 494 0.10 MB/sec execute 35 sec latency 188.455 ms 1 513 0.11 MB/sec execute 36 sec latency 228.310 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 561 0.14 MB/sec execute 37 sec latency 196.096 ms 1 616 0.19 MB/sec execute 38 sec latency 154.669 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 654 0.19 MB/sec execute 39 sec latency 225.729 ms 1 673 0.18 MB/sec execute 40 sec latency 340.670 ms 1 689 0.18 MB/sec execute 41 sec latency 330.783 ms 1 710 0.18 MB/sec execute 42 sec latency 208.900 ms 1 730 0.17 MB/sec execute 43 sec latency 280.824 ms 1 765 0.17 MB/sec execute 44 sec latency 130.366 ms 1 794 0.17 MB/sec execute 45 sec latency 267.298 ms 1 825 0.17 MB/sec execute 46 sec latency 105.531 ms 1 874 0.17 MB/sec execute 47 sec latency 330.220 ms 1 900 0.16 MB/sec execute 48 sec latency 168.217 ms 1 923 0.16 MB/sec execute 49 sec latency 284.721 ms 1 980 0.16 MB/sec execute 50 sec latency 76.361 ms 1 1017 0.17 MB/sec execute 51 sec latency 234.049 ms 1 1064 0.17 MB/sec execute 52 sec latency 112.615 ms 1 1118 0.17 MB/sec execute 53 sec latency 96.799 ms 1 1172 0.17 MB/sec execute 54 sec latency 72.917 ms 1 1216 0.17 MB/sec execute 55 sec latency 80.520 ms 1 1254 0.17 MB/sec execute 56 sec latency 164.611 ms 1 1378 0.21 MB/sec execute 57 sec latency 135.731 ms 1 1421 0.21 MB/sec execute 58 sec latency 121.768 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed May 13 02:54:04 EDT 2026 with return code 0 46509 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 46509 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished