== replay-dual test 26: dbench and tar with mds failover ========================================================== 02:53:34 (1788591214) Starting client oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre Started clients oleg106-client.virtnet: 192.168.201.106@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45164 Started dbench loop with 45165 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3733504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 20480 7485440 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Sep 5 02:53:40 EDT 2026 waiting for dbench pid 45233 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 17 0.20 MB/sec warmup 1 sec latency 24.400 ms 1 52 1.05 MB/sec warmup 2 sec latency 217.383 ms 1 99 1.51 MB/sec warmup 3 sec latency 138.738 ms 1 124 1.38 MB/sec warmup 4 sec latency 276.520 ms 1 155 1.31 MB/sec warmup 5 sec latency 156.285 ms test_26 fail mds1 1 times 1 182 1.22 MB/sec warmup 6 sec latency 225.321 ms 1 199 1.12 MB/sec warmup 7 sec latency 260.337 ms 1 222 1.08 MB/sec warmup 8 sec latency 280.295 ms Failing mds1 on oleg106-server 1 248 1.04 MB/sec warmup 9 sec latency 204.584 ms Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 1 279 1.04 MB/sec warmup 10 sec latency 125.086 ms 1 305 1.02 MB/sec warmup 11 sec latency 192.424 ms 1 305 0.93 MB/sec warmup 12 sec latency 1192.663 ms 1 305 0.86 MB/sec warmup 13 sec latency 2192.902 ms 02:53:54 (1788591234) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server 1 305 0.80 MB/sec warmup 14 sec latency 3195.690 ms 1 305 0.75 MB/sec warmup 15 sec latency 4196.026 ms 1 305 0.70 MB/sec warmup 16 sec latency 5196.401 ms 1 305 0.66 MB/sec warmup 17 sec latency 6196.599 ms 1 305 0.62 MB/sec warmup 18 sec latency 7196.809 ms 1 305 0.59 MB/sec warmup 19 sec latency 8198.911 ms 1 305 0.00 MB/sec execute 1 sec latency 10199.502 ms 1 305 0.00 MB/sec execute 2 sec latency 11200.150 ms 1 305 0.00 MB/sec execute 3 sec latency 12200.354 ms Failover mds1 to oleg106-server 1 305 0.00 MB/sec execute 4 sec latency 13201.159 ms mount facets: mds1 1 305 0.00 MB/sec execute 5 sec latency 14202.520 ms 1 305 0.00 MB/sec execute 6 sec latency 15202.726 ms 1 305 0.00 MB/sec execute 7 sec latency 16205.348 ms 1 305 0.00 MB/sec execute 8 sec latency 17205.550 ms 1 305 0.00 MB/sec execute 9 sec latency 18206.881 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 305 0.00 MB/sec execute 10 sec latency 19207.968 ms 1 305 0.00 MB/sec execute 11 sec latency 20208.215 ms 1 305 0.00 MB/sec execute 12 sec latency 21208.409 ms 1 305 0.00 MB/sec execute 13 sec latency 22208.617 ms 1 305 0.00 MB/sec execute 14 sec latency 23210.520 ms 1 305 0.00 MB/sec execute 15 sec latency 24210.772 ms 1 305 0.00 MB/sec execute 16 sec latency 25212.423 ms 1 305 0.00 MB/sec execute 17 sec latency 26214.777 ms 1 305 0.00 MB/sec execute 18 sec latency 27215.762 ms 1 305 0.00 MB/sec execute 19 sec latency 28219.157 ms 1 305 0.00 MB/sec execute 20 sec latency 29219.423 ms 1 305 0.00 MB/sec execute 21 sec latency 30221.142 ms 1 305 0.00 MB/sec execute 22 sec latency 31221.958 ms 1 305 0.00 MB/sec execute 23 sec latency 32223.109 ms 1 305 0.00 MB/sec execute 24 sec latency 33224.507 ms 1 305 0.00 MB/sec execute 25 sec latency 34224.698 ms 1 305 0.00 MB/sec execute 26 sec latency 35224.927 ms oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all 1 305 0.00 MB/sec execute 27 sec latency 36225.102 ms pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 1 305 0.00 MB/sec execute 28 sec latency 37225.361 ms Started lustre-MDT0000 1 305 0.00 MB/sec execute 29 sec latency 38226.140 ms 02:54:30 (1788591270) targets are mounted 02:54:30 (1788591270) facet_failover done 1 305 0.00 MB/sec execute 30 sec latency 39228.420 ms 1 305 0.00 MB/sec execute 31 sec latency 40232.822 ms 1 305 0.00 MB/sec execute 32 sec latency 41234.652 ms 1 305 0.00 MB/sec execute 33 sec latency 42248.391 ms 1 323 0.02 MB/sec execute 34 sec latency 42515.277 ms 1 358 0.04 MB/sec execute 35 sec latency 132.909 ms 1 389 0.07 MB/sec execute 36 sec latency 153.953 ms 1 424 0.10 MB/sec execute 37 sec latency 117.980 ms 1 453 0.12 MB/sec execute 38 sec latency 151.090 ms oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 485 0.15 MB/sec execute 39 sec latency 172.902 ms 1 511 0.16 MB/sec execute 40 sec latency 174.204 ms 1 542 0.19 MB/sec execute 41 sec latency 159.974 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 612 0.23 MB/sec execute 42 sec latency 160.920 ms 1 649 0.23 MB/sec execute 43 sec latency 166.048 ms 1 673 0.23 MB/sec execute 44 sec latency 251.890 ms 1 700 0.22 MB/sec execute 45 sec latency 135.748 ms 1 726 0.22 MB/sec execute 46 sec latency 241.400 ms 1 759 0.21 MB/sec execute 47 sec latency 136.386 ms 1 789 0.21 MB/sec execute 48 sec latency 235.961 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 38912 3657728 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3740672 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 49152 7398400 1% /mnt/lustre 1 817 0.21 MB/sec execute 49 sec latency 268.647 ms 1 865 0.21 MB/sec execute 50 sec latency 154.372 ms 1 892 0.20 MB/sec execute 51 sec latency 190.832 ms 1 916 0.20 MB/sec execute 52 sec latency 263.921 ms 1 948 0.20 MB/sec execute 53 sec latency 122.008 ms 1 971 0.20 MB/sec execute 54 sec latency 125.293 ms test_26 fail mds1 2 times 1 988 0.19 MB/sec execute 55 sec latency 301.166 ms 1 1041 0.21 MB/sec execute 56 sec latency 145.640 ms 1 1077 0.20 MB/sec execute 57 sec latency 117.270 ms Failing mds1 on oleg106-server 1 1128 0.20 MB/sec execute 58 sec latency 98.848 ms Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 1 1160 0.20 MB/sec execute 59 sec latency 132.434 ms 1 1160 0.20 MB/sec execute 60 sec latency 1101.802 ms 1 1160 0.19 MB/sec execute 61 sec latency 2104.993 ms 02:55:03 (1788591303) shut down 1 1160 0.19 MB/sec execute 62 sec latency 3105.141 ms facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server 1 1160 0.19 MB/sec execute 63 sec latency 4105.328 ms 1 1160 0.18 MB/sec execute 64 sec latency 5105.560 ms 1 1160 0.18 MB/sec execute 65 sec latency 6105.800 ms 1 1160 0.18 MB/sec execute 66 sec latency 7105.986 ms 1 1160 0.18 MB/sec execute 67 sec latency 8108.237 ms 1 1160 0.17 MB/sec execute 68 sec latency 9109.781 ms 1 1160 0.17 MB/sec execute 69 sec latency 10109.986 ms 1 1160 0.17 MB/sec execute 70 sec latency 11110.183 ms 1 1160 0.17 MB/sec execute 71 sec latency 12111.636 ms 1 1160 0.16 MB/sec execute 72 sec latency 13112.269 ms Failover mds1 to oleg106-server mount facets: mds1 1 1160 0.16 MB/sec execute 73 sec latency 14113.630 ms 1 1160 0.16 MB/sec execute 74 sec latency 15123.093 ms 1 1160 0.16 MB/sec execute 75 sec latency 16123.433 ms 1 1160 0.15 MB/sec execute 76 sec latency 17123.628 ms 1 1160 0.15 MB/sec execute 77 sec latency 18123.825 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1160 0.15 MB/sec execute 78 sec latency 19124.067 ms 1 1160 0.15 MB/sec execute 79 sec latency 20124.805 ms 1 1160 0.15 MB/sec execute 80 sec latency 21128.621 ms 1 1160 0.15 MB/sec execute 81 sec latency 22128.830 ms 1 1160 0.14 MB/sec execute 82 sec latency 23129.208 ms 1 1160 0.14 MB/sec execute 83 sec latency 24129.416 ms 1 1160 0.14 MB/sec execute 84 sec latency 25131.274 ms 1 1160 0.14 MB/sec execute 85 sec latency 26131.549 ms 1 1160 0.14 MB/sec execute 86 sec latency 27136.281 ms 1 1160 0.14 MB/sec execute 87 sec latency 28136.654 ms 1 1160 0.13 MB/sec execute 88 sec latency 29137.093 ms 1 1160 0.13 MB/sec execute 89 sec latency 30138.235 ms 1 1160 0.13 MB/sec execute 90 sec latency 31138.746 ms 1 1160 0.13 MB/sec execute 91 sec latency 32138.950 ms 1 1160 0.13 MB/sec execute 92 sec latency 33139.245 ms 1 1160 0.13 MB/sec execute 93 sec latency 34139.482 ms oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all 1 1160 0.13 MB/sec execute 94 sec latency 35139.707 ms pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 1 1160 0.12 MB/sec execute 95 sec latency 36140.145 ms Started lustre-MDT0000 02:55:36 (1788591336) targets are mounted 02:55:36 (1788591336) facet_failover done 1 1160 0.12 MB/sec execute 96 sec latency 37140.601 ms 1 1160 0.12 MB/sec execute 97 sec latency 38141.696 ms 1 1160 0.12 MB/sec execute 98 sec latency 39143.454 ms 1 1161 0.12 MB/sec execute 99 sec latency 40088.751 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 125 429.246 42515.262 Close 114 16.812 70.799 Rename 7 100.830 132.411 Unlink 13 31.499 53.617 Qpathinfo 69 607.721 40088.722 Qfileinfo 54 2.549 50.691 Qfsinfo 28 3.889 22.388 Sfileinfo 25 63.601 121.154 Find 36 8.624 82.570 WriteX 264 16.892 101.706 ReadX 122 6.421 119.987 Flush 24 139.216 301.149 Throughput 0.118827 MB/sec 1 clients 1 procs max_latency=42515.277 ms stopping dbench on /mnt/lustre at Sat Sep 5 02:55:45 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Sep 5 02:55:52 EDT 2026 waiting for dbench pid 47198 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 43 1.64 MB/sec warmup 1 sec latency 115.482 ms 1 111 2.62 MB/sec warmup 2 sec latency 119.445 ms 1 139 2.01 MB/sec warmup 3 sec latency 197.369 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3732480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 28672 3725312 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 47104 7457792 1% /mnt/lustre 1 157 1.63 MB/sec warmup 4 sec latency 228.953 ms 1 181 1.46 MB/sec warmup 5 sec latency 232.577 ms 1 204 1.33 MB/sec warmup 6 sec latency 246.604 ms 1 239 1.31 MB/sec warmup 7 sec latency 164.901 ms 1 279 1.30 MB/sec warmup 8 sec latency 161.437 ms 1 308 1.24 MB/sec warmup 9 sec latency 220.211 ms 1 349 1.25 MB/sec warmup 10 sec latency 123.387 ms test_26 fail mds1 3 times 1 397 1.28 MB/sec warmup 11 sec latency 169.662 ms Failing mds1 on oleg106-server 1 429 1.26 MB/sec warmup 12 sec latency 145.422 ms 1 469 1.26 MB/sec warmup 13 sec latency 190.262 ms Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 1 501 1.24 MB/sec warmup 14 sec latency 185.101 ms 1 510 1.18 MB/sec warmup 15 sec latency 721.759 ms 1 510 1.11 MB/sec warmup 16 sec latency 1721.959 ms 1 510 1.04 MB/sec warmup 17 sec latency 2722.197 ms 02:56:09 (1788591369) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server 1 510 0.98 MB/sec warmup 18 sec latency 3722.491 ms 1 510 0.93 MB/sec warmup 19 sec latency 4722.748 ms 1 510 0.00 MB/sec execute 1 sec latency 6724.899 ms 1 510 0.00 MB/sec execute 2 sec latency 7726.282 ms 1 510 0.00 MB/sec execute 3 sec latency 8726.478 ms 1 510 0.00 MB/sec execute 4 sec latency 9726.687 ms 1 510 0.00 MB/sec execute 5 sec latency 10726.834 ms 1 510 0.00 MB/sec execute 6 sec latency 11727.095 ms 1 510 0.00 MB/sec execute 7 sec latency 12727.323 ms 1 510 0.00 MB/sec execute 8 sec latency 13727.813 ms Failover mds1 to oleg106-server mount facets: mds1 1 510 0.00 MB/sec execute 9 sec latency 14727.936 ms 1 510 0.00 MB/sec execute 10 sec latency 15728.147 ms 1 510 0.00 MB/sec execute 11 sec latency 16735.011 ms 1 510 0.00 MB/sec execute 12 sec latency 17744.639 ms 1 510 0.00 MB/sec execute 13 sec latency 18745.042 ms 1 510 0.00 MB/sec execute 14 sec latency 19745.233 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 510 0.00 MB/sec execute 15 sec latency 20747.165 ms 1 510 0.00 MB/sec execute 16 sec latency 21748.103 ms 1 510 0.00 MB/sec execute 17 sec latency 22748.389 ms 1 510 0.00 MB/sec execute 18 sec latency 23748.648 ms 1 510 0.00 MB/sec execute 19 sec latency 24749.154 ms 1 510 0.00 MB/sec execute 20 sec latency 25755.461 ms 1 510 0.00 MB/sec execute 21 sec latency 26755.655 ms 1 510 0.00 MB/sec execute 22 sec latency 27756.593 ms 1 510 0.00 MB/sec execute 23 sec latency 28756.853 ms 1 510 0.00 MB/sec execute 24 sec latency 29758.250 ms 1 510 0.00 MB/sec execute 25 sec latency 30758.437 ms 1 510 0.00 MB/sec execute 26 sec latency 31758.620 ms 1 510 0.00 MB/sec execute 27 sec latency 32759.086 ms 1 510 0.00 MB/sec execute 28 sec latency 33760.644 ms oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all 1 510 0.00 MB/sec execute 29 sec latency 34760.837 ms pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 1 510 0.00 MB/sec execute 30 sec latency 35761.017 ms 1 510 0.00 MB/sec execute 31 sec latency 36763.310 ms Started lustre-MDT0000 02:56:43 (1788591403) targets are mounted 02:56:43 (1788591403) facet_failover done 1 510 0.00 MB/sec execute 32 sec latency 37763.563 ms 1 510 0.00 MB/sec execute 33 sec latency 38763.807 ms 1 510 0.00 MB/sec execute 34 sec latency 39764.237 ms 1 522 0.01 MB/sec execute 35 sec latency 39962.638 ms 1 615 0.09 MB/sec execute 36 sec latency 157.913 ms 1 661 0.09 MB/sec execute 37 sec latency 203.924 ms 1 674 0.09 MB/sec execute 38 sec latency 408.096 ms 1 697 0.09 MB/sec execute 39 sec latency 174.339 ms 1 721 0.09 MB/sec execute 40 sec latency 158.028 ms oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 734 0.09 MB/sec execute 41 sec latency 273.215 ms 1 769 0.09 MB/sec execute 42 sec latency 111.379 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 789 0.09 MB/sec execute 43 sec latency 251.913 ms 1 808 0.09 MB/sec execute 44 sec latency 356.155 ms 1 863 0.09 MB/sec execute 45 sec latency 152.665 ms 1 887 0.09 MB/sec execute 46 sec latency 177.584 ms 1 914 0.09 MB/sec execute 47 sec latency 208.047 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 957 0.08 MB/sec execute 48 sec latency 107.542 ms 1 983 0.09 MB/sec execute 49 sec latency 355.372 ms 1 1031 0.10 MB/sec execute 50 sec latency 179.430 ms 1 1086 0.10 MB/sec execute 51 sec latency 76.010 ms 1 1147 0.10 MB/sec execute 52 sec latency 85.993 ms 1 1198 0.10 MB/sec execute 53 sec latency 104.625 ms 1 1244 0.10 MB/sec execute 54 sec latency 77.043 ms 1 1274 0.10 MB/sec execute 55 sec latency 126.872 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sat Sep 5 02:57:08 EDT 2026 with return code 0 1 1418 0.15 MB/sec execute 56 sec latency 134.651 ms 47198 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47198 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished