== replay-dual test 26: dbench and tar with mds failover ========================================================== 07:16:54 (1781695014) Starting client oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre Started clients oleg221-client.virtnet: 192.168.202.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44830 Started dbench loop with 44831 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Jun 17 07:16:56 EDT 2026 waiting for dbench pid 44863 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3720192 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3739648 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 29696 7459840 1% /mnt/lustre 1 106 4.98 MB/sec warmup 1 sec latency 43.872 ms 1 199 3.92 MB/sec warmup 2 sec latency 81.374 ms 1 306 3.74 MB/sec warmup 3 sec latency 42.269 ms 1 422 3.72 MB/sec warmup 4 sec latency 39.671 ms test_26 fail mds1 1 times Failing mds1 on oleg221-server 1 519 3.58 MB/sec warmup 5 sec latency 49.716 ms Stopping /mnt/lustre-mds1 (opts:) on oleg221-server 1 608 3.49 MB/sec warmup 6 sec latency 459.960 ms 07:17:03 (1781695023) shut down facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server 1 608 3.00 MB/sec warmup 7 sec latency 1460.169 ms 1 608 2.62 MB/sec warmup 8 sec latency 2460.321 ms 1 608 2.33 MB/sec warmup 9 sec latency 3460.480 ms 1 608 2.10 MB/sec warmup 10 sec latency 4460.669 ms 1 608 1.91 MB/sec warmup 11 sec latency 5460.837 ms 1 608 1.75 MB/sec warmup 12 sec latency 6461.064 ms 1 608 1.61 MB/sec warmup 13 sec latency 7461.273 ms 1 608 1.50 MB/sec warmup 14 sec latency 8461.452 ms 1 608 1.40 MB/sec warmup 15 sec latency 9461.615 ms 1 608 1.31 MB/sec warmup 16 sec latency 10461.780 ms Failover mds1 to oleg221-server mount facets: mds1 1 608 1.23 MB/sec warmup 17 sec latency 11461.960 ms 1 608 1.16 MB/sec warmup 18 sec latency 12462.113 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 608 1.10 MB/sec warmup 19 sec latency 13462.325 ms oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all 1 608 0.00 MB/sec execute 1 sec latency 15462.648 ms pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 07:17:17 (1781695037) targets are mounted 07:17:17 (1781695037) facet_failover done 1 608 0.00 MB/sec execute 2 sec latency 16462.792 ms 1 608 0.00 MB/sec execute 3 sec latency 17463.006 ms 1 608 0.00 MB/sec execute 4 sec latency 18463.209 ms 1 608 0.00 MB/sec execute 5 sec latency 19463.468 ms 1 608 0.00 MB/sec execute 6 sec latency 20463.662 ms 1 609 0.00 MB/sec execute 7 sec latency 21403.820 ms 1 728 0.04 MB/sec execute 8 sec latency 103.099 ms oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 825 0.06 MB/sec execute 9 sec latency 52.556 ms 1 933 0.08 MB/sec execute 10 sec latency 62.989 ms 1 1011 0.12 MB/sec execute 11 sec latency 80.216 ms 1 1163 0.17 MB/sec execute 12 sec latency 88.939 ms 1 1387 0.39 MB/sec execute 13 sec latency 29.274 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 35840 3604480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3690496 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7294976 1% /mnt/lustre 1 1490 0.36 MB/sec execute 14 sec latency 163.879 ms 1 1570 0.34 MB/sec execute 15 sec latency 61.782 ms 1 1646 0.33 MB/sec execute 16 sec latency 69.697 ms test_26 fail mds1 2 times Failing mds1 on oleg221-server 1 1789 0.31 MB/sec execute 17 sec latency 38.945 ms Stopping /mnt/lustre-mds1 (opts:) on oleg221-server 1 1868 0.30 MB/sec execute 18 sec latency 498.681 ms 07:17:34 (1781695054) shut down facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server 1 1868 0.28 MB/sec execute 19 sec latency 1498.821 ms 1 1868 0.27 MB/sec execute 20 sec latency 2498.973 ms 1 1868 0.26 MB/sec execute 21 sec latency 3499.155 ms 1 1868 0.25 MB/sec execute 22 sec latency 4499.307 ms 1 1868 0.23 MB/sec execute 23 sec latency 5499.451 ms 1 1868 0.22 MB/sec execute 24 sec latency 6499.600 ms 1 1868 0.22 MB/sec execute 25 sec latency 7499.750 ms 1 1868 0.21 MB/sec execute 26 sec latency 8499.883 ms 1 1868 0.20 MB/sec execute 27 sec latency 9500.033 ms 1 1868 0.19 MB/sec execute 28 sec latency 10500.212 ms Failover mds1 to oleg221-server mount facets: mds1 1 1868 0.19 MB/sec execute 29 sec latency 11500.349 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1868 0.18 MB/sec execute 30 sec latency 12500.480 ms 1 1868 0.17 MB/sec execute 31 sec latency 13500.619 ms oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 1 1868 0.17 MB/sec execute 32 sec latency 14500.769 ms Started lustre-MDT0000 07:17:48 (1781695068) targets are mounted 07:17:48 (1781695068) facet_failover done 1 1868 0.16 MB/sec execute 33 sec latency 15500.895 ms 1 1868 0.16 MB/sec execute 34 sec latency 16501.063 ms 1 1868 0.15 MB/sec execute 35 sec latency 17501.269 ms 1 1868 0.15 MB/sec execute 36 sec latency 18501.468 ms 1 1899 0.15 MB/sec execute 37 sec latency 19292.371 ms 1 2068 0.15 MB/sec execute 38 sec latency 28.536 ms oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2287 0.17 MB/sec execute 39 sec latency 31.188 ms 1 2437 0.18 MB/sec execute 40 sec latency 74.463 ms 1 2566 0.19 MB/sec execute 41 sec latency 452.668 ms 1 2733 0.26 MB/sec execute 42 sec latency 938.133 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3572736 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3607552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 71680 7180288 1% /mnt/lustre 1 2989 0.29 MB/sec execute 43 sec latency 49.413 ms 1 3281 0.33 MB/sec execute 44 sec latency 44.006 ms 1 3507 0.39 MB/sec execute 45 sec latency 34.692 ms test_26 fail mds1 3 times 1 3645 0.38 MB/sec execute 46 sec latency 30.751 ms Failing mds1 on oleg221-server Stopping /mnt/lustre-mds1 (opts:) on oleg221-server 1 3788 0.40 MB/sec execute 47 sec latency 38.516 ms 1 3788 0.39 MB/sec execute 48 sec latency 1003.315 ms 07:18:04 (1781695084) shut down facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server 1 3788 0.38 MB/sec execute 49 sec latency 2003.529 ms 1 3788 0.38 MB/sec execute 50 sec latency 3003.757 ms 1 3788 0.37 MB/sec execute 51 sec latency 4003.973 ms 1 3788 0.36 MB/sec execute 52 sec latency 5004.184 ms 1 3788 0.35 MB/sec execute 53 sec latency 6004.424 ms 1 3788 0.35 MB/sec execute 54 sec latency 7004.615 ms 1 3788 0.34 MB/sec execute 55 sec latency 8004.873 ms 1 3788 0.34 MB/sec execute 56 sec latency 9005.088 ms 1 3788 0.33 MB/sec execute 57 sec latency 10005.292 ms 1 3788 0.32 MB/sec execute 58 sec latency 11005.499 ms Failover mds1 to oleg221-server mount facets: mds1 1 3788 0.32 MB/sec execute 59 sec latency 12005.700 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3788 0.31 MB/sec execute 60 sec latency 13005.957 ms 1 3788 0.31 MB/sec execute 61 sec latency 14006.122 ms oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all 1 3788 0.30 MB/sec execute 62 sec latency 15006.302 ms pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 07:18:18 (1781695098) targets are mounted 07:18:18 (1781695098) facet_failover done 1 3788 0.30 MB/sec execute 63 sec latency 16006.469 ms 1 3788 0.29 MB/sec execute 64 sec latency 17006.621 ms 1 3788 0.29 MB/sec execute 65 sec latency 18006.825 ms 1 3788 0.28 MB/sec execute 66 sec latency 19006.961 ms 1 3792 0.28 MB/sec execute 67 sec latency 19958.723 ms 1 3939 0.28 MB/sec execute 68 sec latency 28.179 ms oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4085 0.28 MB/sec execute 69 sec latency 26.755 ms 1 4236 0.28 MB/sec execute 70 sec latency 142.386 ms 1 4350 0.27 MB/sec execute 71 sec latency 155.565 ms 1 4491 0.28 MB/sec execute 72 sec latency 57.527 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3968 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3515392 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3539968 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 75776 7055360 2% /mnt/lustre 1 4645 0.29 MB/sec execute 73 sec latency 56.079 ms 1 4810 0.29 MB/sec execute 74 sec latency 30.885 ms 1 4995 0.32 MB/sec execute 75 sec latency 294.922 ms 1 5150 0.32 MB/sec execute 76 sec latency 75.385 ms test_26 fail mds1 4 times Failing mds1 on oleg221-server Stopping /mnt/lustre-mds1 (opts:) on oleg221-server 1 5286 0.32 MB/sec execute 77 sec latency 75.272 ms 1 5335 0.31 MB/sec execute 78 sec latency 730.064 ms 07:18:34 (1781695114) shut down facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server 1 5335 0.31 MB/sec execute 79 sec latency 1730.223 ms 1 5335 0.31 MB/sec execute 80 sec latency 2730.408 ms 1 5335 0.30 MB/sec execute 81 sec latency 3730.605 ms 1 5335 0.30 MB/sec execute 82 sec latency 4730.861 ms 1 5335 0.29 MB/sec execute 83 sec latency 5731.014 ms 1 5335 0.29 MB/sec execute 84 sec latency 6731.162 ms 1 5335 0.29 MB/sec execute 85 sec latency 7731.296 ms 1 5335 0.28 MB/sec execute 86 sec latency 8731.458 ms 1 5335 0.28 MB/sec execute 87 sec latency 9731.646 ms 1 5335 0.28 MB/sec execute 88 sec latency 10731.772 ms Failover mds1 to oleg221-server mount facets: mds1 1 5335 0.27 MB/sec execute 89 sec latency 11731.912 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5335 0.27 MB/sec execute 90 sec latency 12732.020 ms 1 5335 0.27 MB/sec execute 91 sec latency 13732.156 ms oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 07:18:48 (1781695128) targets are mounted 07:18:48 (1781695128) facet_failover done 1 5335 0.27 MB/sec execute 92 sec latency 14732.294 ms 1 5335 0.26 MB/sec execute 93 sec latency 15732.426 ms 1 5335 0.26 MB/sec execute 94 sec latency 16732.690 ms 1 5335 0.26 MB/sec execute 95 sec latency 17732.940 ms 1 5335 0.25 MB/sec execute 96 sec latency 18733.125 ms 1 5335 0.25 MB/sec execute 97 sec latency 19733.276 ms 1 5335 0.25 MB/sec execute 98 sec latency 20733.522 ms 1 5358 0.25 MB/sec execute 99 sec latency 21467.483 ms 1 cleanup 100 sec oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 cleanup 101 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 859 64.629 21403.796 Close 641 2.743 9.277 Rename 37 26.022 41.974 Unlink 148 144.301 19958.699 Qpathinfo 766 33.408 21467.470 Qfileinfo 137 1.296 4.110 Qfsinfo 131 0.641 3.936 Sfileinfo 84 16.523 39.868 Find 285 3.366 103.087 WriteX 424 4.375 17.182 ReadX 1255 0.765 88.921 LockX 2 2.591 2.673 UnlockX 2 2.152 2.614 Flush 71 37.477 210.414 Throughput 0.247104 MB/sec 1 clients 1 procs max_latency=21467.483 ms stopping dbench on /mnt/lustre at Wed Jun 17 07:18:57 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Jun 17 07:18:59 EDT 2026 waiting for dbench pid 48814 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 129 5.76 MB/sec warmup 1 sec latency 46.784 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4352 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 44032 3636224 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 60416 3635200 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 104448 7271424 2% /mnt/lustre 1 207 4.05 MB/sec warmup 2 sec latency 62.875 ms 1 323 3.91 MB/sec warmup 3 sec latency 182.358 ms 1 462 4.04 MB/sec warmup 4 sec latency 33.596 ms 1 664 4.23 MB/sec warmup 5 sec latency 56.199 ms test_26 fail mds1 5 times Failing mds1 on oleg221-server 1 769 3.56 MB/sec warmup 6 sec latency 105.619 ms Stopping /mnt/lustre-mds1 (opts:) on oleg221-server 1 799 3.07 MB/sec warmup 7 sec latency 614.855 ms 07:19:07 (1781695147) shut down facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server 1 799 2.68 MB/sec warmup 8 sec latency 1615.066 ms 1 799 2.39 MB/sec warmup 9 sec latency 2615.227 ms 1 799 2.15 MB/sec warmup 10 sec latency 3615.401 ms 1 799 1.95 MB/sec warmup 11 sec latency 4615.572 ms 1 799 1.79 MB/sec warmup 12 sec latency 5615.705 ms 1 799 1.65 MB/sec warmup 13 sec latency 6615.867 ms 1 799 1.53 MB/sec warmup 14 sec latency 7616.020 ms 1 799 1.43 MB/sec warmup 15 sec latency 8616.145 ms 1 799 1.34 MB/sec warmup 16 sec latency 9616.374 ms 1 799 1.26 MB/sec warmup 17 sec latency 10616.516 ms Failover mds1 to oleg221-server mount facets: mds1 1 799 1.19 MB/sec warmup 18 sec latency 11616.678 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 799 1.13 MB/sec warmup 19 sec latency 12616.845 ms oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 1 799 0.00 MB/sec execute 1 sec latency 14617.060 ms Started lustre-MDT0000 07:19:20 (1781695160) targets are mounted 07:19:21 (1781695161) facet_failover done 1 799 0.00 MB/sec execute 2 sec latency 15617.198 ms 1 799 0.00 MB/sec execute 3 sec latency 16617.318 ms 1 799 0.00 MB/sec execute 4 sec latency 17617.450 ms 1 799 0.00 MB/sec execute 5 sec latency 18617.580 ms 1 905 0.03 MB/sec execute 6 sec latency 18646.214 ms oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1000 0.08 MB/sec execute 7 sec latency 64.771 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1099 0.18 MB/sec execute 8 sec latency 78.942 ms 1 1257 0.18 MB/sec execute 9 sec latency 35.635 ms 1 1481 0.46 MB/sec execute 10 sec latency 55.907 ms 1 1603 0.43 MB/sec execute 11 sec latency 81.550 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3747840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 45056 3691520 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 60416 7439360 1% /mnt/lustre 1 1741 0.40 MB/sec execute 12 sec latency 113.440 ms 1 1895 0.38 MB/sec execute 13 sec latency 52.936 ms 1 2080 0.37 MB/sec execute 14 sec latency 27.771 ms test_26 fail mds1 6 times 1 2282 0.38 MB/sec execute 15 sec latency 27.570 ms Failing mds1 on oleg221-server Stopping /mnt/lustre-mds1 (opts:) on oleg221-server 1 2387 0.41 MB/sec execute 16 sec latency 148.807 ms 07:19:36 (1781695176) shut down facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server 1 2387 0.38 MB/sec execute 17 sec latency 1119.025 ms 1 2387 0.36 MB/sec execute 18 sec latency 2119.198 ms 1 2387 0.34 MB/sec execute 19 sec latency 3119.335 ms 1 2387 0.32 MB/sec execute 20 sec latency 4119.494 ms 1 2387 0.31 MB/sec execute 21 sec latency 5119.743 ms 1 2387 0.29 MB/sec execute 22 sec latency 6119.919 ms 1 2387 0.28 MB/sec execute 23 sec latency 7120.053 ms 1 2387 0.27 MB/sec execute 24 sec latency 8120.192 ms 1 2387 0.26 MB/sec execute 25 sec latency 9120.334 ms 1 2387 0.25 MB/sec execute 26 sec latency 10120.478 ms Failover mds1 to oleg221-server mount facets: mds1 1 2387 0.24 MB/sec execute 27 sec latency 11120.587 ms 1 2387 0.23 MB/sec execute 28 sec latency 12120.760 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2387 0.22 MB/sec execute 29 sec latency 13120.890 ms oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all 1 2387 0.22 MB/sec execute 30 sec latency 14121.037 ms pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 07:19:50 (1781695190) targets are mounted 07:19:50 (1781695190) facet_failover done 1 2387 0.21 MB/sec execute 31 sec latency 15121.172 ms 1 2387 0.20 MB/sec execute 32 sec latency 16121.288 ms 1 2387 0.20 MB/sec execute 33 sec latency 17121.406 ms 1 2387 0.19 MB/sec execute 34 sec latency 18121.544 ms 1 2387 0.19 MB/sec execute 35 sec latency 19121.696 ms 1 2416 0.18 MB/sec execute 36 sec latency 19774.263 ms 1 2558 0.20 MB/sec execute 37 sec latency 239.671 ms oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2589 0.20 MB/sec execute 38 sec latency 736.285 ms 1 2902 0.29 MB/sec execute 39 sec latency 61.247 ms 1 3176 0.32 MB/sec execute 40 sec latency 64.319 ms 1 3372 0.34 MB/sec execute 41 sec latency 57.517 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3738624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 39936 3695616 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 52224 7434240 1% /mnt/lustre 1 3636 0.41 MB/sec execute 42 sec latency 27.561 ms 1 3750 0.42 MB/sec execute 43 sec latency 99.118 ms 1 3843 0.42 MB/sec execute 44 sec latency 46.001 ms 1 3944 0.41 MB/sec execute 45 sec latency 37.536 ms test_26 fail mds1 7 times Failing mds1 on oleg221-server Stopping /mnt/lustre-mds1 (opts:) on oleg221-server 1 4068 0.40 MB/sec execute 46 sec latency 37.953 ms 1 4110 0.40 MB/sec execute 47 sec latency 678.167 ms 07:20:07 (1781695207) shut down facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server 1 4110 0.39 MB/sec execute 48 sec latency 1678.291 ms 1 4110 0.38 MB/sec execute 49 sec latency 2678.420 ms 1 4110 0.37 MB/sec execute 50 sec latency 3678.564 ms 1 4110 0.36 MB/sec execute 51 sec latency 4678.703 ms 1 4110 0.36 MB/sec execute 52 sec latency 5678.851 ms 1 4110 0.35 MB/sec execute 53 sec latency 6679.028 ms 1 4110 0.34 MB/sec execute 54 sec latency 7679.207 ms 1 4110 0.34 MB/sec execute 55 sec latency 8679.371 ms 1 4110 0.33 MB/sec execute 56 sec latency 9679.508 ms 1 4110 0.33 MB/sec execute 57 sec latency 10679.687 ms Failover mds1 to oleg221-server mount facets: mds1 1 4110 0.32 MB/sec execute 58 sec latency 11679.865 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4110 0.32 MB/sec execute 59 sec latency 12679.984 ms 1 4110 0.31 MB/sec execute 60 sec latency 13680.137 ms oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 1 4110 0.31 MB/sec execute 61 sec latency 14680.281 ms Started lustre-MDT0000 07:20:20 (1781695220) targets are mounted 07:20:20 (1781695220) facet_failover done 1 4110 0.30 MB/sec execute 62 sec latency 15680.409 ms 1 4110 0.30 MB/sec execute 63 sec latency 16680.569 ms 1 4110 0.29 MB/sec execute 64 sec latency 17680.728 ms 1 4110 0.29 MB/sec execute 65 sec latency 18680.845 ms 1 4236 0.28 MB/sec execute 66 sec latency 18859.972 ms oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4360 0.29 MB/sec execute 67 sec latency 117.123 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4473 0.28 MB/sec execute 68 sec latency 81.527 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 4593 0.30 MB/sec execute 69 sec latency 53.301 ms 1 4792 0.30 MB/sec execute 70 sec latency 31.747 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Jun 17 07:20:30 EDT 2026 with return code 0 1 4979 0.33 MB/sec execute 71 sec latency 304.458 ms 48814 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48814 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished