== replay-dual test 26: dbench and tar with mds failover ========================================================== 20:35:42 (1786840542) Starting client oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre Started clients oleg360-client.virtnet: 192.168.203.160@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45069 Started dbench loop with 45070 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Aug 15 20:35:44 EDT 2026 waiting for dbench pid 45104 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 111 5.24 MB/sec warmup 1 sec latency 37.886 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 28672 3722240 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 30720 7457792 1% /mnt/lustre 1 207 4.05 MB/sec warmup 2 sec latency 81.196 ms 1 337 4.07 MB/sec warmup 3 sec latency 44.411 ms 1 468 4.11 MB/sec warmup 4 sec latency 34.156 ms test_26 fail mds1 1 times Failing mds1 on oleg360-server 1 654 4.21 MB/sec warmup 5 sec latency 39.363 ms Stopping /mnt/lustre-mds1 (opts:) on oleg360-server 1 695 3.53 MB/sec warmup 6 sec latency 585.010 ms 20:35:50 (1786840550) shut down facet: mds1 facet_host: oleg360-server facet_failover_host: oleg360-server 1 695 3.03 MB/sec warmup 7 sec latency 1585.144 ms 1 695 2.65 MB/sec warmup 8 sec latency 2585.306 ms 1 695 2.35 MB/sec warmup 9 sec latency 3585.459 ms 1 695 2.12 MB/sec warmup 10 sec latency 4585.633 ms 1 695 1.93 MB/sec warmup 11 sec latency 5585.858 ms 1 695 1.77 MB/sec warmup 12 sec latency 6586.020 ms 1 695 1.63 MB/sec warmup 13 sec latency 7586.192 ms 1 695 1.51 MB/sec warmup 14 sec latency 8586.362 ms 1 695 1.41 MB/sec warmup 15 sec latency 9586.531 ms 1 695 1.32 MB/sec warmup 16 sec latency 10586.680 ms Failover mds1 to oleg360-server mount facets: mds1 1 695 1.25 MB/sec warmup 17 sec latency 11586.899 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 695 1.18 MB/sec warmup 18 sec latency 12587.163 ms 1 695 1.12 MB/sec warmup 19 sec latency 13587.383 ms oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 20:36:04 (1786840564) targets are mounted 20:36:04 (1786840564) facet_failover done 1 695 0.00 MB/sec execute 1 sec latency 15588.148 ms 1 695 0.00 MB/sec execute 2 sec latency 16588.315 ms 1 695 0.00 MB/sec execute 3 sec latency 17588.459 ms 1 695 0.00 MB/sec execute 4 sec latency 18588.611 ms 1 695 0.00 MB/sec execute 5 sec latency 19588.737 ms 1 789 0.04 MB/sec execute 6 sec latency 19729.206 ms oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 923 0.08 MB/sec execute 7 sec latency 55.685 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1022 0.18 MB/sec execute 8 sec latency 77.114 ms 1 1199 0.21 MB/sec execute 9 sec latency 32.776 ms 1 1399 0.48 MB/sec execute 10 sec latency 37.488 ms 1 1544 0.45 MB/sec execute 11 sec latency 78.673 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3665920 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 44032 3585024 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7250944 1% /mnt/lustre 1 1643 0.42 MB/sec execute 12 sec latency 69.405 ms 1 1820 0.40 MB/sec execute 13 sec latency 41.818 ms 1 2000 0.38 MB/sec execute 14 sec latency 53.332 ms test_26 fail mds1 2 times Failing mds1 on oleg360-server 1 2161 0.37 MB/sec execute 15 sec latency 26.872 ms Stopping /mnt/lustre-mds1 (opts:) on oleg360-server 1 2273 0.38 MB/sec execute 16 sec latency 484.107 ms 20:36:20 (1786840580) shut down facet: mds1 facet_host: oleg360-server facet_failover_host: oleg360-server 1 2273 0.35 MB/sec execute 17 sec latency 1484.314 ms 1 2273 0.33 MB/sec execute 18 sec latency 2484.484 ms 1 2273 0.32 MB/sec execute 19 sec latency 3484.643 ms 1 2273 0.30 MB/sec execute 20 sec latency 4484.817 ms 1 2273 0.29 MB/sec execute 21 sec latency 5485.013 ms 1 2273 0.27 MB/sec execute 22 sec latency 6485.186 ms 1 2273 0.26 MB/sec execute 23 sec latency 7485.342 ms 1 2273 0.25 MB/sec execute 24 sec latency 8485.499 ms 1 2273 0.24 MB/sec execute 25 sec latency 9485.662 ms 1 2273 0.23 MB/sec execute 26 sec latency 10485.819 ms Failover mds1 to oleg360-server mount facets: mds1 1 2273 0.22 MB/sec execute 27 sec latency 11485.982 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2273 0.21 MB/sec execute 28 sec latency 12486.119 ms 1 2273 0.21 MB/sec execute 29 sec latency 13486.294 ms oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all 1 2273 0.20 MB/sec execute 30 sec latency 14486.476 ms pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 20:36:34 (1786840594) targets are mounted 20:36:34 (1786840594) facet_failover done 1 2273 0.19 MB/sec execute 31 sec latency 15486.660 ms 1 2273 0.19 MB/sec execute 32 sec latency 16486.813 ms 1 2273 0.18 MB/sec execute 33 sec latency 17486.974 ms 1 2273 0.18 MB/sec execute 34 sec latency 18487.143 ms 1 2273 0.17 MB/sec execute 35 sec latency 19487.279 ms 1 2273 0.17 MB/sec execute 36 sec latency 20487.403 ms 1 2408 0.18 MB/sec execute 37 sec latency 20573.874 ms oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 2566 0.20 MB/sec execute 38 sec latency 124.551 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2633 0.21 MB/sec execute 39 sec latency 707.617 ms 1 2969 0.31 MB/sec execute 40 sec latency 56.257 ms 1 3235 0.32 MB/sec execute 41 sec latency 51.465 ms 1 3506 0.41 MB/sec execute 42 sec latency 32.920 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3607552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 59392 3639296 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 75776 7246848 2% /mnt/lustre 1 3650 0.41 MB/sec execute 43 sec latency 29.099 ms 1 3792 0.42 MB/sec execute 44 sec latency 42.819 ms 1 3929 0.42 MB/sec execute 45 sec latency 28.752 ms test_26 fail mds1 3 times 1 4065 0.41 MB/sec execute 46 sec latency 31.281 ms Failing mds1 on oleg360-server Stopping /mnt/lustre-mds1 (opts:) on oleg360-server 1 4211 0.40 MB/sec execute 47 sec latency 35.986 ms 1 4214 0.40 MB/sec execute 48 sec latency 978.991 ms 20:36:52 (1786840612) shut down facet: mds1 facet_host: oleg360-server facet_failover_host: oleg360-server 1 4214 0.39 MB/sec execute 49 sec latency 1979.165 ms 1 4214 0.38 MB/sec execute 50 sec latency 2979.344 ms 1 4214 0.37 MB/sec execute 51 sec latency 3979.504 ms 1 4214 0.37 MB/sec execute 52 sec latency 4979.679 ms 1 4214 0.36 MB/sec execute 53 sec latency 5979.826 ms 1 4214 0.35 MB/sec execute 54 sec latency 6979.972 ms 1 4214 0.35 MB/sec execute 55 sec latency 7980.116 ms 1 4214 0.34 MB/sec execute 56 sec latency 8980.289 ms 1 4214 0.33 MB/sec execute 57 sec latency 9980.486 ms 1 4214 0.33 MB/sec execute 58 sec latency 10980.647 ms Failover mds1 to oleg360-server mount facets: mds1 1 4214 0.32 MB/sec execute 59 sec latency 11980.814 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4214 0.32 MB/sec execute 60 sec latency 12980.976 ms 1 4214 0.31 MB/sec execute 61 sec latency 13981.124 ms oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all 1 4214 0.31 MB/sec execute 62 sec latency 14981.257 ms pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 20:37:06 (1786840626) targets are mounted 20:37:06 (1786840626) facet_failover done 1 4214 0.30 MB/sec execute 63 sec latency 15981.446 ms 1 4214 0.30 MB/sec execute 64 sec latency 16981.610 ms 1 4214 0.29 MB/sec execute 65 sec latency 17981.740 ms 1 4214 0.29 MB/sec execute 66 sec latency 18981.874 ms 1 4292 0.29 MB/sec execute 67 sec latency 19285.477 ms oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 4391 0.29 MB/sec execute 68 sec latency 130.708 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4528 0.29 MB/sec execute 69 sec latency 73.083 ms 1 4701 0.30 MB/sec execute 70 sec latency 61.041 ms 1 4858 0.30 MB/sec execute 71 sec latency 30.424 ms 1 5047 0.33 MB/sec execute 72 sec latency 274.713 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4352 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3710976 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 60416 3648512 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 80896 7359488 2% /mnt/lustre 1 5174 0.33 MB/sec execute 73 sec latency 99.814 ms 1 5326 0.33 MB/sec execute 74 sec latency 55.876 ms 1 5453 0.32 MB/sec execute 75 sec latency 72.894 ms test_26 fail mds1 4 times 1 5657 0.32 MB/sec execute 76 sec latency 57.525 ms Failing mds1 on oleg360-server Stopping /mnt/lustre-mds1 (opts:) on oleg360-server 1 5771 0.32 MB/sec execute 77 sec latency 318.264 ms 20:37:22 (1786840642) shut down facet: mds1 facet_host: oleg360-server facet_failover_host: oleg360-server 1 5771 0.32 MB/sec execute 78 sec latency 1318.436 ms 1 5771 0.31 MB/sec execute 79 sec latency 2318.579 ms 1 5771 0.31 MB/sec execute 80 sec latency 3318.723 ms 1 5771 0.30 MB/sec execute 81 sec latency 4318.893 ms 1 5771 0.30 MB/sec execute 82 sec latency 5319.046 ms 1 5771 0.30 MB/sec execute 83 sec latency 6319.185 ms 1 5771 0.29 MB/sec execute 84 sec latency 7319.352 ms 1 5771 0.29 MB/sec execute 85 sec latency 8320.574 ms 1 5771 0.29 MB/sec execute 86 sec latency 9320.717 ms 1 5771 0.28 MB/sec execute 87 sec latency 10320.834 ms Failover mds1 to oleg360-server mount facets: mds1 1 5771 0.28 MB/sec execute 88 sec latency 11321.047 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5771 0.28 MB/sec execute 89 sec latency 12321.168 ms 1 5771 0.27 MB/sec execute 90 sec latency 13321.275 ms oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all 1 5771 0.27 MB/sec execute 91 sec latency 14321.424 ms pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 20:37:35 (1786840655) targets are mounted 20:37:35 (1786840655) facet_failover done 1 5771 0.27 MB/sec execute 92 sec latency 15321.590 ms 1 5771 0.27 MB/sec execute 93 sec latency 16321.767 ms 1 5771 0.26 MB/sec execute 94 sec latency 17321.957 ms 1 5771 0.26 MB/sec execute 95 sec latency 18322.162 ms 1 5771 0.26 MB/sec execute 96 sec latency 19322.290 ms 1 5890 0.27 MB/sec execute 97 sec latency 19761.115 ms oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 5996 0.27 MB/sec execute 98 sec latency 89.290 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6166 0.27 MB/sec execute 99 sec latency 53.509 ms 1 cleanup 100 sec 1 cleanup 101 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 981 97.194 20573.863 Close 759 2.697 7.181 Rename 44 25.978 47.139 Unlink 166 10.319 35.958 Qpathinfo 927 5.208 152.940 Qfileinfo 173 1.133 6.050 Qfsinfo 148 0.703 13.309 Sfileinfo 97 16.192 39.388 Find 335 2.609 29.175 WriteX 523 4.043 19.629 ReadX 1549 0.469 77.099 LockX 4 2.707 3.087 UnlockX 4 2.452 2.924 Flush 73 40.376 130.695 Throughput 0.272004 MB/sec 1 clients 1 procs max_latency=20573.874 ms stopping dbench on /mnt/lustre at Sat Aug 15 20:37:45 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4480 2204032 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3699712 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 59392 3653632 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 79872 7353344 2% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Aug 15 20:37:47 EDT 2026 waiting for dbench pid 49164 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 66 2.88 MB/sec warmup 1 sec latency 31.781 ms 1 206 4.05 MB/sec warmup 2 sec latency 39.846 ms test_26 fail mds1 5 times 1 357 4.26 MB/sec warmup 3 sec latency 31.180 ms Failing mds1 on oleg360-server Stopping /mnt/lustre-mds1 (opts:) on oleg360-server 1 503 4.37 MB/sec warmup 4 sec latency 29.567 ms 20:37:52 (1786840672) shut down 1 513 3.55 MB/sec warmup 5 sec latency 933.844 ms facet: mds1 facet_host: oleg360-server facet_failover_host: oleg360-server 1 513 2.95 MB/sec warmup 6 sec latency 1934.052 ms 1 513 2.53 MB/sec warmup 7 sec latency 2934.279 ms 1 513 2.22 MB/sec warmup 8 sec latency 3934.488 ms 1 513 1.97 MB/sec warmup 9 sec latency 4934.635 ms 1 513 1.77 MB/sec warmup 10 sec latency 5934.855 ms 1 513 1.61 MB/sec warmup 11 sec latency 6935.067 ms 1 513 1.48 MB/sec warmup 12 sec latency 7935.294 ms 1 513 1.36 MB/sec warmup 13 sec latency 8935.531 ms 1 513 1.27 MB/sec warmup 14 sec latency 9935.761 ms 1 513 1.18 MB/sec warmup 15 sec latency 10935.981 ms Failover mds1 to oleg360-server mount facets: mds1 1 513 1.11 MB/sec warmup 16 sec latency 11936.125 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 513 1.04 MB/sec warmup 17 sec latency 12936.283 ms 1 513 0.98 MB/sec warmup 18 sec latency 13936.439 ms oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 20:38:06 (1786840686) targets are mounted 20:38:06 (1786840686) facet_failover done 1 513 0.93 MB/sec warmup 19 sec latency 14936.586 ms 1 513 0.00 MB/sec execute 1 sec latency 16936.851 ms 1 513 0.00 MB/sec execute 2 sec latency 17936.995 ms 1 513 0.00 MB/sec execute 3 sec latency 18937.111 ms 1 582 0.52 MB/sec execute 4 sec latency 19642.744 ms 1 711 0.70 MB/sec execute 5 sec latency 251.724 ms oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 827 0.63 MB/sec execute 6 sec latency 50.604 ms 1 955 0.58 MB/sec execute 7 sec latency 64.135 ms 1 1124 0.66 MB/sec execute 8 sec latency 81.505 ms 1 1270 0.60 MB/sec execute 9 sec latency 28.309 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3708928 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3691520 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7400448 1% /mnt/lustre 1 1486 0.83 MB/sec execute 10 sec latency 203.868 ms 1 1613 0.77 MB/sec execute 11 sec latency 115.364 ms 1 1791 0.72 MB/sec execute 12 sec latency 42.414 ms test_26 fail mds1 6 times 1 1982 0.67 MB/sec execute 13 sec latency 68.283 ms Failing mds1 on oleg360-server Stopping /mnt/lustre-mds1 (opts:) on oleg360-server 1 2158 0.64 MB/sec execute 14 sec latency 29.638 ms 1 2168 0.60 MB/sec execute 15 sec latency 902.868 ms 20:38:22 (1786840702) shut down facet: mds1 facet_host: oleg360-server facet_failover_host: oleg360-server 1 2168 0.56 MB/sec execute 16 sec latency 1903.019 ms 1 2168 0.53 MB/sec execute 17 sec latency 2903.170 ms 1 2168 0.50 MB/sec execute 18 sec latency 3903.290 ms 1 2168 0.47 MB/sec execute 19 sec latency 4903.394 ms 1 2168 0.45 MB/sec execute 20 sec latency 5903.556 ms 1 2168 0.43 MB/sec execute 21 sec latency 6903.708 ms 1 2168 0.41 MB/sec execute 22 sec latency 7903.818 ms 1 2168 0.39 MB/sec execute 23 sec latency 8903.968 ms 1 2168 0.38 MB/sec execute 24 sec latency 9904.137 ms 1 2168 0.36 MB/sec execute 25 sec latency 10904.295 ms Failover mds1 to oleg360-server mount facets: mds1 1 2168 0.35 MB/sec execute 26 sec latency 11904.508 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2168 0.33 MB/sec execute 27 sec latency 12904.678 ms 1 2168 0.32 MB/sec execute 28 sec latency 13904.822 ms oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 20:38:36 (1786840716) targets are mounted 20:38:36 (1786840716) facet_failover done 1 2168 0.31 MB/sec execute 29 sec latency 14904.974 ms 1 2168 0.30 MB/sec execute 30 sec latency 15905.094 ms 1 2168 0.29 MB/sec execute 31 sec latency 16905.253 ms 1 2168 0.28 MB/sec execute 32 sec latency 17905.465 ms 1 2168 0.27 MB/sec execute 33 sec latency 18905.663 ms 1 2295 0.30 MB/sec execute 34 sec latency 19333.169 ms oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 2426 0.30 MB/sec execute 35 sec latency 90.050 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2566 0.31 MB/sec execute 36 sec latency 168.696 ms 1 2566 0.30 MB/sec execute 37 sec latency 1184.904 ms 1 2776 0.37 MB/sec execute 38 sec latency 1283.022 ms 1 3004 0.41 MB/sec execute 39 sec latency 57.239 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3703808 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 38912 3695616 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 62464 7399424 1% /mnt/lustre 1 3279 0.44 MB/sec execute 40 sec latency 87.014 ms 1 3523 0.50 MB/sec execute 41 sec latency 43.439 ms 1 3685 0.50 MB/sec execute 42 sec latency 29.068 ms 1 3807 0.51 MB/sec execute 43 sec latency 40.463 ms test_26 fail mds1 7 times Failing mds1 on oleg360-server 1 3907 0.51 MB/sec execute 44 sec latency 34.822 ms Stopping /mnt/lustre-mds1 (opts:) on oleg360-server 1 3948 0.50 MB/sec execute 45 sec latency 588.425 ms 20:38:53 (1786840733) shut down facet: mds1 facet_host: oleg360-server facet_failover_host: oleg360-server 1 3948 0.49 MB/sec execute 46 sec latency 1588.607 ms 1 3948 0.48 MB/sec execute 47 sec latency 2588.809 ms 1 3948 0.47 MB/sec execute 48 sec latency 3589.023 ms 1 3948 0.46 MB/sec execute 49 sec latency 4589.191 ms 1 3948 0.45 MB/sec execute 50 sec latency 5589.333 ms 1 3948 0.44 MB/sec execute 51 sec latency 6589.521 ms 1 3948 0.43 MB/sec execute 52 sec latency 7589.753 ms 1 3948 0.42 MB/sec execute 53 sec latency 8589.940 ms 1 3948 0.41 MB/sec execute 54 sec latency 9590.108 ms 1 3948 0.41 MB/sec execute 55 sec latency 10590.291 ms Failover mds1 to oleg360-server mount facets: mds1 1 3948 0.40 MB/sec execute 56 sec latency 11590.474 ms 1 3948 0.39 MB/sec execute 57 sec latency 12590.605 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3948 0.38 MB/sec execute 58 sec latency 13590.990 ms 1 3948 0.38 MB/sec execute 59 sec latency 14591.181 ms oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 20:39:07 (1786840747) targets are mounted 20:39:07 (1786840747) facet_failover done 1 3948 0.37 MB/sec execute 60 sec latency 15591.355 ms 1 3948 0.37 MB/sec execute 61 sec latency 16592.006 ms 1 3948 0.36 MB/sec execute 62 sec latency 17592.208 ms 1 3948 0.35 MB/sec execute 63 sec latency 18592.659 ms 1 3958 0.35 MB/sec execute 64 sec latency 19517.101 ms 1 4074 0.34 MB/sec execute 65 sec latency 33.420 ms oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4211 0.34 MB/sec execute 66 sec latency 28.126 ms 1 4301 0.34 MB/sec execute 67 sec latency 256.278 ms 1 4435 0.34 MB/sec execute 68 sec latency 55.046 ms 1 4549 0.34 MB/sec execute 69 sec latency 67.819 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 4729 0.35 MB/sec execute 70 sec latency 42.124 ms 1 4896 0.36 MB/sec execute 71 sec latency 174.469 ms 1 5052 0.38 MB/sec execute 72 sec latency 242.044 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sat Aug 15 20:39:20 EDT 2026 with return code 0 49164 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 49164 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished