== replay-dual test 26: dbench and tar with mds failover ========================================================== 07:14:19 (1781608459) Starting client oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre Started clients oleg654-client.virtnet: 192.168.206.154@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 46346 Started dbench loop with 46347 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 16 07:14:21 EDT 2026 waiting for dbench pid 46378 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 129 5.76 MB/sec warmup 1 sec latency 53.618 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1736 1269568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25984 3579776 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27396 7185384 1% /mnt/lustre 1 207 4.05 MB/sec warmup 2 sec latency 152.127 ms 1 320 3.87 MB/sec warmup 3 sec latency 39.922 ms 1 421 3.72 MB/sec warmup 4 sec latency 54.884 ms 1 533 3.72 MB/sec warmup 5 sec latency 45.818 ms test_26 fail mds1 1 times 1 702 3.53 MB/sec warmup 6 sec latency 48.864 ms Failing mds1 on oleg654-server Stopping /mnt/lustre-mds1 (opts:) on oleg654-server 1 791 3.07 MB/sec warmup 7 sec latency 56.016 ms 1 817 2.68 MB/sec warmup 8 sec latency 705.452 ms 07:14:29 (1781608469) shut down facet: mds1 facet_host: oleg654-server facet_failover_host: oleg654-server 1 817 2.39 MB/sec warmup 9 sec latency 1705.640 ms 1 817 2.15 MB/sec warmup 10 sec latency 2706.263 ms 1 817 1.95 MB/sec warmup 11 sec latency 3706.606 ms 1 817 1.79 MB/sec warmup 12 sec latency 4707.706 ms 1 817 1.65 MB/sec warmup 13 sec latency 5708.118 ms 1 817 1.53 MB/sec warmup 14 sec latency 6708.347 ms 1 817 1.43 MB/sec warmup 15 sec latency 7708.533 ms 1 817 1.34 MB/sec warmup 16 sec latency 8708.694 ms 1 817 1.26 MB/sec warmup 17 sec latency 9709.427 ms 1 817 1.19 MB/sec warmup 18 sec latency 10709.592 ms Failover mds1 to oleg654-server mount facets: mds1 1 817 1.13 MB/sec warmup 19 sec latency 11709.752 ms 1 817 0.00 MB/sec execute 1 sec latency 13710.170 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 817 0.00 MB/sec execute 2 sec latency 14711.451 ms 1 817 0.00 MB/sec execute 3 sec latency 15711.600 ms 1 817 0.00 MB/sec execute 4 sec latency 16711.723 ms oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 07:14:46 (1781608486) targets are mounted 07:14:46 (1781608486) facet_failover done 1 817 0.00 MB/sec execute 5 sec latency 17712.096 ms 1 872 0.03 MB/sec execute 6 sec latency 18299.653 ms 1 896 0.03 MB/sec execute 7 sec latency 636.647 ms oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 988 0.05 MB/sec execute 8 sec latency 38.303 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1114 0.17 MB/sec execute 9 sec latency 80.031 ms 1 1229 0.16 MB/sec execute 10 sec latency 36.716 ms 1 1419 0.41 MB/sec execute 11 sec latency 43.628 ms 1 1493 0.38 MB/sec execute 12 sec latency 259.379 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1976 1269328 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 17128 3575648 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 32864 3564440 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49992 7140088 1% /mnt/lustre 1 1570 0.36 MB/sec execute 13 sec latency 71.323 ms 1 1676 0.34 MB/sec execute 14 sec latency 47.412 ms 1 1821 0.32 MB/sec execute 15 sec latency 37.308 ms 1 1969 0.31 MB/sec execute 16 sec latency 58.755 ms 1 2095 0.30 MB/sec execute 17 sec latency 30.727 ms test_26 fail mds1 2 times 1 2237 0.29 MB/sec execute 18 sec latency 38.203 ms Failing mds1 on oleg654-server Stopping /mnt/lustre-mds1 (opts:) on oleg654-server 1 2394 0.34 MB/sec execute 19 sec latency 45.894 ms 1 2412 0.33 MB/sec execute 20 sec latency 848.413 ms 07:15:01 (1781608501) shut down facet: mds1 facet_host: oleg654-server facet_failover_host: oleg654-server 1 2412 0.31 MB/sec execute 21 sec latency 1848.583 ms 1 2412 0.30 MB/sec execute 22 sec latency 2850.234 ms 1 2412 0.28 MB/sec execute 23 sec latency 3850.426 ms 1 2412 0.27 MB/sec execute 24 sec latency 4850.689 ms 1 2412 0.26 MB/sec execute 25 sec latency 5851.252 ms 1 2412 0.25 MB/sec execute 26 sec latency 6851.427 ms 1 2412 0.24 MB/sec execute 27 sec latency 7851.754 ms 1 2412 0.23 MB/sec execute 28 sec latency 8852.084 ms 1 2412 0.23 MB/sec execute 29 sec latency 9852.262 ms 1 2412 0.22 MB/sec execute 30 sec latency 10852.783 ms Failover mds1 to oleg654-server mount facets: mds1 1 2412 0.21 MB/sec execute 31 sec latency 11852.899 ms 1 2412 0.20 MB/sec execute 32 sec latency 12853.057 ms 1 2412 0.20 MB/sec execute 33 sec latency 13853.345 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 2412 0.19 MB/sec execute 34 sec latency 14853.484 ms 1 2412 0.19 MB/sec execute 35 sec latency 15853.626 ms oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all 1 2412 0.18 MB/sec execute 36 sec latency 16853.790 ms pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 07:15:17 (1781608517) targets are mounted 07:15:17 (1781608517) facet_failover done 1 2412 0.18 MB/sec execute 37 sec latency 17853.958 ms 1 2412 0.17 MB/sec execute 38 sec latency 18854.130 ms 1 2412 0.17 MB/sec execute 39 sec latency 19854.324 ms 1 2412 0.16 MB/sec execute 40 sec latency 20854.472 ms 1 2428 0.16 MB/sec execute 41 sec latency 21659.227 ms 1 2605 0.18 MB/sec execute 42 sec latency 44.023 ms oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2932 0.28 MB/sec execute 43 sec latency 39.148 ms 1 3246 0.30 MB/sec execute 44 sec latency 33.185 ms 1 3524 0.38 MB/sec execute 45 sec latency 28.686 ms 1 3691 0.38 MB/sec execute 46 sec latency 23.831 ms 1 3840 0.39 MB/sec execute 47 sec latency 39.863 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2376 1268928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 31128 3571076 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 39092 3563452 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 70220 7134528 1% /mnt/lustre 1 3955 0.39 MB/sec execute 48 sec latency 71.714 ms 1 4102 0.38 MB/sec execute 49 sec latency 30.437 ms 1 4250 0.38 MB/sec execute 50 sec latency 80.006 ms 1 4372 0.37 MB/sec execute 51 sec latency 47.713 ms test_26 fail mds1 3 times 1 4503 0.37 MB/sec execute 52 sec latency 39.220 ms Failing mds1 on oleg654-server Stopping /mnt/lustre-mds1 (opts:) on oleg654-server 1 4619 0.39 MB/sec execute 53 sec latency 183.556 ms 07:15:35 (1781608535) shut down facet: mds1 facet_host: oleg654-server facet_failover_host: oleg654-server 1 4619 0.38 MB/sec execute 54 sec latency 1183.767 ms 1 4619 0.37 MB/sec execute 55 sec latency 2183.983 ms 1 4619 0.37 MB/sec execute 56 sec latency 3184.313 ms 1 4619 0.36 MB/sec execute 57 sec latency 4184.470 ms 1 4619 0.35 MB/sec execute 58 sec latency 5184.755 ms 1 4619 0.35 MB/sec execute 59 sec latency 6185.164 ms 1 4619 0.34 MB/sec execute 60 sec latency 7185.453 ms 1 4619 0.34 MB/sec execute 61 sec latency 8185.618 ms 1 4619 0.33 MB/sec execute 62 sec latency 9185.780 ms 1 4619 0.33 MB/sec execute 63 sec latency 10185.950 ms Failover mds1 to oleg654-server mount facets: mds1 1 4619 0.32 MB/sec execute 64 sec latency 11186.056 ms 1 4619 0.32 MB/sec execute 65 sec latency 12186.222 ms 1 4619 0.31 MB/sec execute 66 sec latency 13186.385 ms 1 4619 0.31 MB/sec execute 67 sec latency 14187.189 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4619 0.30 MB/sec execute 68 sec latency 15187.315 ms 1 4619 0.30 MB/sec execute 69 sec latency 16187.464 ms oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 1 4619 0.29 MB/sec execute 70 sec latency 17187.584 ms Started lustre-MDT0000 07:15:51 (1781608551) targets are mounted 07:15:51 (1781608551) facet_failover done 1 4619 0.29 MB/sec execute 71 sec latency 18187.799 ms 1 4619 0.29 MB/sec execute 72 sec latency 19187.921 ms 1 4619 0.28 MB/sec execute 73 sec latency 20188.068 ms 1 4758 0.28 MB/sec execute 74 sec latency 20267.990 ms oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4953 0.32 MB/sec execute 75 sec latency 38.922 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5068 0.31 MB/sec execute 76 sec latency 44.471 ms 1 5186 0.31 MB/sec execute 77 sec latency 37.107 ms 1 5305 0.31 MB/sec execute 78 sec latency 35.181 ms 1 5426 0.30 MB/sec execute 79 sec latency 33.686 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2848 1268456 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 32568 3574068 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 40608 3566000 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 73176 7140068 2% /mnt/lustre 1 5571 0.30 MB/sec execute 80 sec latency 49.750 ms 1 5724 0.30 MB/sec execute 81 sec latency 33.545 ms 1 5907 0.31 MB/sec execute 82 sec latency 52.752 ms 1 6026 0.32 MB/sec execute 83 sec latency 145.395 ms 1 6308 0.35 MB/sec execute 84 sec latency 36.431 ms test_26 fail mds1 4 times Failing mds1 on oleg654-server 1 6542 0.37 MB/sec execute 85 sec latency 56.022 ms Stopping /mnt/lustre-mds1 (opts:) on oleg654-server 1 6741 0.37 MB/sec execute 86 sec latency 511.169 ms 07:16:07 (1781608567) shut down facet: mds1 facet_host: oleg654-server facet_failover_host: oleg654-server 1 6741 0.37 MB/sec execute 87 sec latency 1511.348 ms 1 6741 0.36 MB/sec execute 88 sec latency 2511.560 ms 1 6741 0.36 MB/sec execute 89 sec latency 3511.737 ms 1 6741 0.35 MB/sec execute 90 sec latency 4511.899 ms 1 6741 0.35 MB/sec execute 91 sec latency 5512.061 ms 1 6741 0.35 MB/sec execute 92 sec latency 6512.267 ms 1 6741 0.34 MB/sec execute 93 sec latency 7512.433 ms 1 6741 0.34 MB/sec execute 94 sec latency 8512.588 ms 1 6741 0.34 MB/sec execute 95 sec latency 9512.754 ms 1 6741 0.33 MB/sec execute 96 sec latency 10512.922 ms Failover mds1 to oleg654-server mount facets: mds1 1 6741 0.33 MB/sec execute 97 sec latency 11513.107 ms 1 6741 0.33 MB/sec execute 98 sec latency 12513.272 ms 1 6741 0.32 MB/sec execute 99 sec latency 13513.419 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 cleanup 100 sec 1 cleanup 101 sec oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all 1 cleanup 102 sec pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 07:16:23 (1781608583) targets are mounted 07:16:23 (1781608583) facet_failover done 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 0 cleanup 109 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 982 58.099 20267.980 Close 756 31.942 21548.885 Rename 45 28.375 44.007 Unlink 167 9.966 29.400 Qpathinfo 950 5.646 37.953 Qfileinfo 178 1.156 6.672 Qfsinfo 146 0.579 4.013 Sfileinfo 91 256.810 21659.215 Find 337 2.546 49.972 WriteX 513 4.746 37.950 ReadX 1689 0.533 68.823 LockX 4 4.210 5.786 UnlockX 4 3.124 4.187 Flush 63 22.535 79.993 Throughput 0.322047 MB/sec 1 clients 1 procs max_latency=21659.227 ms stopping dbench on /mnt/lustre at Tue Jun 16 07:16:29 EDT 2026 with return code 0 oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 16 07:16:31 EDT 2026 waiting for dbench pid 50592 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 137 6.02 MB/sec warmup 1 sec latency 37.544 ms 1 199 3.92 MB/sec warmup 2 sec latency 184.859 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2600 1268704 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 22496 3584396 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 29124 3577268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 51620 7161664 1% /mnt/lustre 1 315 3.83 MB/sec warmup 3 sec latency 101.526 ms 1 440 3.85 MB/sec warmup 4 sec latency 49.069 ms 1 642 4.21 MB/sec warmup 5 sec latency 35.927 ms 1 796 3.58 MB/sec warmup 6 sec latency 58.956 ms 1 884 3.10 MB/sec warmup 7 sec latency 507.987 ms test_26 fail mds1 5 times Failing mds1 on oleg654-server 1 969 2.73 MB/sec warmup 8 sec latency 722.585 ms Stopping /mnt/lustre-mds1 (opts:) on oleg654-server 1 1076 2.55 MB/sec warmup 9 sec latency 428.524 ms 07:16:41 (1781608601) shut down facet: mds1 facet_host: oleg654-server facet_failover_host: oleg654-server 1 1076 2.29 MB/sec warmup 10 sec latency 1428.693 ms 1 1076 2.08 MB/sec warmup 11 sec latency 2428.880 ms 1 1076 1.91 MB/sec warmup 12 sec latency 3429.058 ms 1 1076 1.76 MB/sec warmup 13 sec latency 4429.230 ms 1 1076 1.64 MB/sec warmup 14 sec latency 5429.382 ms 1 1076 1.53 MB/sec warmup 15 sec latency 6429.577 ms 1 1076 1.43 MB/sec warmup 16 sec latency 7429.732 ms 1 1076 1.35 MB/sec warmup 17 sec latency 8429.897 ms 1 1076 1.27 MB/sec warmup 18 sec latency 9430.049 ms 1 1076 1.21 MB/sec warmup 19 sec latency 10430.207 ms Failover mds1 to oleg654-server mount facets: mds1 1 1076 0.00 MB/sec execute 1 sec latency 12430.923 ms 1 1076 0.00 MB/sec execute 2 sec latency 13431.086 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1076 0.00 MB/sec execute 3 sec latency 14431.222 ms 1 1076 0.00 MB/sec execute 4 sec latency 15431.383 ms oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 1 1076 0.00 MB/sec execute 5 sec latency 16431.513 ms Started lustre-MDT0000 07:16:57 (1781608617) targets are mounted 07:16:57 (1781608617) facet_failover done 1 1076 0.00 MB/sec execute 6 sec latency 17431.637 ms 1 1173 0.01 MB/sec execute 7 sec latency 17740.134 ms 1 1296 0.07 MB/sec execute 8 sec latency 40.711 ms oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1461 0.35 MB/sec execute 9 sec latency 82.075 ms 1 1543 0.32 MB/sec execute 10 sec latency 169.697 ms 1 1635 0.30 MB/sec execute 11 sec latency 75.959 ms 1 1805 0.28 MB/sec execute 12 sec latency 41.469 ms 1 1965 0.27 MB/sec execute 13 sec latency 74.881 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1269412 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 4844 3594632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 31432 3568760 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 36276 7163392 1% /mnt/lustre 1 2045 0.26 MB/sec execute 14 sec latency 64.319 ms 1 2168 0.25 MB/sec execute 15 sec latency 51.700 ms 1 2367 0.31 MB/sec execute 16 sec latency 45.822 ms 1 2487 0.34 MB/sec execute 17 sec latency 36.839 ms test_26 fail mds1 6 times 1 2779 0.50 MB/sec execute 18 sec latency 35.745 ms Failing mds1 on oleg654-server Stopping /mnt/lustre-mds1 (opts:) on oleg654-server 1 2987 0.56 MB/sec execute 19 sec latency 137.433 ms 07:17:11 (1781608631) shut down facet: mds1 facet_host: oleg654-server facet_failover_host: oleg654-server 1 2987 0.53 MB/sec execute 20 sec latency 1137.563 ms 1 2987 0.50 MB/sec execute 21 sec latency 2137.771 ms 1 2987 0.48 MB/sec execute 22 sec latency 3137.958 ms 1 2987 0.46 MB/sec execute 23 sec latency 4138.118 ms 1 2987 0.44 MB/sec execute 24 sec latency 5138.288 ms 1 2987 0.42 MB/sec execute 25 sec latency 6138.443 ms 1 2987 0.41 MB/sec execute 26 sec latency 7138.615 ms 1 2987 0.39 MB/sec execute 27 sec latency 8138.785 ms 1 2987 0.38 MB/sec execute 28 sec latency 9138.952 ms 1 2987 0.37 MB/sec execute 29 sec latency 10139.097 ms Failover mds1 to oleg654-server mount facets: mds1 1 2987 0.35 MB/sec execute 30 sec latency 11139.177 ms 1 2987 0.34 MB/sec execute 31 sec latency 12139.306 ms 1 2987 0.33 MB/sec execute 32 sec latency 13139.422 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 2987 0.32 MB/sec execute 33 sec latency 14139.688 ms 1 2987 0.31 MB/sec execute 34 sec latency 15139.858 ms oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all 1 2987 0.30 MB/sec execute 35 sec latency 16140.013 ms pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 07:17:27 (1781608647) targets are mounted 07:17:27 (1781608647) facet_failover done 1 2987 0.29 MB/sec execute 36 sec latency 17140.146 ms 1 2990 0.29 MB/sec execute 37 sec latency 18024.523 ms 1 3354 0.33 MB/sec execute 38 sec latency 45.931 ms oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3644 0.40 MB/sec execute 39 sec latency 25.371 ms 1 3817 0.42 MB/sec execute 40 sec latency 28.757 ms 1 3948 0.42 MB/sec execute 41 sec latency 38.231 ms 1 4079 0.41 MB/sec execute 42 sec latency 29.760 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2100 1269204 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 30916 3562532 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37792 3567740 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 68708 7130272 1% /mnt/lustre 1 4211 0.40 MB/sec execute 43 sec latency 76.737 ms 1 4338 0.40 MB/sec execute 44 sec latency 52.622 ms 1 4445 0.40 MB/sec execute 45 sec latency 298.806 ms 1 4569 0.40 MB/sec execute 46 sec latency 30.693 ms 1 4773 0.41 MB/sec execute 47 sec latency 22.569 ms test_26 fail mds1 7 times Failing mds1 on oleg654-server 1 4988 0.46 MB/sec execute 48 sec latency 31.805 ms Stopping /mnt/lustre-mds1 (opts:) on oleg654-server 1 5063 0.45 MB/sec execute 49 sec latency 406.919 ms 07:17:41 (1781608661) shut down facet: mds1 facet_host: oleg654-server facet_failover_host: oleg654-server 1 5063 0.44 MB/sec execute 50 sec latency 1407.047 ms 1 5063 0.44 MB/sec execute 51 sec latency 2407.205 ms 1 5063 0.43 MB/sec execute 52 sec latency 3407.370 ms 1 5063 0.42 MB/sec execute 53 sec latency 4407.676 ms 1 5063 0.41 MB/sec execute 54 sec latency 5407.861 ms 1 5063 0.40 MB/sec execute 55 sec latency 6408.035 ms 1 5063 0.40 MB/sec execute 56 sec latency 7408.183 ms 1 5063 0.39 MB/sec execute 57 sec latency 8408.357 ms 1 5063 0.38 MB/sec execute 58 sec latency 9408.535 ms 1 5063 0.38 MB/sec execute 59 sec latency 10408.711 ms Failover mds1 to oleg654-server mount facets: mds1 1 5063 0.37 MB/sec execute 60 sec latency 11408.872 ms 1 5063 0.36 MB/sec execute 61 sec latency 12409.029 ms 1 5063 0.36 MB/sec execute 62 sec latency 13409.136 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5063 0.35 MB/sec execute 63 sec latency 14409.287 ms 1 5063 0.35 MB/sec execute 64 sec latency 15410.004 ms oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 1 5063 0.34 MB/sec execute 65 sec latency 16410.176 ms Started lustre-MDT0000 07:17:57 (1781608677) targets are mounted 07:17:57 (1781608677) facet_failover done 1 5063 0.34 MB/sec execute 66 sec latency 17410.386 ms 1 5063 0.33 MB/sec execute 67 sec latency 18410.503 ms 1 5063 0.33 MB/sec execute 68 sec latency 19410.669 ms 1 5168 0.32 MB/sec execute 69 sec latency 19470.828 ms oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5286 0.32 MB/sec execute 70 sec latency 40.271 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5397 0.32 MB/sec execute 71 sec latency 36.182 ms 1 5547 0.31 MB/sec execute 72 sec latency 40.106 ms 1 5720 0.31 MB/sec execute 73 sec latency 29.428 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 5907 0.33 MB/sec execute 74 sec latency 85.452 ms 1 6014 0.32 MB/sec execute 75 sec latency 38.523 ms 1 6315 0.37 MB/sec execute 76 sec latency 60.724 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Jun 16 07:18:08 EDT 2026 with return code 0 50592 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 50592 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished