== replay-dual test 26: dbench and tar with mds failover ========================================================== 18:02:08 (1756677728) Starting client oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre Started clients oleg108-client.virtnet: 192.168.201.108@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44408 Started dbench loop with 44409 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' UUID 1K-blocks Used Available Use% Mounted on running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Aug 31 18:02:10 EDT 2025 lustre-MDT0000_UUID 2210688 3456 2205184 1% /mnt/lustre[MDT:0] waiting for dbench pid 44451 lustre-OST0000_UUID 3771392 3072 3694592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3732480 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7427072 1% /mnt/lustre dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 140 6.02 MB/sec warmup 1 sec latency 49.343 ms 1 251 4.73 MB/sec warmup 2 sec latency 40.510 ms 1 382 4.52 MB/sec warmup 3 sec latency 30.765 ms test_26 fail mds1 1 times Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 503 4.37 MB/sec warmup 4 sec latency 54.004 ms 1 522 3.61 MB/sec warmup 5 sec latency 803.229 ms 18:02:16 (1756677736) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 522 3.01 MB/sec warmup 6 sec latency 1803.445 ms 1 522 2.58 MB/sec warmup 7 sec latency 2803.607 ms 1 522 2.26 MB/sec warmup 8 sec latency 3803.762 ms 1 522 2.01 MB/sec warmup 9 sec latency 4803.903 ms 1 522 1.81 MB/sec warmup 10 sec latency 5804.051 ms 1 522 1.64 MB/sec warmup 11 sec latency 6804.253 ms 1 522 1.50 MB/sec warmup 12 sec latency 7804.459 ms 1 522 1.39 MB/sec warmup 13 sec latency 8804.622 ms 1 522 1.29 MB/sec warmup 14 sec latency 9804.777 ms 1 522 1.20 MB/sec warmup 15 sec latency 10804.933 ms Failover mds1 to oleg108-server mount facets: mds1 1 522 1.13 MB/sec warmup 16 sec latency 11805.135 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 522 1.06 MB/sec warmup 17 sec latency 12805.354 ms 1 522 1.00 MB/sec warmup 18 sec latency 13805.525 ms oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 1 522 0.95 MB/sec warmup 19 sec latency 14805.671 ms Started lustre-MDT0000 18:02:30 (1756677750) targets are mounted 18:02:30 (1756677750) facet_failover done 1 522 0.00 MB/sec execute 1 sec latency 16805.937 ms 1 522 0.00 MB/sec execute 2 sec latency 17806.163 ms 1 522 0.00 MB/sec execute 3 sec latency 18806.291 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 692 0.78 MB/sec execute 4 sec latency 18842.382 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 799 0.68 MB/sec execute 5 sec latency 91.201 ms 1 926 0.62 MB/sec execute 6 sec latency 78.252 ms 1 1086 0.70 MB/sec execute 7 sec latency 34.776 ms 1 1224 0.63 MB/sec execute 8 sec latency 34.891 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 34816 3634176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3625984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7260160 1% /mnt/lustre 1 1418 0.88 MB/sec execute 9 sec latency 59.839 ms 1 1568 0.81 MB/sec execute 10 sec latency 42.434 ms 1 1699 0.74 MB/sec execute 11 sec latency 50.383 ms test_26 fail mds1 2 times 1 1888 0.69 MB/sec execute 12 sec latency 66.418 ms Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 2030 0.66 MB/sec execute 13 sec latency 168.732 ms 18:02:45 (1756677765) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 2030 0.61 MB/sec execute 14 sec latency 1168.865 ms 1 2030 0.57 MB/sec execute 15 sec latency 2169.005 ms 1 2030 0.53 MB/sec execute 16 sec latency 3169.166 ms 1 2030 0.50 MB/sec execute 17 sec latency 4169.296 ms 1 2030 0.47 MB/sec execute 18 sec latency 5169.462 ms 1 2030 0.45 MB/sec execute 19 sec latency 6169.628 ms 1 2030 0.43 MB/sec execute 20 sec latency 7169.774 ms 1 2030 0.41 MB/sec execute 21 sec latency 8170.054 ms 1 2030 0.39 MB/sec execute 22 sec latency 9170.240 ms 1 2030 0.37 MB/sec execute 23 sec latency 10170.441 ms 1 2030 0.36 MB/sec execute 24 sec latency 11170.597 ms Failover mds1 to oleg108-server mount facets: mds1 1 2030 0.34 MB/sec execute 25 sec latency 12170.714 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2030 0.33 MB/sec execute 26 sec latency 13170.914 ms 1 2030 0.32 MB/sec execute 27 sec latency 14171.035 ms oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 18:02:58 (1756677778) targets are mounted 18:02:58 (1756677778) facet_failover done 1 2030 0.30 MB/sec execute 28 sec latency 15171.181 ms 1 2030 0.29 MB/sec execute 29 sec latency 16171.314 ms 1 2030 0.28 MB/sec execute 30 sec latency 17171.454 ms 1 2030 0.27 MB/sec execute 31 sec latency 18171.632 ms 1 2030 0.27 MB/sec execute 32 sec latency 19171.832 ms 1 2030 0.26 MB/sec execute 33 sec latency 20171.991 ms 1 2030 0.25 MB/sec execute 34 sec latency 21172.271 ms 1 2030 0.24 MB/sec execute 35 sec latency 22172.567 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2160 0.24 MB/sec execute 36 sec latency 22342.466 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2345 0.27 MB/sec execute 37 sec latency 42.644 ms 1 2449 0.26 MB/sec execute 38 sec latency 128.431 ms 1 2607 0.28 MB/sec execute 39 sec latency 68.260 ms 1 2906 0.37 MB/sec execute 40 sec latency 80.045 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5120 2203392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 38912 3568640 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 36864 3604480 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 75776 7173120 2% /mnt/lustre 1 3028 0.38 MB/sec execute 41 sec latency 86.022 ms 1 3365 0.41 MB/sec execute 42 sec latency 54.047 ms 1 3624 0.48 MB/sec execute 43 sec latency 32.899 ms test_26 fail mds1 3 times 1 3775 0.49 MB/sec execute 44 sec latency 32.352 ms Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 3884 0.48 MB/sec execute 45 sec latency 40.658 ms 1 3915 0.48 MB/sec execute 46 sec latency 754.232 ms 18:03:17 (1756677797) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 3915 0.47 MB/sec execute 47 sec latency 1754.363 ms 1 3915 0.46 MB/sec execute 48 sec latency 2754.555 ms 1 3915 0.45 MB/sec execute 49 sec latency 3754.820 ms 1 3915 0.44 MB/sec execute 50 sec latency 4754.986 ms 1 3915 0.43 MB/sec execute 51 sec latency 5755.141 ms 1 3915 0.42 MB/sec execute 52 sec latency 6755.307 ms 1 3915 0.42 MB/sec execute 53 sec latency 7755.484 ms 1 3915 0.41 MB/sec execute 54 sec latency 8755.650 ms 1 3915 0.40 MB/sec execute 55 sec latency 9755.891 ms 1 3915 0.39 MB/sec execute 56 sec latency 10756.159 ms Failover mds1 to oleg108-server mount facets: mds1 1 3915 0.39 MB/sec execute 57 sec latency 11756.331 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3915 0.38 MB/sec execute 58 sec latency 12756.528 ms 1 3915 0.37 MB/sec execute 59 sec latency 13756.712 ms oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all 1 3915 0.37 MB/sec execute 60 sec latency 14756.888 ms pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 18:03:31 (1756677811) targets are mounted 18:03:31 (1756677811) facet_failover done 1 3915 0.36 MB/sec execute 61 sec latency 15757.166 ms 1 3915 0.35 MB/sec execute 62 sec latency 16757.420 ms 1 3915 0.35 MB/sec execute 63 sec latency 17757.701 ms 1 3915 0.34 MB/sec execute 64 sec latency 18758.010 ms 1 3915 0.34 MB/sec execute 65 sec latency 19758.178 ms 1 3915 0.33 MB/sec execute 66 sec latency 20758.327 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4044 0.33 MB/sec execute 67 sec latency 20774.084 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4168 0.32 MB/sec execute 68 sec latency 32.119 ms 1 4289 0.32 MB/sec execute 69 sec latency 201.724 ms 1 4391 0.32 MB/sec execute 70 sec latency 106.633 ms 1 4493 0.32 MB/sec execute 71 sec latency 83.197 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6144 2202368 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3580928 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 37888 3577856 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 80896 7158784 2% /mnt/lustre 1 4653 0.33 MB/sec execute 72 sec latency 47.239 ms 1 4811 0.33 MB/sec execute 73 sec latency 33.952 ms 1 5023 0.37 MB/sec execute 74 sec latency 31.402 ms test_26 fail mds1 4 times 1 5158 0.36 MB/sec execute 75 sec latency 97.274 ms Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 5280 0.36 MB/sec execute 76 sec latency 151.383 ms 1 5280 0.35 MB/sec execute 77 sec latency 1011.308 ms 18:03:48 (1756677828) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 5280 0.35 MB/sec execute 78 sec latency 2011.599 ms 1 5280 0.35 MB/sec execute 79 sec latency 3011.824 ms 1 5280 0.34 MB/sec execute 80 sec latency 4012.028 ms 1 5280 0.34 MB/sec execute 81 sec latency 5012.249 ms 1 5280 0.33 MB/sec execute 82 sec latency 6012.419 ms 1 5280 0.33 MB/sec execute 83 sec latency 7012.657 ms 1 5280 0.33 MB/sec execute 84 sec latency 8012.893 ms 1 5280 0.32 MB/sec execute 85 sec latency 9013.177 ms 1 5280 0.32 MB/sec execute 86 sec latency 10013.386 ms 1 5280 0.31 MB/sec execute 87 sec latency 11013.673 ms Failover mds1 to oleg108-server mount facets: mds1 1 5280 0.31 MB/sec execute 88 sec latency 12013.868 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5280 0.31 MB/sec execute 89 sec latency 13014.061 ms 1 5280 0.30 MB/sec execute 90 sec latency 14014.200 ms oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 18:04:02 (1756677842) targets are mounted 18:04:02 (1756677842) facet_failover done 1 5280 0.30 MB/sec execute 91 sec latency 15014.425 ms 1 5280 0.30 MB/sec execute 92 sec latency 16014.547 ms 1 5280 0.29 MB/sec execute 93 sec latency 17014.711 ms 1 5280 0.29 MB/sec execute 94 sec latency 18014.961 ms 1 5280 0.29 MB/sec execute 95 sec latency 19015.102 ms 1 5336 0.28 MB/sec execute 96 sec latency 19521.258 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5420 0.28 MB/sec execute 97 sec latency 65.387 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5557 0.28 MB/sec execute 98 sec latency 152.453 ms 1 5714 0.28 MB/sec execute 99 sec latency 31.168 ms 1 cleanup 100 sec 1 cleanup 101 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 934 80.638 20774.069 Close 726 2.926 10.916 Rename 37 29.593 68.229 Unlink 155 9.635 35.946 Qpathinfo 878 30.627 22342.449 Qfileinfo 166 1.012 4.703 Qfsinfo 139 0.593 3.082 Sfileinfo 86 15.994 35.115 Find 315 2.899 20.579 WriteX 531 4.042 18.067 ReadX 1339 0.738 29.910 LockX 2 3.481 4.267 UnlockX 2 3.803 4.543 Flush 74 46.342 284.070 Throughput 0.280382 MB/sec 1 clients 1 procs max_latency=22342.466 ms UUID 1K-blocks Used Available Use% Mounted on stopping dbench on /mnt/lustre at Sun Aug 31 18:04:12 EDT 2025 with return code 0 lustre-MDT0000_UUID 2210560 6784 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 44032 3716096 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 38912 3728384 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 82944 7444480 2% /mnt/lustre clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt test_26 fail mds1 5 times Failing mds1 on oleg108-server '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Aug 31 18:04:16 EDT 2025 waiting for dbench pid 48574 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 751.154 ms 18:04:18 (1756677858) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 31 0.52 MB/sec warmup 2 sec latency 1751.498 ms 1 31 0.35 MB/sec warmup 3 sec latency 2751.705 ms 1 31 0.26 MB/sec warmup 4 sec latency 3751.945 ms 1 31 0.21 MB/sec warmup 5 sec latency 4752.124 ms 1 31 0.17 MB/sec warmup 6 sec latency 5752.280 ms 1 31 0.15 MB/sec warmup 7 sec latency 6752.510 ms 1 31 0.13 MB/sec warmup 8 sec latency 7752.781 ms 1 31 0.12 MB/sec warmup 9 sec latency 8752.951 ms 1 31 0.10 MB/sec warmup 10 sec latency 9753.144 ms 1 31 0.10 MB/sec warmup 11 sec latency 10753.480 ms Failover mds1 to oleg108-server mount facets: mds1 1 31 0.09 MB/sec warmup 12 sec latency 11753.687 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 31 0.08 MB/sec warmup 13 sec latency 12753.833 ms 1 31 0.07 MB/sec warmup 14 sec latency 13753.974 ms oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 1 31 0.07 MB/sec warmup 15 sec latency 14754.075 ms Started lustre-MDT0000 18:04:32 (1756677872) targets are mounted 18:04:32 (1756677872) facet_failover done 1 31 0.07 MB/sec warmup 16 sec latency 15754.226 ms 1 31 0.06 MB/sec warmup 17 sec latency 16754.358 ms 1 31 0.06 MB/sec warmup 18 sec latency 17754.548 ms 1 31 0.06 MB/sec warmup 19 sec latency 18754.689 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 173 4.97 MB/sec execute 1 sec latency 19646.916 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 289 4.30 MB/sec execute 2 sec latency 33.652 ms 1 405 4.08 MB/sec execute 3 sec latency 46.850 ms 1 550 4.25 MB/sec execute 4 sec latency 39.693 ms 1 669 3.81 MB/sec execute 5 sec latency 164.262 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5120 2203392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3716096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 41984 3683328 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 52224 7399424 1% /mnt/lustre 1 759 3.21 MB/sec execute 6 sec latency 130.133 ms 1 873 2.79 MB/sec execute 7 sec latency 77.214 ms 1 977 2.47 MB/sec execute 8 sec latency 51.713 ms test_26 fail mds1 6 times 1 1155 2.32 MB/sec execute 9 sec latency 58.112 ms Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 1405 2.39 MB/sec execute 10 sec latency 29.321 ms 1 1412 2.17 MB/sec execute 11 sec latency 952.078 ms 18:04:48 (1756677888) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 1412 1.99 MB/sec execute 12 sec latency 1952.213 ms 1 1412 1.84 MB/sec execute 13 sec latency 2952.393 ms 1 1412 1.71 MB/sec execute 14 sec latency 3952.571 ms 1 1412 1.59 MB/sec execute 15 sec latency 4952.755 ms 1 1412 1.49 MB/sec execute 16 sec latency 5952.926 ms 1 1412 1.41 MB/sec execute 17 sec latency 6953.064 ms 1 1412 1.33 MB/sec execute 18 sec latency 7953.208 ms 1 1412 1.26 MB/sec execute 19 sec latency 8953.367 ms 1 1412 1.20 MB/sec execute 20 sec latency 9953.536 ms 1 1412 1.14 MB/sec execute 21 sec latency 10953.698 ms Failover mds1 to oleg108-server mount facets: mds1 1 1412 1.09 MB/sec execute 22 sec latency 11953.837 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1412 1.04 MB/sec execute 23 sec latency 12954.018 ms 1 1412 1.00 MB/sec execute 24 sec latency 13954.170 ms oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1412 0.96 MB/sec execute 25 sec latency 14954.313 ms 18:05:01 (1756677901) targets are mounted 18:05:01 (1756677901) facet_failover done 1 1412 0.92 MB/sec execute 26 sec latency 15954.459 ms 1 1412 0.89 MB/sec execute 27 sec latency 16954.647 ms 1 1412 0.85 MB/sec execute 28 sec latency 17954.827 ms 1 1412 0.82 MB/sec execute 29 sec latency 18954.978 ms 1 1412 0.80 MB/sec execute 30 sec latency 19955.143 ms 1 1447 0.77 MB/sec execute 31 sec latency 20669.322 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1566 0.75 MB/sec execute 32 sec latency 242.548 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1696 0.73 MB/sec execute 33 sec latency 104.270 ms 1 1861 0.71 MB/sec execute 34 sec latency 31.092 ms 1 2006 0.70 MB/sec execute 35 sec latency 138.080 ms 1 2198 0.69 MB/sec execute 36 sec latency 25.690 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 9216 3721216 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 36864 3706880 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 46080 7428096 1% /mnt/lustre 1 2366 0.70 MB/sec execute 37 sec latency 134.184 ms 1 2489 0.70 MB/sec execute 38 sec latency 54.955 ms 1 2881 0.77 MB/sec execute 39 sec latency 59.354 ms test_26 fail mds1 7 times 1 3010 0.79 MB/sec execute 40 sec latency 59.822 ms Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 1 3210 0.78 MB/sec execute 41 sec latency 237.984 ms 18:05:18 (1756677918) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server 1 3210 0.77 MB/sec execute 42 sec latency 1238.112 ms 1 3210 0.75 MB/sec execute 43 sec latency 2238.292 ms 1 3210 0.73 MB/sec execute 44 sec latency 3238.444 ms 1 3210 0.72 MB/sec execute 45 sec latency 4238.599 ms 1 3210 0.70 MB/sec execute 46 sec latency 5238.756 ms 1 3210 0.68 MB/sec execute 47 sec latency 6238.967 ms 1 3210 0.67 MB/sec execute 48 sec latency 7239.134 ms 1 3210 0.66 MB/sec execute 49 sec latency 8239.287 ms 1 3210 0.64 MB/sec execute 50 sec latency 9239.430 ms 1 3210 0.63 MB/sec execute 51 sec latency 10239.585 ms Failover mds1 to oleg108-server mount facets: mds1 1 3210 0.62 MB/sec execute 52 sec latency 11239.737 ms 1 3210 0.61 MB/sec execute 53 sec latency 12239.886 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3210 0.60 MB/sec execute 54 sec latency 13240.017 ms 1 3210 0.59 MB/sec execute 55 sec latency 14240.159 ms oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 18:05:32 (1756677932) targets are mounted 18:05:32 (1756677932) facet_failover done 1 3210 0.57 MB/sec execute 56 sec latency 15240.358 ms 1 3210 0.56 MB/sec execute 57 sec latency 16240.522 ms 1 3210 0.55 MB/sec execute 58 sec latency 17240.655 ms 1 3210 0.55 MB/sec execute 59 sec latency 18240.849 ms 1 3210 0.54 MB/sec execute 60 sec latency 19240.995 ms 1 3210 0.53 MB/sec execute 61 sec latency 20241.119 ms 1 3331 0.54 MB/sec execute 62 sec latency 20635.496 ms oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3572 0.58 MB/sec execute 63 sec latency 35.321 ms 1 3724 0.57 MB/sec execute 64 sec latency 31.482 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 3885 0.58 MB/sec execute 65 sec latency 26.549 ms 1 4021 0.58 MB/sec execute 66 sec latency 28.254 ms 1 4125 0.57 MB/sec execute 67 sec latency 57.851 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sun Aug 31 18:05:44 EDT 2025 with return code 0 48574 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48574 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished