== replay-dual test 26: dbench and tar with mds failover ========================================================== 13:07:27 (1761239247) Starting client oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre Started clients oleg447-client.virtnet: 192.168.204.147@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65094 Started dbench loop with 65095 striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Oct 23 13:07:30 EDT 2025 waiting for dbench pid 65145 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1972 1269332 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1776 1269528 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13852 3579652 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 15396 7185128 1% /mnt/lustre 1 51 2.10 MB/sec warmup 1 sec latency 55.173 ms 1 137 3.00 MB/sec warmup 2 sec latency 85.330 ms 1 184 2.46 MB/sec warmup 3 sec latency 152.811 ms 1 231 2.22 MB/sec warmup 4 sec latency 281.307 ms 1 289 2.14 MB/sec warmup 5 sec latency 83.938 ms 1 350 2.09 MB/sec warmup 6 sec latency 82.382 ms 1 421 2.12 MB/sec warmup 7 sec latency 76.126 ms test_26 fail mds1 1 times 1 497 2.15 MB/sec warmup 8 sec latency 67.453 ms Failing mds1 on oleg447-server Stopping /mnt/lustre-mds1 (opts:) on oleg447-server 1 594 2.29 MB/sec warmup 9 sec latency 75.248 ms 1 616 2.09 MB/sec warmup 10 sec latency 779.599 ms 13:07:41 (1761239261) shut down facet: mds1 facet_host: oleg447-server facet_failover_host: oleg447-server 1 616 1.90 MB/sec warmup 11 sec latency 1779.884 ms 1 616 1.75 MB/sec warmup 12 sec latency 2780.077 ms 1 616 1.61 MB/sec warmup 13 sec latency 3780.830 ms 1 616 1.50 MB/sec warmup 14 sec latency 4781.027 ms 1 616 1.40 MB/sec warmup 15 sec latency 5781.244 ms 1 616 1.31 MB/sec warmup 16 sec latency 6781.447 ms 1 616 1.23 MB/sec warmup 17 sec latency 7781.624 ms 1 616 1.16 MB/sec warmup 18 sec latency 8781.864 ms 1 616 1.10 MB/sec warmup 19 sec latency 9782.467 ms Failover mds1 to oleg447-server 1 616 0.00 MB/sec execute 1 sec latency 11783.489 ms mount facets: mds1 1 616 0.00 MB/sec execute 2 sec latency 12784.711 ms 1 616 0.00 MB/sec execute 3 sec latency 13785.379 ms 1 616 0.00 MB/sec execute 4 sec latency 14785.593 ms 1 616 0.00 MB/sec execute 5 sec latency 15785.797 ms 1 616 0.00 MB/sec execute 6 sec latency 16786.530 ms 1 616 0.00 MB/sec execute 7 sec latency 17786.728 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 616 0.00 MB/sec execute 8 sec latency 18787.613 ms 1 616 0.00 MB/sec execute 9 sec latency 19788.708 ms 1 616 0.00 MB/sec execute 10 sec latency 20788.911 ms 1 616 0.00 MB/sec execute 11 sec latency 21794.786 ms 1 616 0.00 MB/sec execute 12 sec latency 22796.053 ms 1 616 0.00 MB/sec execute 13 sec latency 23796.279 ms 1 616 0.00 MB/sec execute 14 sec latency 24800.528 ms 1 616 0.00 MB/sec execute 15 sec latency 25800.967 ms 1 616 0.00 MB/sec execute 16 sec latency 26801.190 ms 1 616 0.00 MB/sec execute 17 sec latency 27801.356 ms 1 616 0.00 MB/sec execute 18 sec latency 28804.375 ms 1 616 0.00 MB/sec execute 19 sec latency 29808.442 ms 1 616 0.00 MB/sec execute 20 sec latency 30808.647 ms oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 1 616 0.00 MB/sec execute 21 sec latency 31808.907 ms pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 1 616 0.00 MB/sec execute 22 sec latency 32809.734 ms 13:08:13 (1761239293) targets are mounted 13:08:13 (1761239293) facet_failover done 1 616 0.00 MB/sec execute 23 sec latency 33810.220 ms 1 616 0.00 MB/sec execute 24 sec latency 34811.782 ms 1 638 0.00 MB/sec execute 25 sec latency 35672.765 ms 1 697 0.01 MB/sec execute 26 sec latency 70.872 ms oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 759 0.01 MB/sec execute 27 sec latency 71.319 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 806 0.02 MB/sec execute 28 sec latency 73.843 ms 1 874 0.02 MB/sec execute 29 sec latency 74.222 ms 1 884 0.02 MB/sec execute 30 sec latency 818.365 ms 1 923 0.03 MB/sec execute 31 sec latency 914.496 ms 1 981 0.03 MB/sec execute 32 sec latency 73.563 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2112 1269192 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1912 1269392 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 39812 3558304 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14744 3591576 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 54556 7149880 1% /mnt/lustre 1 1055 0.06 MB/sec execute 33 sec latency 70.180 ms 1 1126 0.06 MB/sec execute 34 sec latency 65.736 ms 1 1176 0.06 MB/sec execute 35 sec latency 102.401 ms 1 1231 0.06 MB/sec execute 36 sec latency 70.637 ms 1 1277 0.06 MB/sec execute 37 sec latency 102.956 ms 1 1419 0.13 MB/sec execute 38 sec latency 78.335 ms 1 1486 0.13 MB/sec execute 39 sec latency 166.139 ms 1 1560 0.13 MB/sec execute 40 sec latency 61.611 ms test_26 fail mds2 2 times 1 1610 0.13 MB/sec execute 41 sec latency 71.650 ms Failing mds2 on oleg447-server 1 1660 0.12 MB/sec execute 42 sec latency 106.491 ms Stopping /mnt/lustre-mds2 (opts:) on oleg447-server 1 1736 0.12 MB/sec execute 43 sec latency 89.091 ms 1 1824 0.12 MB/sec execute 44 sec latency 53.634 ms 13:08:35 (1761239315) shut down facet: mds2 facet_host: oleg447-server facet_failover_host: oleg447-server 1 1902 0.12 MB/sec execute 45 sec latency 80.031 ms 1 1996 0.12 MB/sec execute 46 sec latency 57.562 ms 1 2063 0.12 MB/sec execute 47 sec latency 65.504 ms 1 2145 0.12 MB/sec execute 48 sec latency 75.929 ms 1 2195 0.12 MB/sec execute 49 sec latency 61.146 ms 1 2259 0.12 MB/sec execute 50 sec latency 400.436 ms 1 2259 0.11 MB/sec execute 51 sec latency 1401.005 ms 1 2259 0.11 MB/sec execute 52 sec latency 2402.049 ms 1 2259 0.11 MB/sec execute 53 sec latency 3402.358 ms 1 2259 0.11 MB/sec execute 54 sec latency 4403.603 ms Failover mds2 to oleg447-server mount facets: mds2 1 2259 0.11 MB/sec execute 55 sec latency 5403.759 ms 1 2259 0.10 MB/sec execute 56 sec latency 6406.996 ms 1 2259 0.10 MB/sec execute 57 sec latency 7407.194 ms 1 2259 0.10 MB/sec execute 58 sec latency 8410.518 ms 1 2259 0.10 MB/sec execute 59 sec latency 9412.341 ms 1 2259 0.10 MB/sec execute 60 sec latency 10412.592 ms 1 2259 0.10 MB/sec execute 61 sec latency 11413.225 ms 1 2259 0.09 MB/sec execute 62 sec latency 12413.415 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.09 MB/sec execute 63 sec latency 13414.783 ms 1 2259 0.09 MB/sec execute 64 sec latency 14414.968 ms 1 2259 0.09 MB/sec execute 65 sec latency 15415.472 ms oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 1 2259 0.09 MB/sec execute 66 sec latency 16415.745 ms pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 1 2259 0.09 MB/sec execute 67 sec latency 17415.939 ms Started lustre-MDT0001 13:08:58 (1761239338) targets are mounted 13:08:58 (1761239338) facet_failover done 1 2259 0.09 MB/sec execute 68 sec latency 18416.158 ms 1 2259 0.08 MB/sec execute 69 sec latency 19417.358 ms 1 2259 0.08 MB/sec execute 70 sec latency 20417.546 ms 1 2327 0.10 MB/sec execute 71 sec latency 20617.457 ms 1 2368 0.10 MB/sec execute 72 sec latency 85.558 ms oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 2416 0.10 MB/sec execute 73 sec latency 93.737 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2457 0.10 MB/sec execute 74 sec latency 79.271 ms 1 2508 0.10 MB/sec execute 75 sec latency 106.534 ms 1 2591 0.11 MB/sec execute 76 sec latency 80.360 ms 1 2764 0.14 MB/sec execute 77 sec latency 87.157 ms 1 2902 0.15 MB/sec execute 78 sec latency 80.651 ms 1 2979 0.16 MB/sec execute 79 sec latency 82.492 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2136 1269168 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 45292 3560412 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23840 3579480 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 69132 7139892 1% /mnt/lustre 1 3024 0.16 MB/sec execute 80 sec latency 103.216 ms 1 3210 0.16 MB/sec execute 81 sec latency 99.274 ms 1 3296 0.17 MB/sec execute 82 sec latency 103.447 ms 1 3359 0.17 MB/sec execute 83 sec latency 66.924 ms 1 3494 0.21 MB/sec execute 84 sec latency 89.099 ms 1 3564 0.20 MB/sec execute 85 sec latency 62.717 ms 1 3645 0.21 MB/sec execute 86 sec latency 62.750 ms 1 3717 0.20 MB/sec execute 87 sec latency 60.686 ms 1 3789 0.21 MB/sec execute 88 sec latency 88.441 ms test_26 fail mds1 3 times 1 3843 0.21 MB/sec execute 89 sec latency 83.813 ms Failing mds1 on oleg447-server 1 3901 0.21 MB/sec execute 90 sec latency 77.846 ms Stopping /mnt/lustre-mds1 (opts:) on oleg447-server 1 3955 0.21 MB/sec execute 91 sec latency 83.052 ms 1 4018 0.21 MB/sec execute 92 sec latency 79.867 ms 1 4083 0.21 MB/sec execute 93 sec latency 73.175 ms 1 4083 0.20 MB/sec execute 94 sec latency 1002.545 ms 1 4083 0.20 MB/sec execute 95 sec latency 2002.829 ms 1 4083 0.20 MB/sec execute 96 sec latency 3009.369 ms 1 4083 0.20 MB/sec execute 97 sec latency 4009.651 ms 1 4083 0.19 MB/sec execute 98 sec latency 5011.312 ms 13:09:29 (1761239369) shut down facet: mds1 facet_host: oleg447-server facet_failover_host: oleg447-server 1 4083 0.19 MB/sec execute 99 sec latency 6015.571 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec Failover mds1 to oleg447-server mount facets: mds1 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 cleanup 114 sec 1 cleanup 115 sec 1 cleanup 116 sec oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 1 cleanup 117 sec pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 1 cleanup 118 sec Started lustre-MDT0000 13:09:49 (1761239389) targets are mounted 13:09:49 (1761239389) facet_failover done 1 cleanup 119 sec 1 cleanup 120 sec 1 cleanup 121 sec oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 cleanup 122 sec 0 cleanup 122 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 611 128.124 35672.748 Close 446 9.415 37.281 Rename 26 64.419 106.510 Unlink 96 18.310 43.899 Qpathinfo 532 62.588 26779.770 Qfileinfo 97 1.272 13.250 Qfsinfo 98 2.010 26.766 Sfileinfo 50 43.125 79.252 Find 202 9.519 47.731 WriteX 303 10.054 60.658 ReadX 960 1.003 43.078 LockX 2 6.012 6.752 UnlockX 2 7.024 7.784 Flush 43 43.176 85.542 Throughput 0.192789 MB/sec 1 clients 1 procs max_latency=35672.765 ms stopping dbench on /mnt/lustre at Thu Oct 23 13:09:53 EDT 2025 with return code 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Oct 23 13:09:59 EDT 2025 waiting for dbench pid 68298 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2684 1268620 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2456 1268848 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11720 3593936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12368 3567472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 24088 7161408 1% /mnt/lustre 1 31 1.05 MB/sec warmup 1 sec latency 17.941 ms 1 99 2.26 MB/sec warmup 2 sec latency 100.644 ms 1 140 2.01 MB/sec warmup 3 sec latency 161.507 ms 1 174 1.76 MB/sec warmup 4 sec latency 159.725 ms 1 215 1.67 MB/sec warmup 5 sec latency 243.872 ms 1 268 1.67 MB/sec warmup 6 sec latency 135.595 ms 1 323 1.68 MB/sec warmup 7 sec latency 106.796 ms 1 381 1.69 MB/sec warmup 8 sec latency 79.758 ms test_26 fail mds2 4 times 1 438 1.71 MB/sec warmup 9 sec latency 98.614 ms Failing mds2 on oleg447-server 1 498 1.73 MB/sec warmup 10 sec latency 80.938 ms Stopping /mnt/lustre-mds2 (opts:) on oleg447-server 1 603 1.90 MB/sec warmup 11 sec latency 89.901 ms 1 695 1.76 MB/sec warmup 12 sec latency 61.309 ms 13:10:12 (1761239412) shut down facet: mds2 facet_host: oleg447-server facet_failover_host: oleg447-server 1 770 1.64 MB/sec warmup 13 sec latency 59.693 ms 1 870 1.54 MB/sec warmup 14 sec latency 50.225 ms 1 900 1.44 MB/sec warmup 15 sec latency 533.684 ms 1 970 1.36 MB/sec warmup 16 sec latency 62.808 ms 1 995 1.29 MB/sec warmup 17 sec latency 616.882 ms 1 995 1.22 MB/sec warmup 18 sec latency 1617.131 ms 1 995 1.15 MB/sec warmup 19 sec latency 2617.561 ms 1 995 0.00 MB/sec execute 1 sec latency 4617.951 ms 1 995 0.00 MB/sec execute 2 sec latency 5620.357 ms Failover mds2 to oleg447-server mount facets: mds2 1 995 0.00 MB/sec execute 3 sec latency 6621.300 ms 1 995 0.00 MB/sec execute 4 sec latency 7621.616 ms 1 995 0.00 MB/sec execute 5 sec latency 8621.865 ms 1 995 0.00 MB/sec execute 6 sec latency 9622.974 ms 1 995 0.00 MB/sec execute 7 sec latency 10623.682 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 995 0.00 MB/sec execute 8 sec latency 11623.940 ms 1 995 0.00 MB/sec execute 9 sec latency 12624.786 ms 1 995 0.00 MB/sec execute 10 sec latency 13625.005 ms oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 1 995 0.00 MB/sec execute 11 sec latency 14625.365 ms pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0001 13:10:31 (1761239431) targets are mounted 13:10:31 (1761239431) facet_failover done 1 995 0.00 MB/sec execute 12 sec latency 15625.578 ms 1 995 0.00 MB/sec execute 13 sec latency 16626.430 ms 1 995 0.00 MB/sec execute 14 sec latency 17627.451 ms 1 995 0.00 MB/sec execute 15 sec latency 18628.605 ms 1 1074 0.06 MB/sec execute 16 sec latency 18660.825 ms oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1164 0.06 MB/sec execute 17 sec latency 74.890 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1244 0.07 MB/sec execute 18 sec latency 43.277 ms 1 1418 0.22 MB/sec execute 19 sec latency 53.399 ms 1 1486 0.21 MB/sec execute 20 sec latency 168.125 ms 1 1580 0.20 MB/sec execute 21 sec latency 66.313 ms 1 1671 0.20 MB/sec execute 22 sec latency 55.532 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3048 1268256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2884 1268420 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 20928 3582188 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 46608 3556088 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 67536 7138276 1% /mnt/lustre 1 1762 0.19 MB/sec execute 23 sec latency 69.021 ms 1 1861 0.19 MB/sec execute 24 sec latency 54.065 ms 1 1923 0.18 MB/sec execute 25 sec latency 504.139 ms 1 1976 0.17 MB/sec execute 26 sec latency 977.597 ms 1 2056 0.17 MB/sec execute 27 sec latency 76.048 ms 1 2154 0.17 MB/sec execute 28 sec latency 53.230 ms test_26 fail mds1 5 times 1 2235 0.17 MB/sec execute 29 sec latency 56.632 ms Failing mds1 on oleg447-server Stopping /mnt/lustre-mds1 (opts:) on oleg447-server 1 2348 0.20 MB/sec execute 30 sec latency 80.089 ms 1 2405 0.20 MB/sec execute 31 sec latency 110.916 ms 1 2479 0.21 MB/sec execute 32 sec latency 48.689 ms 13:10:51 (1761239451) shut down facet: mds1 facet_host: oleg447-server facet_failover_host: oleg447-server 1 2600 0.21 MB/sec execute 33 sec latency 71.714 ms 1 2880 0.30 MB/sec execute 34 sec latency 43.649 ms 1 2989 0.33 MB/sec execute 35 sec latency 63.503 ms 1 3210 0.34 MB/sec execute 36 sec latency 59.771 ms 1 3362 0.37 MB/sec execute 37 sec latency 59.862 ms 1 3569 0.43 MB/sec execute 38 sec latency 34.713 ms 1 3689 0.43 MB/sec execute 39 sec latency 32.095 ms 1 3798 0.45 MB/sec execute 40 sec latency 51.489 ms 1 3892 0.44 MB/sec execute 41 sec latency 47.962 ms 1 3988 0.43 MB/sec execute 42 sec latency 43.610 ms Failover mds1 to oleg447-server mount facets: mds1 1 4079 0.42 MB/sec execute 43 sec latency 46.319 ms 1 4083 0.41 MB/sec execute 44 sec latency 996.396 ms 1 4083 0.40 MB/sec execute 45 sec latency 1996.642 ms 1 4083 0.39 MB/sec execute 46 sec latency 2996.953 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.39 MB/sec execute 47 sec latency 3997.313 ms 1 4083 0.38 MB/sec execute 48 sec latency 4997.546 ms 1 4083 0.37 MB/sec execute 49 sec latency 5998.186 ms oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 1 4083 0.36 MB/sec execute 50 sec latency 6999.031 ms Started lustre-MDT0000 13:11:10 (1761239470) targets are mounted 13:11:10 (1761239470) facet_failover done 1 4083 0.36 MB/sec execute 51 sec latency 7999.217 ms 1 4083 0.35 MB/sec execute 52 sec latency 8999.625 ms 1 4083 0.34 MB/sec execute 53 sec latency 10000.263 ms 1 4083 0.34 MB/sec execute 54 sec latency 11000.503 ms 1 4171 0.33 MB/sec execute 55 sec latency 11049.970 ms oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4260 0.33 MB/sec execute 56 sec latency 71.775 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4309 0.32 MB/sec execute 57 sec latency 84.853 ms 1 4372 0.32 MB/sec execute 58 sec latency 67.437 ms 1 4444 0.32 MB/sec execute 59 sec latency 71.374 ms 1 4493 0.32 MB/sec execute 60 sec latency 71.606 ms 1 4549 0.31 MB/sec execute 61 sec latency 73.607 ms 1 4623 0.32 MB/sec execute 62 sec latency 78.504 ms 1 4706 0.32 MB/sec execute 63 sec latency 44.983 ms 1 4785 0.32 MB/sec execute 64 sec latency 68.692 ms 1 4884 0.33 MB/sec execute 65 sec latency 57.840 ms 1 4994 0.35 MB/sec execute 66 sec latency 56.019 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 5062 0.35 MB/sec execute 67 sec latency 62.371 ms 1 5123 0.34 MB/sec execute 68 sec latency 73.074 ms 1 5162 0.34 MB/sec execute 69 sec latency 81.484 ms 1 5215 0.33 MB/sec execute 70 sec latency 146.997 ms 1 5274 0.33 MB/sec execute 71 sec latency 90.126 ms 1 5336 0.33 MB/sec execute 72 sec latency 127.295 ms 1 5390 0.32 MB/sec execute 73 sec latency 134.716 ms 1 5489 0.32 MB/sec execute 74 sec latency 99.297 ms 1 5575 0.32 MB/sec execute 75 sec latency 132.142 ms 1 5695 0.31 MB/sec execute 76 sec latency 61.332 ms 1 5771 0.31 MB/sec execute 77 sec latency 67.321 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Oct 23 13:11:36 EDT 2025 with return code 0 68298 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 68298 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished