== replay-dual test 26: dbench and tar with mds failover ========================================================== 10:50:30 (1749480630) Starting client oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre Started clients oleg416-client.virtnet: 192.168.204.116@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 4987 Started dbench loop with 4988 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Jun 9 10:50:32 EDT 2025 waiting for dbench pid 5029 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 255 9.66 MB/sec warmup 1 sec latency 22.199 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2292 1285396 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1285580 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26112 3555904 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27656 7161380 1% /mnt/lustre 1 501 8.73 MB/sec warmup 2 sec latency 22.350 ms 1 699 7.07 MB/sec warmup 3 sec latency 435.096 ms 1 910 5.44 MB/sec warmup 4 sec latency 22.900 ms test_26 fail mds1 1 times 1 1181 4.60 MB/sec warmup 5 sec latency 19.607 ms Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 1490 4.34 MB/sec warmup 6 sec latency 149.324 ms 10:50:38 (1749480638) shut down facet: mds1 facet_host: oleg416-server facet_failover_host: oleg416-server 1 1490 3.72 MB/sec warmup 7 sec latency 1149.545 ms 1 1490 3.26 MB/sec warmup 8 sec latency 2149.879 ms 1 1490 2.89 MB/sec warmup 9 sec latency 3150.089 ms 1 1490 2.60 MB/sec warmup 10 sec latency 4150.338 ms 1 1490 2.37 MB/sec warmup 11 sec latency 5150.553 ms 1 1490 2.17 MB/sec warmup 12 sec latency 6150.883 ms 1 1490 2.00 MB/sec warmup 13 sec latency 7151.114 ms 1 1490 1.86 MB/sec warmup 14 sec latency 8151.436 ms 1 1490 1.74 MB/sec warmup 15 sec latency 9151.706 ms 1 1490 1.63 MB/sec warmup 16 sec latency 10151.872 ms Failover mds1 to oleg416-server mount facets: mds1 1 1490 1.53 MB/sec warmup 17 sec latency 11152.007 ms 1 1490 1.45 MB/sec warmup 18 sec latency 12152.178 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1490 1.37 MB/sec warmup 19 sec latency 13152.318 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 10:50:52 (1749480652) targets are mounted 10:50:52 (1749480652) facet_failover done 1 1490 0.00 MB/sec execute 1 sec latency 15152.613 ms 1 1490 0.00 MB/sec execute 2 sec latency 16152.772 ms 1 1490 0.00 MB/sec execute 3 sec latency 17152.999 ms 1 1490 0.00 MB/sec execute 4 sec latency 18153.126 ms 1 1532 0.01 MB/sec execute 5 sec latency 19000.685 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1855 0.05 MB/sec execute 6 sec latency 20.836 ms 1 2148 0.09 MB/sec execute 7 sec latency 18.711 ms 1 2389 0.24 MB/sec execute 8 sec latency 45.216 ms 1 2888 0.71 MB/sec execute 9 sec latency 17.171 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2952 1284736 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2592 1285096 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 6124 3586204 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 31976 3564832 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38100 7151036 1% /mnt/lustre 1 3370 0.94 MB/sec execute 10 sec latency 17.309 ms 1 3760 1.25 MB/sec execute 11 sec latency 16.080 ms 1 3955 1.17 MB/sec execute 12 sec latency 20.422 ms 1 4179 1.08 MB/sec execute 13 sec latency 20.328 ms test_26 fail mds2 2 times Failing mds2 on oleg416-server Stopping /mnt/lustre-mds2 (opts:) on oleg416-server 1 4452 1.05 MB/sec execute 14 sec latency 19.696 ms 10:51:07 (1749480667) shut down facet: mds2 facet_host: oleg416-server facet_failover_host: oleg416-server 1 4499 0.99 MB/sec execute 15 sec latency 778.979 ms 1 4499 0.93 MB/sec execute 16 sec latency 1779.092 ms 1 4499 0.87 MB/sec execute 17 sec latency 2779.179 ms 1 4499 0.82 MB/sec execute 18 sec latency 3779.276 ms 1 4499 0.78 MB/sec execute 19 sec latency 4779.405 ms 1 4499 0.74 MB/sec execute 20 sec latency 5779.548 ms 1 4499 0.71 MB/sec execute 21 sec latency 6779.686 ms 1 4499 0.67 MB/sec execute 22 sec latency 7779.842 ms 1 4499 0.65 MB/sec execute 23 sec latency 8780.011 ms 1 4499 0.62 MB/sec execute 24 sec latency 9780.119 ms Failover mds2 to oleg416-server mount facets: mds2 1 4499 0.59 MB/sec execute 25 sec latency 10780.292 ms 1 4499 0.57 MB/sec execute 26 sec latency 11780.410 ms 1 4499 0.55 MB/sec execute 27 sec latency 12780.555 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4499 0.53 MB/sec execute 28 sec latency 13780.705 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0001 10:51:21 (1749480681) targets are mounted 10:51:21 (1749480681) facet_failover done 1 4499 0.51 MB/sec execute 29 sec latency 14780.886 ms 1 4499 0.49 MB/sec execute 30 sec latency 15781.045 ms 1 4499 0.48 MB/sec execute 31 sec latency 16781.290 ms 1 4499 0.46 MB/sec execute 32 sec latency 17781.579 ms 1 4583 0.47 MB/sec execute 33 sec latency 18479.535 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4955 0.56 MB/sec execute 34 sec latency 16.380 ms 1 5224 0.55 MB/sec execute 35 sec latency 16.407 ms 1 5519 0.54 MB/sec execute 36 sec latency 20.668 ms 1 5825 0.54 MB/sec execute 37 sec latency 18.158 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4548 1283140 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12040 3572952 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38656 3554220 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50696 7127172 1% /mnt/lustre 1 6109 0.58 MB/sec execute 38 sec latency 52.501 ms 1 6435 0.65 MB/sec execute 39 sec latency 397.978 ms 1 6867 0.71 MB/sec execute 40 sec latency 24.255 ms 1 7226 0.78 MB/sec execute 41 sec latency 19.769 ms test_26 fail mds1 3 times Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 7462 0.79 MB/sec execute 42 sec latency 20.735 ms 1 7464 0.77 MB/sec execute 43 sec latency 995.057 ms 1 7464 0.75 MB/sec execute 44 sec latency 1995.365 ms 1 7464 0.74 MB/sec execute 45 sec latency 2995.567 ms 1 7464 0.72 MB/sec execute 46 sec latency 3995.791 ms 1 7464 0.70 MB/sec execute 47 sec latency 4995.947 ms 10:51:40 (1749480700) shut down facet: mds1 facet_host: oleg416-server facet_failover_host: oleg416-server 1 7464 0.69 MB/sec execute 48 sec latency 5996.112 ms 1 7464 0.68 MB/sec execute 49 sec latency 6996.318 ms 1 7464 0.66 MB/sec execute 50 sec latency 7996.590 ms 1 7464 0.65 MB/sec execute 51 sec latency 8996.890 ms 1 7464 0.64 MB/sec execute 52 sec latency 9997.145 ms 1 7464 0.63 MB/sec execute 53 sec latency 10997.404 ms 1 7464 0.61 MB/sec execute 54 sec latency 11997.564 ms 1 7464 0.60 MB/sec execute 55 sec latency 12997.687 ms 1 7464 0.59 MB/sec execute 56 sec latency 13997.870 ms 1 7464 0.58 MB/sec execute 57 sec latency 14998.069 ms Failover mds1 to oleg416-server mount facets: mds1 1 7464 0.57 MB/sec execute 58 sec latency 15998.224 ms 1 7464 0.56 MB/sec execute 59 sec latency 16998.340 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7464 0.55 MB/sec execute 60 sec latency 17998.459 ms 1 7464 0.54 MB/sec execute 61 sec latency 18998.572 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 10:51:53 (1749480713) targets are mounted 10:51:53 (1749480713) facet_failover done 1 7464 0.53 MB/sec execute 62 sec latency 19998.682 ms 1 7464 0.53 MB/sec execute 63 sec latency 20998.968 ms 1 7464 0.52 MB/sec execute 64 sec latency 21999.177 ms 1 7464 0.51 MB/sec execute 65 sec latency 22999.444 ms 1 7464 0.50 MB/sec execute 66 sec latency 23999.612 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 7572 0.49 MB/sec execute 67 sec latency 24331.906 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7765 0.49 MB/sec execute 68 sec latency 28.851 ms 1 7977 0.49 MB/sec execute 69 sec latency 33.313 ms 1 8174 0.50 MB/sec execute 70 sec latency 29.588 ms 1 8359 0.50 MB/sec execute 71 sec latency 30.778 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5008 1282680 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3012 1284676 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23240 3582208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 50192 3551244 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 73432 7133452 2% /mnt/lustre 1 8636 0.53 MB/sec execute 72 sec latency 21.510 ms 1 8751 0.53 MB/sec execute 73 sec latency 522.418 ms 1 8933 0.52 MB/sec execute 74 sec latency 23.791 ms 1 9189 0.52 MB/sec execute 75 sec latency 22.355 ms test_26 fail mds2 4 times Failing mds2 on oleg416-server 1 9439 0.53 MB/sec execute 76 sec latency 22.652 ms Stopping /mnt/lustre-mds2 (opts:) on oleg416-server 1 9606 0.53 MB/sec execute 77 sec latency 32.371 ms 10:52:09 (1749480729) shut down facet: mds2 facet_host: oleg416-server facet_failover_host: oleg416-server 1 10025 0.59 MB/sec execute 78 sec latency 21.299 ms 1 10418 0.60 MB/sec execute 79 sec latency 31.368 ms 1 10732 0.64 MB/sec execute 80 sec latency 22.651 ms 1 10899 0.64 MB/sec execute 81 sec latency 26.953 ms 1 11050 0.64 MB/sec execute 82 sec latency 28.067 ms 1 11173 0.63 MB/sec execute 83 sec latency 75.699 ms 1 11173 0.62 MB/sec execute 84 sec latency 1075.906 ms 1 11173 0.61 MB/sec execute 85 sec latency 2076.151 ms 1 11173 0.61 MB/sec execute 86 sec latency 3076.394 ms 1 11173 0.60 MB/sec execute 87 sec latency 4076.615 ms Failover mds2 to oleg416-server mount facets: mds2 1 11173 0.59 MB/sec execute 88 sec latency 5076.793 ms 1 11173 0.59 MB/sec execute 89 sec latency 6076.936 ms 1 11173 0.58 MB/sec execute 90 sec latency 7077.191 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 11173 0.57 MB/sec execute 91 sec latency 8077.421 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 1 11173 0.57 MB/sec execute 92 sec latency 9077.611 ms pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0001 10:52:24 (1749480744) targets are mounted 10:52:24 (1749480744) facet_failover done 1 11173 0.56 MB/sec execute 93 sec latency 10077.860 ms 1 11173 0.56 MB/sec execute 94 sec latency 11078.040 ms 1 11173 0.55 MB/sec execute 95 sec latency 12078.186 ms 1 11173 0.54 MB/sec execute 96 sec latency 13078.371 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 11346 0.54 MB/sec execute 97 sec latency 13386.948 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11569 0.54 MB/sec execute 98 sec latency 26.193 ms 1 11756 0.55 MB/sec execute 99 sec latency 29.849 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1805 24.161 24331.891 Close 1329 1.979 20.724 Rename 78 17.410 32.327 Unlink 368 5.265 19.367 Qpathinfo 1663 22.276 18479.519 Qfileinfo 285 0.514 4.507 Qfsinfo 303 0.989 5.758 Sfileinfo 150 10.530 23.685 Find 632 33.127 19000.675 WriteX 899 2.380 13.441 ReadX 2786 0.118 39.972 LockX 6 1.588 2.474 UnlockX 6 1.749 2.960 Flush 128 15.933 522.372 Throughput 0.547986 MB/sec 1 clients 1 procs max_latency=24331.906 ms stopping dbench on /mnt/lustre at Mon Jun 9 10:52:33 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5364 1282324 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3440 1284248 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23664 3578444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 54252 3548804 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 77916 7127248 2% /mnt/lustre striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Jun 9 10:52:34 EDT 2025 waiting for dbench pid 9621 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 102 4.72 MB/sec warmup 1 sec latency 17.689 ms 1 243 4.60 MB/sec warmup 2 sec latency 35.423 ms test_26 fail mds1 5 times 1 393 4.63 MB/sec warmup 3 sec latency 32.792 ms Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 649 5.27 MB/sec warmup 4 sec latency 29.234 ms 10:52:39 (1749480759) shut down facet: mds1 facet_host: oleg416-server facet_failover_host: oleg416-server 1 653 4.21 MB/sec warmup 5 sec latency 975.919 ms 1 653 3.51 MB/sec warmup 6 sec latency 1976.249 ms 1 653 3.01 MB/sec warmup 7 sec latency 2976.579 ms 1 653 2.63 MB/sec warmup 8 sec latency 3976.912 ms 1 653 2.34 MB/sec warmup 9 sec latency 4977.244 ms 1 653 2.11 MB/sec warmup 10 sec latency 5977.513 ms 1 653 1.91 MB/sec warmup 11 sec latency 6977.766 ms 1 653 1.76 MB/sec warmup 12 sec latency 7978.088 ms 1 653 1.62 MB/sec warmup 13 sec latency 8978.451 ms 1 653 1.50 MB/sec warmup 14 sec latency 9978.800 ms Failover mds1 to oleg416-server mount facets: mds1 1 653 1.40 MB/sec warmup 15 sec latency 10979.010 ms 1 653 1.32 MB/sec warmup 16 sec latency 11979.281 ms 1 653 1.24 MB/sec warmup 17 sec latency 12979.493 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 653 1.17 MB/sec warmup 18 sec latency 13979.755 ms 1 653 1.11 MB/sec warmup 19 sec latency 14980.041 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 10:52:54 (1749480774) targets are mounted 10:52:54 (1749480774) facet_failover done 1 653 0.00 MB/sec execute 1 sec latency 16980.281 ms 1 653 0.00 MB/sec execute 2 sec latency 17980.551 ms 1 653 0.00 MB/sec execute 3 sec latency 18980.742 ms 1 653 0.00 MB/sec execute 4 sec latency 19980.954 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 845 0.10 MB/sec execute 5 sec latency 20278.044 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1009 0.21 MB/sec execute 6 sec latency 49.011 ms 1 1231 0.29 MB/sec execute 7 sec latency 72.853 ms 1 1490 0.62 MB/sec execute 8 sec latency 39.242 ms 1 1696 0.57 MB/sec execute 9 sec latency 24.197 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4640 1283048 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3032 1284656 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 41388 3549512 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19112 3576036 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 60500 7125548 1% /mnt/lustre 1 1984 0.54 MB/sec execute 10 sec latency 22.393 ms 1 2228 0.52 MB/sec execute 11 sec latency 21.780 ms 1 2436 0.59 MB/sec execute 12 sec latency 46.582 ms 1 2716 0.84 MB/sec execute 13 sec latency 28.200 ms test_26 fail mds2 6 times Failing mds2 on oleg416-server Stopping /mnt/lustre-mds2 (opts:) on oleg416-server 1 3170 0.93 MB/sec execute 14 sec latency 33.658 ms 10:53:09 (1749480789) shut down facet: mds2 facet_host: oleg416-server facet_failover_host: oleg416-server 1 3548 1.16 MB/sec execute 15 sec latency 26.744 ms 1 3738 1.13 MB/sec execute 16 sec latency 20.684 ms 1 3885 1.10 MB/sec execute 17 sec latency 25.792 ms 1 4049 1.06 MB/sec execute 18 sec latency 24.726 ms 1 4083 1.00 MB/sec execute 19 sec latency 790.788 ms 1 4083 0.95 MB/sec execute 20 sec latency 1792.112 ms 1 4083 0.91 MB/sec execute 21 sec latency 2792.394 ms 1 4083 0.86 MB/sec execute 22 sec latency 3792.573 ms 1 4083 0.83 MB/sec execute 23 sec latency 4792.805 ms 1 4083 0.79 MB/sec execute 24 sec latency 5792.952 ms Failover mds2 to oleg416-server 1 4083 0.76 MB/sec execute 25 sec latency 6793.078 ms mount facets: mds2 1 4083 0.73 MB/sec execute 26 sec latency 7793.260 ms 1 4083 0.70 MB/sec execute 27 sec latency 8793.474 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4083 0.68 MB/sec execute 28 sec latency 9793.646 ms 1 4083 0.66 MB/sec execute 29 sec latency 10793.825 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 1 4083 0.63 MB/sec execute 30 sec latency 11793.974 ms Started lustre-MDT0001 10:53:25 (1749480805) targets are mounted 10:53:25 (1749480805) facet_failover done 1 4083 0.61 MB/sec execute 31 sec latency 12794.200 ms 1 4083 0.59 MB/sec execute 32 sec latency 13794.399 ms 1 4083 0.58 MB/sec execute 33 sec latency 14794.630 ms 1 4138 0.56 MB/sec execute 34 sec latency 15522.459 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4360 0.56 MB/sec execute 35 sec latency 26.880 ms 1 4526 0.55 MB/sec execute 36 sec latency 26.975 ms 1 4710 0.57 MB/sec execute 37 sec latency 35.181 ms 1 4942 0.63 MB/sec execute 38 sec latency 31.104 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2256 1285432 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38936 3567312 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21220 3585660 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 60156 7152972 1% /mnt/lustre 1 5117 0.62 MB/sec execute 39 sec latency 32.447 ms 1 5292 0.61 MB/sec execute 40 sec latency 22.405 ms 1 5530 0.60 MB/sec execute 41 sec latency 24.722 ms 1 5724 0.59 MB/sec execute 42 sec latency 22.970 ms striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar test_26 fail mds1 7 times 1 5947 0.60 MB/sec execute 43 sec latency 45.041 ms Failing mds1 on oleg416-server Stopping /mnt/lustre-mds1 (opts:) on oleg416-server 1 6144 0.61 MB/sec execute 44 sec latency 155.415 ms 1 6144 0.60 MB/sec execute 45 sec latency 1155.716 ms 1 6144 0.59 MB/sec execute 46 sec latency 2155.997 ms 1 6144 0.57 MB/sec execute 47 sec latency 3156.281 ms 1 6144 0.56 MB/sec execute 48 sec latency 4156.608 ms 1 6144 0.55 MB/sec execute 49 sec latency 5156.976 ms 1 6144 0.54 MB/sec execute 50 sec latency 6157.226 ms 10:53:45 (1749480825) shut down facet: mds1 facet_host: oleg416-server facet_failover_host: oleg416-server 1 6144 0.53 MB/sec execute 51 sec latency 7157.619 ms 1 6144 0.52 MB/sec execute 52 sec latency 8157.986 ms 1 6144 0.51 MB/sec execute 53 sec latency 9158.281 ms 1 6144 0.50 MB/sec execute 54 sec latency 10158.637 ms 1 6144 0.49 MB/sec execute 55 sec latency 11158.886 ms 1 6144 0.48 MB/sec execute 56 sec latency 12159.057 ms 1 6144 0.47 MB/sec execute 57 sec latency 13159.328 ms 1 6144 0.47 MB/sec execute 58 sec latency 14159.583 ms 1 6144 0.46 MB/sec execute 59 sec latency 15159.875 ms 1 6144 0.45 MB/sec execute 60 sec latency 16160.188 ms Failover mds1 to oleg416-server mount facets: mds1 1 6144 0.44 MB/sec execute 61 sec latency 17160.419 ms 1 6144 0.44 MB/sec execute 62 sec latency 18160.631 ms 1 6144 0.43 MB/sec execute 63 sec latency 19160.860 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 6144 0.42 MB/sec execute 64 sec latency 20161.057 ms oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all 1 6144 0.42 MB/sec execute 65 sec latency 21161.253 ms pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 10:54:00 (1749480840) targets are mounted 10:54:00 (1749480840) facet_failover done 1 6144 0.41 MB/sec execute 66 sec latency 22161.570 ms 1 6144 0.40 MB/sec execute 67 sec latency 23161.827 ms 1 6144 0.40 MB/sec execute 68 sec latency 24162.099 ms 1 6144 0.39 MB/sec execute 69 sec latency 25162.307 ms 1 6309 0.43 MB/sec execute 70 sec latency 25857.741 ms oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6765 0.46 MB/sec execute 71 sec latency 29.913 ms 1 7102 0.51 MB/sec execute 72 sec latency 24.076 ms 1 7325 0.52 MB/sec execute 73 sec latency 22.019 ms 1 7451 0.51 MB/sec execute 74 sec latency 23.862 ms 1 7599 0.51 MB/sec execute 75 sec latency 47.202 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Jun 9 10:54:10 EDT 2025 with return code 0 9621 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 9621 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished