== replay-dual test 26: dbench and tar with mds failover ========================================================== 09:26:58 (1747402018) Starting client oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre Started clients oleg349-client.virtnet: 192.168.203.149@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 6790 Started dbench loop with 6791 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri May 16 09:26:59 EDT 2025 waiting for dbench pid 6826 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 287 10.70 MB/sec warmup 1 sec latency 17.205 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2268 1285420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25984 3556088 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27372 7161720 1% /mnt/lustre 1 580 9.85 MB/sec warmup 2 sec latency 20.676 ms 1 796 7.16 MB/sec warmup 3 sec latency 440.695 ms 1 1170 5.75 MB/sec warmup 4 sec latency 13.731 ms test_26 fail mds1 1 times 1 1614 5.23 MB/sec warmup 5 sec latency 14.224 ms Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 1 1877 4.39 MB/sec warmup 6 sec latency 264.320 ms 1 1877 3.77 MB/sec warmup 7 sec latency 1264.483 ms 1 1877 3.30 MB/sec warmup 8 sec latency 2264.645 ms 1 1877 2.93 MB/sec warmup 9 sec latency 3264.838 ms 1 1877 2.64 MB/sec warmup 10 sec latency 4264.999 ms 1 1877 2.40 MB/sec warmup 11 sec latency 5265.123 ms 1 1877 2.20 MB/sec warmup 12 sec latency 6265.299 ms reboot facets: mds1 1 1877 2.03 MB/sec warmup 13 sec latency 7265.479 ms 1 1877 1.88 MB/sec warmup 14 sec latency 8265.652 ms 1 1877 1.76 MB/sec warmup 15 sec latency 9265.843 ms 1 1877 1.65 MB/sec warmup 16 sec latency 10265.981 ms 1 1877 1.55 MB/sec warmup 17 sec latency 11266.113 ms 1 1877 1.46 MB/sec warmup 18 sec latency 12266.244 ms 1 1877 1.39 MB/sec warmup 19 sec latency 13266.477 ms 1 1877 0.00 MB/sec execute 1 sec latency 15266.778 ms 1 1877 0.00 MB/sec execute 2 sec latency 16266.947 ms Failover mds1 to oleg349-server mount facets: mds1 1 1877 0.00 MB/sec execute 3 sec latency 17267.081 ms 1 1877 0.00 MB/sec execute 4 sec latency 18267.223 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1877 0.00 MB/sec execute 5 sec latency 19267.387 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 1 1877 0.00 MB/sec execute 6 sec latency 20267.569 ms pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1877 0.00 MB/sec execute 7 sec latency 21267.779 ms 1 1877 0.00 MB/sec execute 8 sec latency 22268.008 ms 1 1877 0.00 MB/sec execute 9 sec latency 23268.152 ms 1 1877 0.00 MB/sec execute 10 sec latency 24268.306 ms 1 1973 0.01 MB/sec execute 11 sec latency 25073.540 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2376 0.13 MB/sec execute 12 sec latency 13.664 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2919 0.55 MB/sec execute 13 sec latency 15.137 ms 1 3587 0.86 MB/sec execute 14 sec latency 16.091 ms 1 3939 0.91 MB/sec execute 15 sec latency 12.586 ms 1 4231 0.87 MB/sec execute 16 sec latency 16.298 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2820 1284868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36916 3566340 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12768 3590876 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49684 7157216 1% /mnt/lustre 1 4550 0.86 MB/sec execute 17 sec latency 19.677 ms 1 5019 1.04 MB/sec execute 18 sec latency 15.202 ms 1 5379 1.00 MB/sec execute 19 sec latency 12.958 ms test_26 fail mds2 2 times 1 5760 0.98 MB/sec execute 20 sec latency 14.643 ms Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server 1 6012 0.99 MB/sec execute 21 sec latency 156.715 ms reboot facets: mds2 1 6012 0.95 MB/sec execute 22 sec latency 1156.950 ms 1 6012 0.91 MB/sec execute 23 sec latency 2157.195 ms 1 6012 0.87 MB/sec execute 24 sec latency 3157.414 ms 1 6012 0.83 MB/sec execute 25 sec latency 4157.556 ms 1 6012 0.80 MB/sec execute 26 sec latency 5157.744 ms 1 6012 0.77 MB/sec execute 27 sec latency 6157.973 ms 1 6012 0.75 MB/sec execute 28 sec latency 7158.186 ms 1 6012 0.72 MB/sec execute 29 sec latency 8158.343 ms 1 6012 0.70 MB/sec execute 30 sec latency 9158.538 ms 1 6012 0.67 MB/sec execute 31 sec latency 10158.692 ms Failover mds2 to oleg349-server mount facets: mds2 1 6012 0.65 MB/sec execute 32 sec latency 11158.809 ms 1 6012 0.63 MB/sec execute 33 sec latency 12158.908 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 6012 0.61 MB/sec execute 34 sec latency 13159.014 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 1 6012 0.60 MB/sec execute 35 sec latency 14159.138 ms pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 1 6012 0.58 MB/sec execute 36 sec latency 15159.244 ms 1 6012 0.56 MB/sec execute 37 sec latency 16159.357 ms 1 6012 0.55 MB/sec execute 38 sec latency 17159.508 ms 1 6012 0.53 MB/sec execute 39 sec latency 18159.683 ms 1 6012 0.52 MB/sec execute 40 sec latency 19159.816 ms 1 6049 0.52 MB/sec execute 41 sec latency 20030.579 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 6724 0.64 MB/sec execute 42 sec latency 20.813 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7230 0.73 MB/sec execute 43 sec latency 12.989 ms 1 7516 0.75 MB/sec execute 44 sec latency 14.901 ms 1 7856 0.74 MB/sec execute 45 sec latency 13.308 ms 1 8237 0.76 MB/sec execute 46 sec latency 13.415 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3556 1284132 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37332 3556040 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13388 3585528 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50720 7141568 1% /mnt/lustre 1 8633 0.81 MB/sec execute 47 sec latency 42.864 ms 1 8887 0.79 MB/sec execute 48 sec latency 322.913 ms 1 9340 0.79 MB/sec execute 49 sec latency 9.179 ms test_26 fail mds1 3 times 1 9724 0.82 MB/sec execute 50 sec latency 36.612 ms Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 1 10463 0.93 MB/sec execute 51 sec latency 15.935 ms reboot facets: mds1 1 10910 0.99 MB/sec execute 52 sec latency 13.774 ms 1 11157 0.98 MB/sec execute 53 sec latency 14.843 ms 1 11174 0.96 MB/sec execute 54 sec latency 950.072 ms 1 11174 0.94 MB/sec execute 55 sec latency 1950.179 ms 1 11174 0.93 MB/sec execute 56 sec latency 2950.289 ms 1 11174 0.91 MB/sec execute 57 sec latency 3950.406 ms 1 11174 0.90 MB/sec execute 58 sec latency 4950.547 ms 1 11174 0.88 MB/sec execute 59 sec latency 5950.706 ms 1 11174 0.87 MB/sec execute 60 sec latency 6950.825 ms 1 11174 0.85 MB/sec execute 61 sec latency 7950.945 ms Failover mds1 to oleg349-server mount facets: mds1 1 11174 0.84 MB/sec execute 62 sec latency 8951.072 ms 1 11174 0.82 MB/sec execute 63 sec latency 9951.250 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11174 0.81 MB/sec execute 64 sec latency 10951.395 ms 1 11174 0.80 MB/sec execute 65 sec latency 11951.521 ms 1 11174 0.79 MB/sec execute 66 sec latency 12951.644 ms 1 11174 0.78 MB/sec execute 67 sec latency 13951.764 ms 1 11174 0.76 MB/sec execute 68 sec latency 14951.924 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 1 11174 0.75 MB/sec execute 69 sec latency 15952.050 ms pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 1 11174 0.74 MB/sec execute 70 sec latency 16952.165 ms 1 11174 0.73 MB/sec execute 71 sec latency 17952.301 ms 1 11174 0.72 MB/sec execute 72 sec latency 18952.437 ms 1 11174 0.71 MB/sec execute 73 sec latency 19952.592 ms 1 11480 0.71 MB/sec execute 74 sec latency 20050.424 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11820 0.72 MB/sec execute 75 sec latency 16.005 ms 1 12239 0.75 MB/sec execute 76 sec latency 13.350 ms 1 12612 0.75 MB/sec execute 77 sec latency 23.972 ms 1 13032 0.76 MB/sec execute 78 sec latency 11.878 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3048 1284640 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4692 1282996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 45696 3551916 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24128 3570776 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 69824 7122692 1% /mnt/lustre 1 13546 0.81 MB/sec execute 79 sec latency 18.045 ms 1 13887 0.82 MB/sec execute 80 sec latency 416.012 ms 1 14364 0.86 MB/sec execute 81 sec latency 13.209 ms 1 14634 0.87 MB/sec execute 82 sec latency 14.177 ms test_26 fail mds2 4 times Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server 1 14976 0.86 MB/sec execute 83 sec latency 14.481 ms reboot facets: mds2 1 15200 0.86 MB/sec execute 84 sec latency 251.991 ms 1 15200 0.85 MB/sec execute 85 sec latency 1252.154 ms 1 15200 0.84 MB/sec execute 86 sec latency 2252.286 ms 1 15200 0.83 MB/sec execute 87 sec latency 3252.440 ms 1 15200 0.82 MB/sec execute 88 sec latency 4252.577 ms 1 15200 0.81 MB/sec execute 89 sec latency 5252.739 ms 1 15200 0.80 MB/sec execute 90 sec latency 6252.902 ms 1 15200 0.79 MB/sec execute 91 sec latency 7253.044 ms 1 15200 0.78 MB/sec execute 92 sec latency 8253.189 ms 1 15200 0.77 MB/sec execute 93 sec latency 9253.333 ms Failover mds2 to oleg349-server 1 15200 0.77 MB/sec execute 94 sec latency 10253.422 ms mount facets: mds2 1 15200 0.76 MB/sec execute 95 sec latency 11253.511 ms 1 15200 0.75 MB/sec execute 96 sec latency 12253.631 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 15200 0.74 MB/sec execute 97 sec latency 13253.744 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 1 15200 0.73 MB/sec execute 98 sec latency 14253.859 ms 1 15200 0.73 MB/sec execute 99 sec latency 15253.986 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2278 26.574 25073.489 Close 1670 1.369 8.210 Rename 97 11.042 42.841 Unlink 482 45.284 20030.562 Qpathinfo 2087 11.486 20050.409 Qfileinfo 357 0.350 1.823 Qfsinfo 392 0.694 14.632 Sfileinfo 184 5.148 10.286 Find 813 1.416 18.792 WriteX 1140 1.902 8.932 ReadX 3639 0.032 2.256 LockX 8 1.239 1.418 UnlockX 8 1.282 1.397 Flush 169 11.693 416.001 Throughput 0.727447 MB/sec 1 clients 1 procs max_latency=25073.540 ms stopping dbench on /mnt/lustre at Fri May 16 09:29:02 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri May 16 09:29:03 EDT 2025 waiting for dbench pid 11600 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 292 10.70 MB/sec warmup 1 sec latency 18.238 ms 1 601 10.48 MB/sec warmup 2 sec latency 28.525 ms 1 948 7.27 MB/sec warmup 3 sec latency 17.026 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3572 1284116 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5228 1282460 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36992 3529232 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18104 3568116 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 55096 7097348 1% /mnt/lustre 1 1281 5.79 MB/sec warmup 4 sec latency 15.835 ms 1 1660 5.24 MB/sec warmup 5 sec latency 178.307 ms 1 2093 4.44 MB/sec warmup 6 sec latency 13.037 ms 1 2490 4.10 MB/sec warmup 7 sec latency 25.861 ms test_26 fail mds1 5 times Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 1 3197 4.29 MB/sec warmup 8 sec latency 13.410 ms 1 3720 4.31 MB/sec warmup 9 sec latency 31.282 ms 1 4010 4.01 MB/sec warmup 10 sec latency 14.043 ms 1 4083 3.64 MB/sec warmup 11 sec latency 753.000 ms 1 4083 3.34 MB/sec warmup 12 sec latency 1753.156 ms 1 4083 3.08 MB/sec warmup 13 sec latency 2753.319 ms reboot facets: mds1 1 4083 2.86 MB/sec warmup 14 sec latency 3753.483 ms 1 4083 2.67 MB/sec warmup 15 sec latency 4753.703 ms 1 4083 2.51 MB/sec warmup 16 sec latency 5753.856 ms 1 4083 2.36 MB/sec warmup 17 sec latency 6754.018 ms 1 4083 2.23 MB/sec warmup 18 sec latency 7754.210 ms 1 4083 2.11 MB/sec warmup 19 sec latency 8754.378 ms 1 4083 0.00 MB/sec execute 1 sec latency 10754.712 ms 1 4083 0.00 MB/sec execute 2 sec latency 11754.851 ms 1 4083 0.00 MB/sec execute 3 sec latency 12754.995 ms Failover mds1 to oleg349-server 1 4083 0.00 MB/sec execute 4 sec latency 13755.156 ms mount facets: mds1 1 4083 0.00 MB/sec execute 5 sec latency 14755.320 ms 1 4083 0.00 MB/sec execute 6 sec latency 15755.439 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.00 MB/sec execute 7 sec latency 16755.637 ms 1 4083 0.00 MB/sec execute 8 sec latency 17755.790 ms 1 4083 0.00 MB/sec execute 9 sec latency 18755.949 ms 1 4083 0.00 MB/sec execute 10 sec latency 19756.113 ms 1 4083 0.00 MB/sec execute 11 sec latency 20756.275 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4083 0.00 MB/sec execute 12 sec latency 21756.469 ms 1 4083 0.00 MB/sec execute 13 sec latency 22756.594 ms 1 4083 0.00 MB/sec execute 14 sec latency 23756.728 ms 1 4083 0.00 MB/sec execute 15 sec latency 24756.886 ms 1 4083 0.00 MB/sec execute 16 sec latency 25757.009 ms 1 4367 0.03 MB/sec execute 17 sec latency 25814.113 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4687 0.11 MB/sec execute 18 sec latency 17.214 ms 1 5093 0.27 MB/sec execute 19 sec latency 17.855 ms 1 5422 0.27 MB/sec execute 20 sec latency 13.676 ms 1 5893 0.33 MB/sec execute 21 sec latency 9.607 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2692 1284996 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3728 1283960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 45788 3559316 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21000 3585360 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 66788 7144676 1% /mnt/lustre 1 6167 0.37 MB/sec execute 22 sec latency 59.611 ms 1 6555 0.55 MB/sec execute 23 sec latency 368.394 ms 1 7164 0.73 MB/sec execute 24 sec latency 402.072 ms 1 7447 0.76 MB/sec execute 25 sec latency 14.049 ms test_26 fail mds2 6 times Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server 1 7722 0.74 MB/sec execute 26 sec latency 16.200 ms reboot facets: mds2 1 8026 0.74 MB/sec execute 27 sec latency 66.359 ms 1 8106 0.72 MB/sec execute 28 sec latency 717.813 ms 1 8106 0.69 MB/sec execute 29 sec latency 1718.013 ms 1 8106 0.67 MB/sec execute 30 sec latency 2718.179 ms 1 8106 0.65 MB/sec execute 31 sec latency 3718.349 ms 1 8106 0.63 MB/sec execute 32 sec latency 4718.512 ms 1 8106 0.61 MB/sec execute 33 sec latency 5718.667 ms 1 8106 0.59 MB/sec execute 34 sec latency 6718.833 ms 1 8106 0.57 MB/sec execute 35 sec latency 7718.980 ms 1 8106 0.56 MB/sec execute 36 sec latency 8719.162 ms Failover mds2 to oleg349-server 1 8106 0.54 MB/sec execute 37 sec latency 9719.326 ms mount facets: mds2 1 8106 0.53 MB/sec execute 38 sec latency 10719.450 ms 1 8106 0.51 MB/sec execute 39 sec latency 11719.570 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 8106 0.50 MB/sec execute 40 sec latency 12719.697 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 1 8106 0.49 MB/sec execute 41 sec latency 13719.860 ms Started lustre-MDT0001 1 8106 0.48 MB/sec execute 42 sec latency 14720.009 ms 1 8106 0.47 MB/sec execute 43 sec latency 15720.158 ms 1 8106 0.46 MB/sec execute 44 sec latency 16720.332 ms 1 8106 0.45 MB/sec execute 45 sec latency 17720.460 ms 1 8144 0.46 MB/sec execute 46 sec latency 18601.140 ms 1 8519 0.51 MB/sec execute 47 sec latency 19.413 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8797 0.51 MB/sec execute 48 sec latency 18.065 ms 1 9140 0.50 MB/sec execute 49 sec latency 14.547 ms 1 9493 0.52 MB/sec execute 50 sec latency 49.562 ms 1 9995 0.61 MB/sec execute 51 sec latency 15.484 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2508 1285180 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 43160 3561604 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 20124 3583980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 63284 7145584 1% /mnt/lustre 1 10476 0.65 MB/sec execute 52 sec latency 14.651 ms 1 10879 0.72 MB/sec execute 53 sec latency 17.366 ms 1 11098 0.71 MB/sec execute 54 sec latency 27.334 ms 1 11419 0.70 MB/sec execute 55 sec latency 24.808 ms test_26 fail mds1 7 times striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 1 11718 0.72 MB/sec execute 56 sec latency 23.920 ms reboot facets: mds1 1 11769 0.71 MB/sec execute 57 sec latency 878.030 ms 1 11769 0.69 MB/sec execute 58 sec latency 1878.250 ms 1 11769 0.68 MB/sec execute 59 sec latency 2878.469 ms 1 11769 0.67 MB/sec execute 60 sec latency 3878.680 ms 1 11769 0.66 MB/sec execute 61 sec latency 4878.919 ms 1 11769 0.65 MB/sec execute 62 sec latency 5879.133 ms 1 11769 0.64 MB/sec execute 63 sec latency 6879.353 ms 1 11769 0.63 MB/sec execute 64 sec latency 7879.595 ms 1 11769 0.62 MB/sec execute 65 sec latency 8879.812 ms 1 11769 0.61 MB/sec execute 66 sec latency 9880.070 ms Failover mds1 to oleg349-server 1 11769 0.60 MB/sec execute 67 sec latency 10880.217 ms mount facets: mds1 1 11769 0.59 MB/sec execute 68 sec latency 11880.398 ms 1 11769 0.58 MB/sec execute 69 sec latency 12880.553 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11769 0.57 MB/sec execute 70 sec latency 13880.701 ms 1 11769 0.57 MB/sec execute 71 sec latency 14880.887 ms 1 11769 0.56 MB/sec execute 72 sec latency 15881.028 ms 1 11769 0.55 MB/sec execute 73 sec latency 16881.199 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 1 11769 0.54 MB/sec execute 74 sec latency 17881.420 ms Started lustre-MDT0000 1 11769 0.54 MB/sec execute 75 sec latency 18881.579 ms 1 11769 0.53 MB/sec execute 76 sec latency 19881.736 ms 1 11769 0.52 MB/sec execute 77 sec latency 20881.896 ms 1 11769 0.52 MB/sec execute 78 sec latency 21882.050 ms 1 12059 0.55 MB/sec execute 79 sec latency 22215.855 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 12380 0.54 MB/sec execute 80 sec latency 14.851 ms 1 12803 0.54 MB/sec execute 81 sec latency 11.347 ms 1 13214 0.56 MB/sec execute 82 sec latency 12.981 ms 1 13896 0.63 MB/sec execute 83 sec latency 13.428 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri May 16 09:30:47 EDT 2025 with return code 0 11600 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 11600 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished